Elasticsearch snapshot restore: where backup breaks
Mandatory Elasticsearch backup via snapshots looks straightforward while the cluster is small: a scheduled job, a repository, rare checks.
At scale, it is not the mechanism itself that breaks but the tooling around it — and the first thing to give way is usually the snapshot repository, which stops accommodating new index snapshots.
The classic snapshot repository failure is full storage or revoked access rights. The job fails, but no alerting is configured, and for weeks no copies are created at all. This is discovered at restore time, when data for the missed period is already gone.
The second class of failures is overlapping jobs, when the schedule starts a new snapshot before the previous one finishes.
Index snapshots come out incomplete: some shards are missing or checksums do not match, restore stops midway, and replicas diverge — one node serves old data, another already deleted data.
The third is lost shards. The snapshot is formally successful, but some data did not make it because a shard was relocating between nodes during the snapshot. The error is not visible in job logs or cluster status — only when checking via restore to an isolated environment.
We break down how this is addressed: repository health monitoring, serializing scheduled jobs, and regular verification of snapshots by actual restore.
Measurable effect: recovery time and data loss
Scheduled backup turns recovery from an emergency scenario into a routine operation: Elasticsearch snapshots are taken on schedule, not at the moment something has already broken.
Below is what exactly changes in measurable terms: how much data is lost between snapshots, how long cluster recovery takes, and which class of failures goes away entirely.
How much data is lost between snapshots
The recovery point objective is set by snapshot frequency: a fifteen-minute interval limits operation loss to that window rather than an entire business day.
Incremental snapshots carry only changed index segments, so you can back up often — storage grows slowly, and disk load stays within normal limits. Retention is configured additionally: old copies are removed by policy, and the snapshot registry does not grow uncontrollably.
Full cluster restore vs selective restore
A full restore brings the entire cluster back — this is needed when the search layer itself or index settings are damaged.
A selective restore brings up a single index or a separate dataset in minutes without affecting the rest, and covers most incidents: accidental deletion, ingestion failure, corrupted mapping.
Cluster recovery time in this case is measured in minutes, not hours of waiting for reindexing from the primary source.
The backup window does not hit peak load
A snapshot is taken without stopping the cluster, but operation duration matters: if it falls into peak hours, search response degrades.
We schedule it for a minimum-load window and limit backup throughput, so the recovery point objective improves without degrading service for users.
The result is verified not in words: restore is regularly rehearsed in a test environment, and it is measured in practice how much data actually rolled back and how long it took the cluster to return to service.
Snapshot repository options and when they fit
A snapshot repository is the place where the cluster writes copies of indices and state. The storage type is chosen once at registration: it determines whether a snapshot survives node failure, how long restore takes, and how retention policies are laid out. Below are working options and their limits.
Node local disk
A path on a node disk is registered fastest and is suitable for a staging environment and one-off checks of restore logic. But snapshots live exactly as long as the node itself: if the disk fails, there are no copies. Such a repository also is not shared between all nodes, so it is not used in production.
Object storage and network share
Object storage for backups is the main option for a production cluster: snapshots reside outside the nodes, are accessible to each of them, and survive the loss of any one. A network share provides similar fault tolerance but is sensitive to locks and network state.
Sometimes it makes sense to keep several repositories: one for frequent snapshots of hot indices and one for archive.
| Option | When it fits | Limitations |
|---|---|---|
| Node local disk | Staging, one-off restore check | Snapshots die with the node; not visible to other nodes |
| Object storage | Production cluster, long-term retention | Writes go over the network; correct access rights are required |
| Network share | When a fault-tolerant file array already exists | Depends on network and locks; degradation slows snapshots |
| Multiple repositories | Different policies for hot and archive indices | More configurations to keep consistent |
Implementation stages: audit, design, verification, handover
The audit begins not with configuration but with analyzing what already exists: which indices live in Elasticsearch, how fast they grow, where snapshots are stored, and who is responsible for their state.
Without this, a snapshot policy either covers more than needed or leaves the most heavily loaded data unprotected.
Audit of the current backup scheme
- Inventory: list indices and templates, separate working data from service data, record growth and the actually required retention period.
- Snapshot storage check: access rights, free space, behavior under concurrent writes, snapshots that are created but have never been restored.
- Snapshot policy: frequency, retention depth, index binding, launch window outside peak load, rotation rules for old copies.
Verification by restore
- Deploy a copy of the cluster in an isolated environment and restore the snapshot in full — to the extent it would have to be brought up during an incident.
- Compare the result: number of documents and indices, mappings and settings, operation of search queries and aggregations on the restored data.
- Measure recovery time and the point to which data is lost during a failure; if we do not meet the allowed target, change frequency and retention depth.
The restore runbook is handed over along with configurations: step-by-step instructions, roles and access, acceptance criteria, escalation procedure. The criterion for completed work is that the customer's team brings up the cluster without us.
Snapshot policies, retention periods, and restore schemes
A working snapshot policy differs from a configured one in that it can be executed by an on-call engineer who did not write that configuration.
Therefore, we hand over to the customer not only the repository with copies but also the runbook: what is snapshotted, how long it lives, who has the right to write, who can restore, and how to verify it before an incident, not during one.
On-call runbook and restore procedure
- Snapshot policy description — for each index and data stream: frequency, launch time, allowed delay window, behavior when the repository is unavailable.
- Snapshot lifecycle — how long each tier is retained (hourly, daily, weekly), what is deleted automatically, what is protected from deletion until the reporting period closes.
- Repository placement scheme — where copies physically reside, how this relates to cluster nodes, why the repository cannot share a failure domain with production.
- Repository access roles — separate accounts for write, read, and restore; a shared on-call account is excluded, and all operations go into the audit log.
- Restore scheme — separate branches for loss of a single index, rollback of the entire cluster, and transfer to a reserve site, indicating what exactly is required from the customer at each step.
- Verification checklist — regular test restore into an isolated environment, comparison of document counts, mappings, and analysis settings, report with the result.
- Backup documentation — where it is stored, who owns it, what is updated and within what timeframe when a new index is added or topology changes.
After handover, it is clear that restore does not depend on one person: the on-call engineer acts according to the scheme, and changes to the policy go through the same owner as changes to the cluster.
Case study: cluster restore after a site failure
The incident began with a site failure: nodes with primary shards went down, and replicas turned out to be placed in the same availability zone. Elasticsearch snapshots were taken on schedule but resided in the same environment, and restore from them had never been tested.
At the moment of failure, there was nothing to bring the cluster up with: the last suitable snapshot was a week old, and the repository was unavailable along with the site.
Before and after
First, the snapshot repository was moved to independent object storage, and shard placement was rebuilt so that replicas did not overlap with primaries by zone. Then snapshot policies were configured: daily incremental, weekly full, retention by number of copies per tag.
Separately, a copy of the cluster was built in an isolated environment — without this step, a snapshot remains a hope rather than a restore plan.
| Metric | Before | After |
|---|---|---|
| Data available after a site failure | part of the indices | the entire cluster |
| Time to working state | about 19 hours of manual troubleshooting | 40 minutes |
| Snapshot verification | quarterly, manually | weekly, automated test on a copy |
| Snapshot placement | same site | independent storage |
Every week, a snapshot is restored on a copy: document counts and checksums of key indices are compared, and a mismatch sends a notification to the on-call engineer.
Cluster restore has ceased to be an emergency scenario — rollback to the previous release and migration between sites now fit within an on-call shift.
How does a snapshot differ from an index replica?
A replica is a copy of a shard within the same cluster: it keeps reads going when a node drops out but shares a failure domain with the primary shard.
A snapshot is an export of index segments to storage outside the cluster: it survives complete cluster loss and restores from scratch.
The mechanisms cover different classes of failures and therefore complement each other. A replica provides continuity: a node drops out, and the cluster keeps running.
A snapshot provides reversibility: an erroneous mapping or bulk edit rolls back to the state at the time of the snapshot — something replicas would spread across all copies in seconds.
Selective index restore is the reason to keep snapshots not only "for a rainy day": from a snapshot, a single index is brought up, if needed under a different name, compared with the working one, and the required parts are transferred.
Cross-cluster data transfer works the same way — the shared repository is available to both sides, and intermediate exports are not needed.
Common technical doubts about snapshots and restore
An Elasticsearch snapshot is taken at the segment level: the engine captures the state at launch time and continues accepting writes.
Blocks are copied in the background with throughput throttling, so the cluster does not stop serving requests, although disk and network load increases during the export.
Does a snapshot stop writes to indices?
No. Indices remain available for reads and writes, and segment copying proceeds in parallel. We throttle export speed so it does not compete with production traffic or bloat the task queue.
How should a snapshot repository be laid out?
A separate path per environment and purpose: production snapshots, pre-release, archive. We connect the restore repository in read-only mode — this way, accidental overwriting of a production snapshot is excluded at the access-rights level.
How do you make sure a snapshot is usable?
By restoring to an isolated environment, not by checking file size. We compare document counts and key indices, and record the result in the verification log — otherwise rollback turns into a lottery.
How much space do snapshots take in storage?
Less than it seems: duplicate segments are not copied twice, and each subsequent snapshot adds only changed data. We remove old versions according to the retention rule, so the repository size plateaus over time.
Can a single index be restored?
Yes, a snapshot is restored selectively — by index list and with renaming to the target name. This speeds up targeted rollback: we return one index without affecting the rest of the data and without rewriting the entire cluster.
Let's discuss the task: what to clarify about your cluster
To make the work estimate accurate and the implementation plan feasible without surprises, we need a short set of inputs about your Elasticsearch cluster. Exports, access, and a test environment are not required at this stage: a free-form description is enough.
From it, it is already clear whether a full snapshot will fit into the maintenance window or incremental ones are needed, whether a separate repository for archive and schedule changes will be required.
- cluster composition: number of nodes and their roles, whether dedicated nodes are allocated for data ingestion;
- current snapshot repository: file system, object storage, or another option — and what else uses it;
- data volume and daily growth: snapshot frequency and retention depth depend on this;
- retention period: how many generations and over what period must be kept available for restore;
- allowed data loss window: how many minutes of work can be lost during a failure — this sets the interval between snapshots;
- environment constraints: channel throughput to storage, peak load hours, maintenance windows;
- what is already configured: whether restore attempts have been made and where data was deployed.
With these inputs, we return with an estimate and a work plan: order of steps, repository placement, snapshot schedule, separate readability checks, and a trial restore in an isolated environment. Describe the task — we will clarify the details and propose options for timelines and scope.