Elasticsearch ILM is not configured: what breaks in the cluster
A cluster launched for logs and metrics lives without rules: an index per day or per service, a template is defined, but what to do with accumulated data afterward is not described.
Indexes grow, shards multiply, every day adds segments, and the hot tier stops being hot — it holds both fresh data and the entire archive.
The first noticeable symptom is storage overflow on hot-tier nodes. When disk watermarks trigger, Elasticsearch moves indexes to read-only mode, allocation of new shards stops, replicas stop being allocated — the cluster goes yellow, then red.
Then degradation cascades: segments are not merged, more and more segments are scanned per search request, and response time grows.
Search and indexing interfere with each other because they share the same threads, disk I/O, and cache. Indexing evicts queries from the cache, the search queue overflows, clients get timeout failures — and indexing itself cannot flush buffers in time.
Diagnostics become complicated: the cluster is technically alive, nodes look acceptable, but users no longer get responses.
The root cause is not hardware but the absence of a defined index lifecycle: no one moves old data to warm and cold tiers, merges segments, and deletes data past retention. Let's break down how ILM in Elasticsearch closes this — from index rollover to deletion.
The effect of a managed index lifecycle on a production cluster
After ILM implementation, the cluster stops being a homogeneous dump of indexes. Fresh data is written to the hot tier with fast disks; after the rollover phase, Elasticsearch moves it to warm, then cold — with dense segment packing.
Search load is redistributed: the hot tier serves only the current window, and queries on fresh data no longer compete with months of history.
Index rollover is tied to size or document count, not to the calendar. The write index is always bounded, the number of segments is predictable, and search is not spread across hundreds of small shards.
This is what gives stable response time: it depends on the size of the result set, not on how many records have accumulated since the last manual cleanup.
Automatic index deletion closes the last source of growth. The retention policy defines a period or number of generations; the rest goes away without human involvement.
Stored volume becomes predictable — the growth chart shows when capacity will be needed, not when the disk will run out.
A whole class of failures goes away: disk overflow, shards going read-only, nightly manual cleanups, and emergency data migration under load. Rollback to the previous release remains possible — old index generations live until the retention period expires.
Index management schemes: comparison of options and limitations
The index management scheme defines which nodes hold fresh data, when data migration to the cold tier starts, and what happens to the index afterward. Below are four schemes that are actually chosen in practice, with their limitations.
| Scheme | When it fits | Limitations |
|---|---|---|
| Single tier | Small volume, search only across fresh data, requests deeper than a month are rare | Node profile does not get cheaper as indexes age; as volume grows, the footprint grows; there is no archive as a separate entity |
| Two tiers: hot and warm | Data older than a few days is needed only for reports | No deletion stage — the registry grows until it is cleaned manually; the warm footprint eventually receives the same load as hot |
| Three tiers: hot, warm, cold with deletion | Long retention with rare queries, retention periods are required | An error in the deletion condition is irreversible without index snapshots; data migration must keep up with ingestion, otherwise the cold tier overflows |
| Manual administration | Irregular data flow, one-off tasks | No predictability: an index is moved when someone remembers it, retention periods are silently violated |
The three-tier scheme covers the most requirements, but its limitations are removed only by index snapshots and explicit control over migration. If there are no snapshots, age-based deletion is the most dangerous setting in the entire configuration.
Implementation order: from assessment to handover of procedures
The order of work is the same for a cluster with several nodes and for an installation with tens of terabytes: first we record facts, then we design, and only then we change settings.
Reordering the steps is the main reason lifecycle policies start deleting needed data or fail to trigger at all.
- Cluster audit. We capture the actual picture: list of indexes, growth rate, spread of shard sizes, retention periods, current templates, and unclosed aliases. Without these numbers, any hot-warm-cold tier design is done at random.
- Tier design. We determine which data stays on the hot tier, what moves to warm, what to cold, and when it is deleted. Separately, we record behavior during node failure — the policy must not depend on a single instance.
- Assembling policies and index templates. We write the transition rules themselves and bind them to templates so new indexes pick up the lifecycle automatically, not through manual intervention by the on-call engineer.
- Testing on a copy. We run the policies on a test environment with real data structure and accelerated transition times. This way, errors in transition conditions are visible before production, not a month later on the production cluster.
- Rollout to the working environment. We enable policies in stages, by index groups, monitoring transition state and with the ability to roll back a rule before it affects old data.
- Handover of procedures. We hand over the description of tiers, template schemes, the procedure for adding new indexes, and instructions on what to do when a transition deviates — so future changes can be made without us.
The key to the result is the fourth step: testing on a copy takes more time than configuration, but it is what separates a working policy from a set of plausible conditions.
Deliverable set: policies, templates, procedures
After ILM implementation, the team is left not with a “configured cluster” but with a reproducible set of artifacts: from it, the configuration can be raised on a new environment and handed over for support without us.
Everything is stored in a repository and applied through the management interface, so no manual edits remain in the interface.
- Index management policies — transitions between hot-warm-cold tiers, thresholds by age and shard size, retention period and deletion; separate policies for each data class: application logs, audit, metrics.
- Index templates and component templates — mappings, number of shards and replicas, binding to a policy by name mask, so a new index receives settings automatically.
- Tier scheme — which nodes are assigned to which roles, where the hot layer ends, how cold storage is arranged, and what space reserve is set aside for recovery.
- Access policies — roles for log collectors, analysts, and the on-call shift, separate permissions for deleting indexes and working with snapshots.
- Snapshots and recovery — schedule, snapshot lifetime, and a tested scenario for rolling an index back to a required date.
- Procedures and instructions for the team — how to create a new index, how to change the retention period, what to do when the disk fills up, and how to read policy errors.
- Monitoring — dashboards and alerts for transition stages, failed steps, and data volume growth.
Case study: how the cluster was unloaded and stored volume was reduced
The search cluster for application logs and security events ran on a single node profile: writes, fresh reads, and archive were mixed together. Queries on fresh data responded quickly, but quarterly result sets timed out, and the disk filled up faster than the event stream grew.
What it was before
The analysis showed: Elasticsearch indexes were written daily without phase transitions, all shards lived on hot nodes, and a replica took as much space as the primary shard.
There was no index rollover: old ones were deleted manually by the on-call engineer, and segment merging was triggered only after an incident when search degraded.
Search time over 90 days — up to 8 seconds, share of data on the cold tier — zero, manual interventions — 9–12 per month.
What we did
We assembled an ILM policy: rollover by shard size and index age, then phases — hot for writes and fresh reads, warm for data a week old, cold for archive.
In the warm phase, we removed replicas and enabled segment merging down to one shard per index; in the cold phase, we added strong compression. Deletion was tied to age: data older than the retention window goes away entirely, without on-call commands.
Metrics after
- Search time over 90 days: 8 seconds → 600–800 ms.
- Share of data on the cold tier: 0 → 71% of volume.
- Stored volume: compression plus age-based deletion reduced it threefold, registry growth stopped.
- On-call manual interventions: 9–12 → 0 per month.
The class of failures that went away with this — disk overflow and timeouts on historical result sets. Rollback remained possible: snapshots are taken on a schedule, and the previous index is available until the retention window expires.
How does Elasticsearch ILM differ from manual index deletion?
ILM is a policy that describes an index lifecycle in phases: while writing continues, the index lives in the hot phase; by size or age threshold, the policy rolls it over using the write alias, removes some replicas, moves shards to cold nodes, triggers segment merging, and deletes the index on the last transition.
The cluster makes the decisions, while the engineer defines the rule once and then only watches metrics.
Manual rotation works differently: the index is created with a date in its name, and cleanup is done by a scheduled script or manually by the on-call engineer.
While everything goes according to plan, the difference is barely visible, but it shows up during a failure — the job did not run, the template was not applied, and the old days remained on the nodes. The disk fills up, the number of shards grows, and writes hit fill thresholds.
The cost of an error in manual deletion is higher: a command by mask easily hits an index currently being written to, and rollback requires a snapshot and means losing a window of data.
In automation, deletion is tied to conditions — the index goes away only after transitioning to the delete phase and only if the policy allows it. Predictability rests on the write alias, not on an agreement in the on-call engineer's head.
Manual rotation lives as long as its author: the engineer leaves — the script becomes outdated, exceptions accumulate. The policy, however, sits in the template next to the mapping and is inherited by new indexes without additional actions.
Point manual deletion is justified — a one-time archive cleanup, non-standard retention requirements, incident investigation — but as a permanent mode it loses: rules move from the on-call engineer's head into configuration, and registry volume stops depending on who is on shift today.
Common technical questions about lifecycle policies
A lifecycle policy does not enable itself: it is assigned to an index manually or via a template. Phases are calculated from the index age, not from the moment of assignment, so an old set with a long retention period can go straight to deletion.
Before enabling, we run the phase calculation on a copy and check which step each index will land on.
Rollback depends on what has already triggered. A transition between tiers is reversible: the index is returned to the hot tier by a regular segment migration. Deletion is irreversible — we enable this phase last and only when snapshots are present.
Can a policy be applied to old indexes?
Yes — by assigning it to the existing set or via a template for new ones. The policy does not recalculate history from scratch, so for old data we separately check whether the transition to deletion will trigger immediately after connection.
What to do if deletion has already triggered?
Restore the index from the latest snapshot and assign a corrected policy before returning it to the stream. If there are no snapshots, the data cannot be recovered — this class of failures is closed only by regular snapshots and recovery testing.
How much slower is search on the cold tier?
Noticeably slower, and this is the price for volume: we keep the cold tier on large-capacity drives with lower access speed. We move there only what is rarely requested — based on access statistics, not blindly by age.
Are tiers needed on a small cluster?
On two or three nodes, separation gives little: segment migrations consume resources, and the storage gain is small. One policy with index size control, snapshots, and deletion as the last phase is enough.
How to choose retention periods?
From daily growth and data requirements: calculate volume per day, multiply by the period, and compare with available space. We set periods by phase — the hot part is shorter, warm and cold are longer, deletion at the end.
Let's discuss the task: how to configure the lifecycle for your data stream
To make lifecycle rules work on your real stream, not on a test index, we need a short input from the customer: write profile by index, retention horizon, and recovery requirements.
With this set, we calculate the load on the hot tier, the number of migrations between tiers, and the volume that will go to deletion — and arrive at an estimate and work plan.
What is worth sending so the estimate is substantive:
- map of indexes and templates: which services write, at what frequency, average and peak document size;
- retention requirements: how much we keep in the hot, warm, and cold tiers and what we delete permanently;
- acceptable data loss window and recovery time after node failure;
- current constraints: number of nodes, disk types, free space, current Elasticsearch branch;
- maintenance windows: when indexes can be moved, segments merged, and replication changed;
- access points and cluster owner — from whom we receive rights to change policies;
- incident history: what has already broken and which indexes grew faster than forecast.
This set starts the lifecycle configuration: policies, thresholds for transitions between tiers, deletion periods, and rollback rules.
After approval, we test everything on a data copy, put it into operation, and transfer knowledge — procedures, monitoring dashboards, analysis of typical failures — so your team can edit policies itself. Cluster support remains on our side until you decide otherwise.
Describe the task — we will discuss it and come back with an estimate and plan.