Logs overflow disks: what breaks in archival log storage
While the service is running, logs are written to the hot tier — a fast volume where searching fresh records takes seconds. The volume grows linearly with load, while the disk expands in steps, and sooner or later free space runs out.
Then it's not the storage system that kicks in, but the on-call engineer: they manually delete old directories to keep the service from going down.
The cost of such a solution is predictable. Disk overflow with logs brings down not only collection but also neighboring processes on the same volume: the database stops writing, the queue accumulates messages.
Rotation and deletion of logs configured "by eye" cut data by age, not by value — the very period in which the cause of an incident is later sought gets axed.
The bottleneck of storage is usually not in the volume, but in the fact that all data lies in one class. Fresh records are needed for analysis here and now, while the archival part is not opened for months — and still occupies the fast volume. When it runs out, the choice is made in favor of deletion rather than moving to a cold class.
Hence the main risk — loss of event history. Incident analysis, access audit, reporting to a regulator all come down to one question: are there records for the required period, intact and with verifiable timestamps. If they are not there, there is nothing left to reconstruct the picture.
We explain how this is addressed: by separating hot and archival tiers and a migration policy instead of manual cleanup after an accident.
What managed rotation provides: search speed and failure classes
Rotation without a cold tier leaves all history in hot storage: indexes grow, search over them slows down, and the log retention period expires faster than disk space.
Managed rotation moves old data to archival classes of object storage, e.g., S3, and keeps only the working horizon in the hot tier — usually from 30 to 90 days.
Search speed: how incident analysis time changes
Incident analysis proceeds in two modes: fresh events are searched in the hot tier, historical context is pulled from the archive.
As long as the hot index is limited to the working horizon, search responds in seconds and does not degrade as data accumulates, so incident analysis time stops depending on the age of the system.
Restoring the required period from the archive takes from several minutes to an hour — this is a predetermined value that is included in the regulations.
Log retention period is set by policy, not by disk size, and one can drill down into historical events without manual backup uploads.
Hot tier volume and failure classes that go away
The hot tier is reduced by 5–10 times: it retains indexes for the working horizon, the rest goes to the archive.
Along with the volume, specific failure classes go away — disk overflow and write stoppage, nightly index merges that eat up throughput, node crashes under peak load, and recovery after a failure stretched over hours. Recovery time drops to minutes because only the hot part is brought up, not the entire history.
Warm and cold tiers: how access to logs changes
Tiers differ not by the type of medium, but by what remains available for search. Active analysis goes over the last hours and days — there a hot tier with indexes and seconds-level access latency is needed.
Everything older goes to a warm or cold tier, and the choice between them is determined by access frequency, not data volume.
Hot tier: instant access for active analysis
Here lie logs that are returned to when troubleshooting a failure and testing hypotheses: data is stored in full, together with indexes, so field search and correlation of events between services work immediately.
The price for this is storage density, so the horizon is limited to 7–30 days. After that, history is moved to cheaper layers, and by lifecycle policy, without manual uploads.
Cold layer: trade-off between latency and volume
The warm tier is intermediate: data is compressed and stored in object storage, but indexes are preserved, so a typical query returns results in minutes.
The deep cold tier stores packed blocks without a search index — access latency grows to hours, but retention depth is measured in years, and archive volume stops affecting the size of the hot layer.
| Option | When suitable | Limitations |
|---|---|---|
| Hot tier | Failure analysis, live monitoring dashboards, last 7–30 days | Most expensive per unit volume, retention horizon limited |
| Warm tier | Queries to history for a quarter, audit, recurring failures | Access latency — minutes, some fields remain outside the index |
| Cold tier | Compliance, retrospective analysis, storage for a year or longer | Access latency — hours, requires export before analysis |
Project stages: from stream analysis to handover of regulations
The order of work is the same for any source — from application and load balancer streams to scheduler jobs. First we record what is being written today and at what frequency, then we build a data movement scheme to the cold layer, and only after that do we enable policies. Below is the sequence by which the project proceeds.
Stream audit and volume estimation
We count based on facts, not "approximately": how many events per second arrive at peak, what share of records is needed for incident analysis, and what share only for regulatory requirements.
At this stage it usually turns out that a noticeable part of the volume is debug records that no one needs.
- Stream audit: inventory of sources, record format, daily and peak volume, retention periods for each data type.
- Data export scheme: what remains in the hot layer, what goes to S3-compatible storage, in what batches and through which intermediate buffers.
- Policy configuration: lifecycle rules by prefixes and tags, transition between storage classes, compression and object partitioning, deletion upon expiration.
Integrity verification and export acceptance
- Control checks: comparison of record counts and checksums between source and archive, selective reading for past periods, measurement of restore time.
- Trial rollback: we bring up the archive for a specific date and make sure the data is read by the same tool as usual.
- Handover of regulations: stream descriptions, policy texts, export and analysis instructions, control points, and failure response procedures.
After acceptance, the regulation lives without us: adding a new source or changing the retention period is an edit of one policy, not a rebuild of the entire scheme.
What remains in hand: flow diagrams, access policies, regulations
Based on the results of the work, the customer receives not an archive of files, but a reproducible storage tier: configurations, diagrams, and regulations by which archival log storage is recreated or handed over to another team without archaeology.
- Data flow diagram — from sources through buffer to cold class, indicating where logs are compressed, encrypted, and distributed by prefixes.
- Lifecycle rules: after how many days an object leaves the hot class, how many versions remain per tag, and what is permanently deleted.
- Configurations of exporters and rotation in the repository — with explanations of deviations from the standard template.
- Instructions for on-call shifts: what to do in case of export lag, buffer overflow, and connection loss with storage.
- Recovery regulation: in what time frame a sample for an arbitrary period is assembled, who approves the request, and how the result is recorded.
- Placement and quota scheme: involved regions, copies, procedure for verifying integrity of archival objects.
Access policies and role separation
Access is described by a matrix, not by "verbal" roles: who writes to the archive, who reads the sample, and who changes retention rules.
- Role separation: service accounts only write, engineers read on request, changing retention rules — through approval.
- Prefix-level access policies: prohibition of anonymous reading, separate log of archive access requests.
- Procedure for issuing temporary access for incident analysis: key lifetime and revocation moment.
Case study: how the hot tier was reduced and incident analysis accelerated
The engineers had a tier of several dozen services where events and logs were almost entirely in hot storage.
While volumes were moderate, this did not bother anyone; when the stream grew, the hot layer became the main consumer of disk space, and the time to search events during incident analysis stretched to 15–20 minutes per query.
What was before implementation and what changed
Before the work, the hot tier held 90 days, archival exports were launched manually on a schedule, and some runs ended in discrepancies: objects either did not reach the cold layer or were duplicated.
Export integrity was checked by eye and after deleting the original, so "lost" days of logs were found only when they were urgently needed.
We separated the layers by lifetime: the hot one retained 14 days for operational work, the rest goes by a lifecycle rule to the cold storage class Glacier.
Each export generates a manifest with checksums and a record counter, source and archive reconciliation is automatic, and a discrepancy blocks deletion of source data — lossless export has become a verifiable property, not a hope.
The reduction of hot storage gave not only space: the search for the needed event on an incident accelerated from 15–20 minutes to 20–40 seconds, because the query goes over a pre-built index of the affected period, not the entire array.
The share of exports that ended in discrepancies dropped from 7–8% to zero, and incident analysis now does not depend on manual guesses of the on-call engineer.
What to do if part of the logs is already irretrievably lost?
Log loss can rarely be completely undone: if rotation on the host itself has already overwritten the file, recovering data from nowhere is impossible.
But this does not mean that history breaks off at the date the work began — almost always there remain several islands of records from which a coherent picture of events can be assembled.
First, we inventory sources that live longer than application logs: disk snapshots, backups of configurations and databases, archives of load balancers and queues, local copies on nodes with longer rotation.
Each has its own recovery window, and often it is wider than it seems: kernel system log, metrics, and traces lie separately and allow restoring the chronology even when text records are lost.
Next — re-collection. While there is no single storage, we deploy agents on hosts and capture what has not yet gone beyond the rotation boundary; for already lost events, we restore context from metrics, configuration changes, and history in the version control system.
Mixing old archives with a new stream is not allowed: different timestamp precision produces duplicates, so we put normalization and deduplication at the input.
To prevent the gap from recurring, we write the hot stream to operational storage, and by lifecycle rules move it to the cold class — the archive grows slowly but remains readable.
Once a quarter, we bring up an arbitrary date range from the cold layer and check that it opens: without such a check, backups turn into a set of files that no one is sure about.
Integrity, access, and recovery: technical doubts
An archive is usually accepted based on facts: "files are there, size matches." Real doubts begin later — what confirms the immutability of an object, who and on what basis reads the content, how quickly data returns from the deep layer, and which of these meets the requirements of auditors.
How is it confirmed that exported logs have not changed?
Each object is written with a checksum that is verified upon reading and scheduled audit. For the retention period, immutability mode is enabled: the storage itself rejects overwrite and deletion — including from an administrator account.
Who gets access to the archive?
Access is granted to roles, not people: service accounts write, engineers and auditors read — each within their prefix boundaries. Keys do not leave the secret store, and access requests are recorded in a log that lives longer than the logs themselves.
How is data returned from the deep layer?
Reading from the cold class is not instantaneous: restoration is initiated by a request, the copy is brought up to the warm layer for a specified period, then it can be downloaded or sent to analytics. The level is chosen for the task — cheaper and longer or faster and more expensive.
What to present to auditors and how not to pay for excess?
The retention period is set by policy per log type, not "forever": upon expiration, objects are deleted automatically, metadata and integrity reports are exported for audit.
The cost of ownership is determined by the volume and lifetime of objects — the lifecycle moves them to a cheap class, and the registry stops growing.
Let's discuss the task: what to clarify before starting
To come back with an estimate and a work plan, rather than general considerations, we need a minimum of initial data about your infrastructure.
Collecting it takes the customer an hour and a half, then we calculate volumes, select storage classes, and prepare a work plan with a trial export.
- Sources: which systems and services write logs, in what format and at what peak intensity.
- Data volume estimate: daily and monthly growth, share of records read in the first days, and share of cold remainder.
- Retention periods: what must lie under fast access, what goes to the archival class, and for how many years.
- Current scheme: where data is stored now, how rotation works, what is cleaned manually, and what has already been lost.
- Access requirements: who needs access and how quickly, whether immutability of records and separation of rights are needed.
- Constraints: own servers or cloud, security rules, prohibition on exporting data outside the perimeter, export windows.
- Acceptance: who confirms the result, by what date the archive is needed, and whether there are associated events — audit, migration, contractor change.
Describe the task and send what you already have — we will come back with a data volume estimate, work plan, and acceptance criteria.
We do a trial export of one array before the start: this way decisions on storage classes and lifecycle policies are based on measurements, not assumptions.