Configuring Log Rotation and Retention Policies: DevOps Guide

Why logs fill the disk: where log rotation breaks

Uncontrolled growth of log files looks harmless as long as there is free space: the service opens a file for append and holds the descriptor, while the size grows with every request, error, and log entry.

As long as rotation is configured, the file closes by size or by calendar — without it, the only safeguard left is free space on the partition.

A typical story: verbose logging was enabled during incident investigation, someone forgot to return the level back, and within a day the disk is full.

When disk space runs low, it is not the service that degrades first but writing: the application gets a no-space error from the file system, log buffers stop flushing, some messages are lost, and an unhandled exception can crash the process itself.

A log write failure rarely looks like a crash — more often it is "the service responds, but logs nothing," and incident analysis has to be conducted blindly. A separate trap is deleting a file manually: space is not reclaimed while the process holds the open descriptor.

In containers the picture is the same, except the logging driver creates a file for each container, and without limits on size and file count it grows along with the load.

Baseline safeguard — logrotate: rotation by size and calendar, limit on the number of stored archives, compression, and correct descriptor reopening.

Next, we break down how this is handled at the system and service level: from retention policies to disk usage monitoring and alerts about logs overflowing the disk.

What managed rotation and log retention provide

Managed rotation is not "enable logrotate and forget it" but a set of rules: under what conditions a file is closed and compressed, how many copies live locally, what goes to long-term storage, and by what deadline it is deleted.

The rules are formulated in sizes and times, not "when there is a lot," so the volume of logs in the log registry stops being a surprise, and the log retention policy becomes verifiable rather than declarative.

What becomes visible in metrics after switching to a managed scheme:

• Disk usage on the node fluctuates within a known corridor: size-based rotation triggers before the partition hits its limit, and the class of failures "service crashed due to a full disk" disappears from statistics.
• Compressed archives take several times less than the original volume, so history depth grows without storage growth.
• Incident analysis time is reduced: the needed interval lies in one file with a clear name and period label, rather than being spread across dozens of unrotated logs.
• Searching archives remains fast because file boundaries match the boundaries of days and the service's time zones.
• Deletion follows the retention period, not the remaining space, — cleanup stops being an emergency operation under load.
• Settings live in the configuration management system, so a new node gets the same policy without manual editing.

The result is a predictable log registry volume and service stability: disk space stops being a variable affecting availability, and the on-call engineer during incident analysis immediately knows where to look for data. Next, we will break down which parameters set these boundaries and how to verify them.

Comparison of log rotation approaches: size, time, shipping

Predictability of rotation is a choice between a predictable file size and a predictable cut-off moment. Size-based rotation triggers when the file crosses a given threshold, so disk space will not run out even during a load spike.

Scheduled rotation is tied to minutes or hours, and then the file for the needed period lies in one piece — it is convenient for correlating with a new version deployment or incident analysis.

Both schemes cover only the first half of the task — freeing space. The second half is what to do with the rotated file: compress in place, store in an archive, or immediately ship to external log storage, where search goes across all nodes at once rather than one by one.

In practice, a combination is used: size-based cut-off as protection against overflow, scheduled cut-off as a reference point for analysis, compression of old logs, and export to external storage with a retention policy.

Then log archiving stops being a manual operation, and the retention period is set by a rule rather than by the on-call engineer's memory.

Implementation procedure: from audit to handover of the runbook

"Eyeballed" rotation gives one of two outcomes: the disk overflows at the worst possible moment, or incident logs are deleted before anyone gets to them. So we proceed in stages — from inventorying sources to a runbook for the on-call shift.

1. Audit of log sources. Record who writes to disk: path, intensity, behavior during spikes, who holds the file open. Separately analyze containers and the system journal — they have their own retention logic.
2. Rotation scheme and retention windows. For each group, choose a size threshold or interval, number of stored copies, and compression. Separate the classes: keep audit and access logs longer, debug logs for a few days.
3. Rule configuration. Configurations in logrotate format by group: archive compression, post-rotation actions for descriptor reopening, copytruncate where the process does not respond to a signal, directory permissions.
4. Testing under load. Run a stream above peak: the rule triggers at the threshold, lines are not lost at file boundaries, compression does not block writing, old copies are actually deleted.
5. Observability. Alerts for free space, age of the last file, and rotation errors — otherwise a failure will be noticed by an unavailable service, not by a metric.
6. Handover of the runbook. Document and walkthrough with the on-call shift: where logs are stored, what to do on overflow, how to safely free space, and how to roll back a rule.

Implementation artifacts: configurations, schemes, runbooks

Implementation is not complete when rotation works, but when the customer can change, verify, and restore it after a failure without us.

Therefore we hand over a set: version-controlled configurations, a retention scheme with periods, runbooks for on-call engineers, and an access model.

• Rotation configuration for each source: trigger condition (by size or calendar), number of stored copies, compression, deferred archiving, and action after the process reopens the file.
• Retention runbook: table "log class — hot retention period — archive period — medium — who may delete early."
• Source map: list of services and nodes, what is written to a file, what goes to the system journal, where it is duplicated and why.
• Documentation for on-call engineers: what to do when the disk fills, when rotation fails, when verbosity must be urgently increased — with ready-made commands and expected output.
• Access rights to logs: role matrix — direct read, access on request, where personal data is masked, who may change retention periods and configuration.
• Checks and alerts: trigger if rotation did not run, the archive was not exported, or free space is below the threshold, — with routing to a specific on-call engineer.
• Rollback instructions: how to return the previous configuration and restore deleted copies from the archive before the retention period expires.
• Before-and-after measurements: log volume on disk, share of the node they occupy, number of incidents due to overflow — a baseline for comparison after changes.

Case: stopped losing logs under peak load

The service wrote logs to files on each node, and rotation was configured with one line: 100 MB each, keep everything.

At peak, the application continued writing to an already rotated file, and some records went nowhere — "holes" appeared exactly in the minutes when incident analysis was needed most.

Finding the needed event took 20–30 minutes: the engineer went around eight nodes and searched archives manually.

What was changed

Policies were separated by class: application, system journal, and audit got separate rules instead of a common config. Rotation is performed by size and by time, and the daemon receives a signal — the file is not reopened by the application, so no record is lost at the switch.

The rules were described in logrotate, delivery to centralized storage was moved to a separate agent with an on-disk buffer: if storage is unavailable, records accumulate locally rather than disappear. Retention depth was set separately for the hot tier and archive, and old files are compressed.

Metrics before and after

The share of records lost at peak dropped from 3–7% to zero — application counters were compared with the number of lines in storage. Searching for one event shrank from 20–30 minutes to 10–20 seconds by index, without walking nodes and manually parsing archives.

The hot tier decreased by about three times with the same archive depth: the log registry stopped growing linearly, and old data is evicted by policy.

How does size-based rotation differ from time-based rotation?

Size-based rotation triggers when a file outgrows a given threshold: disk space consumption on the partition remains predictable, but the switch moment depends on load — on a quiet day the file lives for a day, during a peak it closes within minutes.

Time-based rotation is tied to a schedule: the file closes by calendar, so the set of files for a period is always the same, and it is convenient to build queries from them. The difference becomes noticeable precisely with an uneven load profile.

To choose a mode, look at peak write frequency, not the daily average.

If the spread between peak and lull is more than an order of magnitude, pure time-based rotation leaves a window in which the partition overflows — then the schedule is set together with a size threshold so the file closes by either of the two conditions.

Retention is calculated from the rate at which files appear: with time-based rotation their number per day is predictable, with size-based rotation it is variable, and a deletion policy based on file count gives different history depth.

Therefore, for frequent tags it is more convenient to calculate depth in days, and limit volume with compression and a total directory cap.

Common technical questions about rotation and log retention

Doubts around rotation are usually not about how to enable it, but about whether we will break a live service and how we will prove the policy works. Below are answers to what most often comes up when handing infrastructure over to another team.

Retention periods, integrity, and incident analysis

Can logs be rotated without restarting the service?

Yes. If the process can reopen the descriptor on a signal — rotation proceeds without breaking connections, the file changes name, and writing continues to a new one.

If it cannot and holds the file open, we use copytruncate: some lines at the moment of copying may end up in both files or be lost, so we choose the window for peak load and test it on staging.

How can I make sure the policy is actually applied?

A configuration is not yet a result. We force rotation on a test environment, compare the number of lines before and after, check whether empty and "stuck" files remain, and set alerts for directory growth and rotation failure. Separately, we verify that compression and deletion do not conflict with active writing.

What does log audit provide and how should the archive be stored?

Audit answers the questions "who changed the policy" and "can the records be trusted." For this, we export the archive to immutable storage with integrity checks, and manage configuration edits through versioning with a change history.

Then restoring a specific time window takes minutes, and there is evidence to confirm data authenticity.

Let's discuss the task and come back with a log rotation plan

The first conversation is about your infrastructure, not our runbooks. It is important for us to understand which services write logs, where they are stored, how much they occupy, and what already hurts: rotation is not configured, the disk overflows overnight.

Separately, we discuss retention periods and what defines them — audit, incident analysis, or security rules.

From this conversation, an implementation plan for policies is assembled — tailored to your environments, not a template:

• What we need from you: list of servers and environments where logs live, current retention configurations and windows, when it is acceptable to change the production environment.
• What we review: volumes and growth rate of logs, who reads them and how often, which files are needed for incident analysis and which are ballast.
• Rotation scheme: by size, by time, or by both thresholds, with compression and breakdown by service.
• Retention policy: how much we keep hot, what goes to archive, what is automatically removed by lifecycle rule.
• Work estimate: list of changes for each environment, effort, rollout order, and rollback step.
• Policy implementation: first non-production environments and testing on a real stream, then production — with monitoring that free space grows and the needed logs remain accessible.

The next step is yours: describe the task in two or three paragraphs or send the current configurations. We will come back with a work estimate and an implementation plan broken down by stages.