Why log aggregation breaks as load grows
While there are about a dozen nodes, logs live in files on disks, and incidents are investigated via ssh and grep.
When servers become more numerous and traffic grows, manual traversal stops working: to reconstruct the picture of a failure, an engineer logs into each node in turn, remembers log paths, and compares timestamps. Investigating one incident takes hours, and all that time the cause remains unclear.
The first point of failure is the agent on the node itself.
At peak load, the record stream exceeds send throughput, and if the agent's buffer overflows, some lines are silently dropped: the storage receives an incomplete picture, and worst of all, the error spikes for which logs are collected are lost.
The tail of the journal looks plausible, and the absence of records is noticed only during investigation.
The second is rotation faster than shipping. The local file is truncated by size or retention period before it can be picked up, and records disappear irrevocably.
At the same time, volume grows avalanche-like: repeating errors in a loop, multi-line stack traces, debug level in production — all this inflates the stream, makes storage more expensive, and slows search because what is needed drowns in noise.
We'll break down how this is addressed: where limits are set on the agent side, how a queue with backpressure is built, and why shipping must outpace rotation.
Measurable effect of a unified event collection system
A unified event collection system is measured not by the number of connected services, but by four indicators: incident investigation time, share of lost records, number of missed alerts, and depth of retrospect.
While events remain in local files with size-based rotation, investigation begins with manually traversing machines, and some records are already overwritten by the time of investigation.
End-to-end search across all services
The request identifier is propagated through services and becomes a mandatory field of every record — a single search point answers the question "what happened with this request" with one filter, without switching between consoles.
Collection agents buffer records to disk and send them after service restart, so the share of lost records decreases from a few percent to tenths of a percent within the retention window.
Storage is built on indexes with a configurable lifetime — for example, on OpenSearch: hot data is kept for fast queries, older data is evicted into compressed segments, and storage volume stops growing linearly with traffic.
Reducing incident investigation time
When an error signal, metric, and service records are in one interface, the on-call engineer moves from "the service is responding with errors" to the specific request and its call chain in minutes, not hours.
Alert rules are described by event fields rather than log lines, so the number of missed alerts drops: a check is not lost when the wording of the error text changes.
A retrospective spanning several weeks shows whether the failure was isolated or accumulated — and this is confirmed by the same data, without exports from host archives.
Log aggregation options: agent, queue, stream
The delivery scheme determines what happens to a record if the receiver is unavailable. Direct collection by an agent, buffering through a queue, and stream processing on the fly differ in the point at which a log is lost or duplicated.
| Option | When it fits | Limitations |
|---|---|---|
| Direct collection by agent | Dozens of sources, stable receiver, latency of seconds is not critical | No accumulation: if the receiver is down, the agent loses records or eats up disk |
| Buffering through a queue | Many sources, load peaks, frequent receiver restarts | Requires a separate broker cluster: maintenance, queue depth, duplicates |
| Stream processing on the fly | Parsing, enrichment, and filtering are needed before writing to storage | Harder to debug: a parsing error corrupts the stream, delivery retry is needed |
When direct collection is enough
If there are dozens of sources and the receiver responds predictably, an agent on each node writes directly — fewer moving parts and places where the route breaks. The price of simplicity: there is no accumulation, and during a long receiver outage records are lost. This scheme is chosen when logs are needed for observability rather than incident investigation.
When buffering and redelivery are needed
A queue decouples the source and receiver: the agent hands off a record and forgets about it, the broker holds it until receipt is acknowledged. Redelivery after a failure becomes a standard mode, but duplicates appear — storage needs a key that collapses the record.
Stream processing adds parsing and enrichment before writing, and a hybrid scheme connects an agent at the source, a queue as a buffer, and a stream processor before storage.
Implementation order: from stream audit to handover
The work order is built in waves: each stage ends with a verifiable result — a scheme, collector configuration, or delivery completeness report.
Before the log aggregation pipeline becomes the primary incident investigation tool, the customer sees the state of the system and knows where the bottlenecks are.
- Audit of event sources. We traverse services, schedulers, and network nodes, record formats, daily and peak volume, and the owner of each source.
- Stream and route scheme. We determine where the collector sits, what buffers before sending, which record types go to operational storage and which to archive, and for how long.
- Collector configuration. Parsing structured records into fields, unified service and environment labels, limits on bursts — one noisy source does not crowd out the rest.
- Delivery completeness check. We reconcile counters at the input and in storage, run test bursts and connection drops, and record behavior when the receiver is unavailable.
- Access control. Roles by teams and services, masking of personal data at input, and an audit log of storage queries.
- Handover and team training. Naming rules, search query and dashboard templates, on-call regulations, and review using real incidents.
What we clarify during the audit
- Which questions the team asks logs most often and how long it is willing to wait for an answer.
- The required storage depth and fields that must not be stored in plain text.
- Which collectors are already deployed and what can be reused from them instead of duplicated.
Configurations, stream schemes, and on-call regulations
Log aggregation rests not on the on-call engineer's intuition but on artifacts — files and documents that can be reread, rolled back, and handed to a colleague.
After implementation, this entire set remains in the customer team's repository, not in correspondence with the contractor: a new engineer reads the same configurations as the system's author.
Configurations and stream schemes
- Collector configurations — reception, parsing, and enrichment of records, unified for all nodes: identical fields on staging and in prod, without manual edits on individual machines.
- Stream and route schemes — what goes where, by what rules streams of different services, environments, and teams are separated.
- Retention policies — record lifetimes by type, what is cleaned automatically, what goes to archive and how it is restored from there.
- Record access policies — roles, who reads which streams, masking of sensitive fields before writing to storage.
- Monitoring dashboards and alert rules — ready-made queries, thresholds, who receives the signal and with what delay.
- Instructions for connecting a new service — a step-by-step path from request to the first record in the shared storage.
Regulations and instructions for on-call engineers
- On-call shift regulations: shift handover, response time, areas of responsibility, escalation procedure.
- Actions for typical failures — disk overflow, queue lag, storage unavailability.
- Post-configuration-change checklist: what to check and in what order so as not to lose the stream.
- Rules for incident investigation and updating documentation based on its results.
Case study: how rebuilding collection reduced incident investigation time
The customer is a platform made up of several dozen services. Incidents were investigated by the on-call shift, and almost every investigation ran not into the complexity of the cause but into missing data: the needed records no longer existed by the time of investigation.
What it was before: scattered files on nodes
Logs were written locally on each node, rotation was by file size, and at peak load — exactly during an outage — fresh records displaced the ones needed for investigation.
The engineer logged into nodes one by one and reconciled time manually: clock skew reached several minutes, so events were arranged in the wrong order.
About 12% of records were lost during rotation and disk overflow, incident search took from 40 to 90 minutes, and the picture was assembled from six sources instead of twenty.
What it became: unified search and minutes instead of hours
The agent on the node reads application and system logs, sets service, environment, and release version labels, and sends them to a shared receiver.
Time is synchronized, the record structure is brought to a unified schema — events from different services are matched directly, without manual stitching.
Incident search takes 3–5 minutes instead of an hour, lost records are 0.2%, and the completeness of the picture has grown to all services, including background jobs and queues.
False alerts have decreased fourfold: the rule is built on the rate of error growth, not on a single record. Investigation and dashboards run on top of a single stream in Grafana, and one query is not assembled twice.
Incident investigation no longer depends on whether the engineer managed to log into the node before rotation.
What to do if events are already collected in different places?
Events collected in different places are not a reason to rewrite all collection.
Usually agents are already installed on some nodes, for example Fluent Bit, with configured parsers, some services write to files, and the current storage holds history on which dashboards and alerts are built. First we create a map of contours and decide for each layer: reuse or replace.
| Layer | Reuse | Replace |
|---|---|---|
| Collection on nodes | existing parsers and parsing rules | scattered field formats |
| Transport | buffering and resends | manual routing by service |
| Storage | indexes, retention periods, access rights | markup schema and field names |
| Consumers | ready-made dashboards and alerts | manual exports on request |
Smooth migration of services
We migrate one service or group of nodes at a time, not everything at once. The new contour is enabled alongside the old one: both write in parallel, and we compare volume, latency, and event markup — a discrepancy in counters and empty fields serve as a stop signal.
The collection scheme lives in the repository as configuration, so rolling back a node means returning to the previous revision.
Dual delivery during the transition period
The agent sends events to two receivers — the current storage and the new one, with an event identifier to cut off duplicates. History is not broken, reports continue to be built, and the new contour accumulates data for verification.
When the entire stream goes through the new collection and discrepancies are zero, we disable the old agents in stages, by groups, leaving a backup channel for several days.
Technical questions asked before the start
Before the start, it is important to understand how log aggregation will fit the current load and what will happen if one of the collection nodes fails.
Below are five questions that we review before the first change to the infrastructure: the delivery scheme, retention policy, and service switchover order depend on them.
What will the network and storage load be?
We calculate based on actuals: how many records per second the service emits, what the average record size is, and what share is filtered out during parsing.
Then we select compression and the level of detail — some debug records are suppressed on the source side, and the volume of data transferred and stored drops multiple times.
What happens if the collector is unavailable?
The agent writes the stream to a local disk buffer and sends it after connectivity is restored, so an interruption does not turn into a hole in the logs.
Separately, we check behavior when the buffer overflows and configure rate limiting so that collection does not displace the service's working process.
Can services be migrated in stages?
Yes, we go by contours: first one service or a non-production environment, then the rest. The old and new contours operate in parallel for some time, record counters are reconciled, and rollback is done by switching configuration without rebuilding applications.
How to handle sensitive fields in records?
Masking and hashing are performed before sending, on the source side — sensitive values no longer reach storage. We separate view access by roles, and log queries themselves are recorded in the audit.
What retention periods and storage tiers are needed?
We separate a hot tier for operational search over recent days and a cold archive with compression for investigations and checks. Rotation and deletion work automatically according to policy, and we set retention periods according to your requirements and volume.
Let's discuss the task and come back with an estimate and plan
To come back with an estimate of the scope of work rather than generalities, we need to understand the sources and profile of the log stream. Describe the task in free form — we will clarify what is missing ourselves.
- Which services and layers emit logs: applications, load balancers, databases, orchestrator, network components — and how they write them now.
- Event format: structured records with fields or unstructured text that will require parsing at the reception stage.
- Order of magnitude: how many events arrive at peak and on average per day, how volume grows under load.
- Retention period by type: where hot data is needed for search and incident investigation, and where a cold archive is sufficient.
- Sensitive data in records and requirements for access separation between teams.
- Network and permission constraints: which nodes can send data outside and how access to the storage cluster is granted.
In response, we send a flow diagram from source to visualization, an estimate of the scope of work by stage, the team composition with roles, and the order of connecting services — one at a time, so that a collection failure is visible immediately.
Describe the task in a convenient form: we will review it and come back with a plan of implementation stages.