Centralized logging: where fragmented log collection breaks
As long as logs are written locally on each machine, the system has no single picture. A service writes to journald, an application to a file inside a volume, a load balancer to its own log; each has its own format, its own level of detail, and its own rotation moment.
During a failure, an engineer gets not data for analysis but a list of servers and the task of finding the cause manually.
Iterating over nodes is the most expensive part. You need to log into each one in turn, remember the path to the log, download a fragment, and match timestamps — but the clocks on the machines diverge, and events from a single request end up in different files with an offset.
A failure spanning several services has to be assembled piece by piece, losing hours in the process.
Some records disappear physically. Container logs live in a layer that is deleted together with the container: a restart — and the context of the crash is gone.
If the application crashes cyclically, the file is overwritten with the same messages, displacing what was there before the failure. Losing logs on restart resets the investigation exactly when the data is most needed.
The second side is disk overflow for logs. Without limits on size and retention, logs consume the partition, and then the very service that writes them degrades or crashes.
Emergency cleanup on a schedule or overly aggressive rotation has the opposite effect: by the time of analysis, the needed records have already been deleted.
We explain how this is solved by centralized logging: a single collection point, unified timestamps, policy-based storage, and search across all services at once, not one by one.
What a single collection and search point across all services provides
When each service writes logs to its own file on its own node, the cause of a failure is sought by brute force: connect to the right machine, guess the interval, manually match timestamps.
A single log collection point on Grafana Loki removes this stage — records from all services go into one storage with shared labels for service, environment, and build version.
End-to-end log search instead of iterating over nodes
Search runs by the request ID that services put in every line. The engineer sees the entire chain — from the external gateway to the queue and database — in one window, without coordinating access to other teams' machines.
Incident analysis time for a typical failure drops from hours to minutes, and this is the first effect the on-call shift notices.
Completeness of the picture across the entire request chain
Labels by service, environment, and build version make it possible to compare the behavior of test and production environments and see at which step a message was lost.
A whole class of failures — discrepancies between services, broken requests, "silent" errors — used to be found only from a user complaint; now it is visible in the stream as an anomaly.
Log retention period and service observability
The retention policy sets the lifetime of records: recent ones are kept hot for analyzing current failures, old ones go to cold storage — volume does not grow uncontrollably, while history remains available for analyzing recurring problems. Service observability no longer depends on who looked at a specific node and when.
Log delivery methods: agent, receiver, service gateway
Records from machines and applications can be collected in several ways, and the choice is dictated not by convenience but by what already exists in the environment. A service writes to a file — no need to change code, the agent reads the stream locally.
Strict fields and request markup are needed — the application itself will have to be modified. Below are real options and their limitations, so they do not surface during implementation.
| Option | When it fits | Limitations |
|---|---|---|
| Node agent — log collection by agent | logs are written to a file or system journal, code cannot be changed | collection overhead on each node; without a buffer disk, it loses the tail when the receiver is unavailable |
| Network log receiver | network devices and legacy services that can only send over the network | no delivery confirmation when the channel breaks, structure reduced to a single line |
| Service gateway next to the application | containers, many short-lived processes, output to a stream | complicates deployment, the gateway has its own lifecycle and updates |
| Direct delivery from the application | precise fields, request markup, linkage with tracing are needed | application modification and dependency on a library; a channel failure drops the in-memory buffer |
Buffering is a common concern for all paths. The agent and gateway write accumulated data to disk and retry when the receiver is available again, while sending directly from code keeps the queue in the process memory and loses it along with the process.
Hence the working scheme is mixed: an agent as the foundation, a gateway where processes are short-lived.
Implementation procedure: from source inventory to handover
We build the log implementation stages so that after handover the system lives without us. We start not with installation but with an inventory of log sources: as long as it is unknown how many services write, in what format, and at what intensity, any configuration is guesswork.
- Inventory of log sources — we walk through services, nodes, and applications, recording the record format, daily volume, and owner. Using this table, we calculate storage load and retention period.
- Log flow diagram — where the collector sits, how records reach storage, which labels (service, environment, level) are used. We select labels so that cardinality remains predictable.
- Deployment — Loki and Grafana, a hot layer on disks and long-term storage in object storage. A Lifecycle rule keeps the last days in fast access and moves the rest to the cold layer.
- Access and isolation — roles in Grafana and separate data sources per team: a developer sees their services and does not read others'.
- Testing under real load — we reproduce the peak stream, measure the delay from writing to appearing in search, and observe behavior when the collector fails and the disk overflows.
- Alerts and dashboards — LogQL rules for errors and spikes, routing notifications to the on-call engineer.
- Handover of infrastructure to the customer — configuration in the repository, operating procedures, team training on a live environment.
What remains with you after handover: configurations and procedures
The project does not end with a dashboard demonstration: at the output you receive a set of artifacts by which the system lives and evolves without our involvement.
Everything is stored in your repository, versioned, and reviewed like ordinary code — a change to a collection rule or threshold is visible in the change history.
- Log collector configurations. Agent files for each node and service: record sources, parsing rules, environment and team labels, retention periods for each stream.
- Schema of record flows. The path from application to storage and the search interface, including behavior when storage is unavailable — disk buffering and the retry procedure.
- Policies for access to logs. Roles and isolation by teams and environments, masking of sensitive fields before writing, separate read rights for the production environment.
- Description of thresholds and alerting rules. What is considered an incident, on which window and frequency it triggers, where the notification goes, and how the rule is disabled during false positives.
- Search documentation. A cheat sheet for typical queries — by request ID, service, time window — and an analysis of two or three real incidents with the cause found.
- On-call procedures. Who looks at the panels and when, response time, escalation procedure, and actions when log storage is unavailable.
- Instructions for connecting a new service. A configuration template and a check that records have appeared, labels are set, and the stream is not duplicated.
We hand over all artifacts with accompanying descriptions: where they are stored, who the owner is, what changes when a service is added.
Case study: how incident analysis was reduced from hours to minutes
Before the project, logs lived in four places: files on application servers, the container stream, the load balancer panel, and database tables.
An engineer opened four interfaces and compared records by time manually — finding the cause of a failure took from two to five hours, and some failures left nothing behind except a client complaint.
We deployed Grafana Loki as a single log storage. The collector on each node assigns labels to records for service, environment, and build version — search uses them, so there is no need to remember what is stored on which machine.
Hot data lives for a month, then is offloaded to cold storage: the registry does not grow uncontrollably, while analysis of long-ago requests remains possible.
What changed in incident handling
Connecting sources proceeded in waves: first critical services, then background jobs and the network layer. In total, 47 sources in three weeks, including those that had not been logged at all before.
A query for a specific client returns a chain of records from all nodes in seconds, without manual time matching.
The time to localize the cause of a failure decreased from hours to 10–15 minutes — on average 12 times according to recorded incidents.
We separately closed a class of failures that previously became known only from users: desynchronization with external systems and failures of background jobs. Now a notification triggers for them, and the on-call engineer sees the problem before the client notices it.
How centralized logging differs from log collection
Log collection is transport: the agent reads files, buffers them, and sends them to a common node. The files end up nearby, but it does not become a tool — iterating over hosts and manual search remain.
Centralized logging begins where a single index, access isolation, and managed retention are added to collection.
A single search replaces iterating over machines: the query runs by service, environment, and version labels, and the output is combined into one feed with context before and after the event.
Correlation appears — by request ID you can see on which service it degraded, without switching between terminals and unpacked archives.
Roles and retention periods are the second boundary.
Access to logs of the payment environment is not granted together with access to the development environment, and the period is set by policy: the hot index keeps the last days for failure analysis, the archive keeps longer, for investigations and regulator requirements.
| Criterion | Files in one place | Centralized logging |
|---|---|---|
| Search | manual host iteration | query by labels, one output |
| Access | shared access to the server | roles by source and environment |
| Storage | grows while there is disk space | policies by source and retention period |
| Structure | unparsed text | fields, labels, request ID |
In our practice it looks like this: Grafana Loki receives the stream, labels form the index, and panels and alerts are built on top of the same source — failure analysis happens in one interface instead of exchanging "send me your file."
Technical concerns: volumes, storage, access
Before starting, people discuss not "why" but specifics: how the system will behave when the volume of records doubles, what will happen when the receiver is unavailable, and who will get access to others' logs.
Connecting ready-made applications does not require code changes: if a service writes to standard output or a file, it is enough to configure the collection agent and labels. Below are the questions asked most often.
How quickly does the volume of records grow, and what limits it?
The volume grows along with the number of services and traffic. We limit it at the input: discard noise — liveness checks, recurring errors — set quotas per application, and downsample when exceeded. Growth becomes predictable, rather than "until the disk runs out."
What happens if the receiver is unavailable?
The agent buffers records on the local disk and sends them after connectivity is restored. The buffer is limited: when it overflows, the oldest records are lost, but the application itself continues to operate and is not blocked.
How is access to records isolated?
We separate by labels and spaces: a team sees only its own services. Read and rule-change roles are granted separately; access to neighboring teams' logs is closed by default.
How long are records stored, and how are they deleted?
We set the period by policy per class: fresh records are kept in fast storage for incident analysis, then move to compressed storage and are deleted on a schedule. Storage does not grow indefinitely.
Let's discuss the task: what to prepare for the first conversation
To discuss logging implementation substantively rather than in general terms, introductory information about your infrastructure is enough — nothing needs to be prepared in advance.
We deploy the panels, collectors, and log storage ourselves; from you we need the context against which the volume is calculated.
After the conversation, we will return with an estimate and work plan: stages, what we cover on our side, and where your administrators' access is needed.
- List of services and environments: which applications, databases, queues, and proxies write logs, which environments are separated — development, test, production. This determines the number of sources and how to distribute records by labels.
- Where records currently reside: files on servers with rotation, container logs, operating system output. It is important to understand what is already collected and what is lost on restart.
- Log search scenarios: incident analysis by request ID, errors of a specific service, support requests. Real questions determine the label scheme, retention period, and set of panels.
- Volume: approximate record stream per day and retention depth needed for incident analysis and audit requirements.
- Access requirements: which teams should see what, whether export to external systems is needed, and whether there are restrictions on personal data in records.
- Site limitations: network segments without outbound access, disk and channel limits, change approval procedure.
Half the list is enough — we will clarify the rest during the conversation. The more specific the list of services and search scenarios, the more accurate the estimate and the fewer changes during implementation.