Graylog: why logs get lost between servers
Logs are spread across dozens of machines: the application writes to journald, the load balancer writes to its own file, the database writes to an internal buffer, and some services even end up in containers that live for minutes.
When payments go down at midnight, the engineer opens SSH sessions to every node, greps through rotated files, and still can't see the whole picture — by the time troubleshooting starts, the oldest files have already been overwritten.
The cost of this setup isn't just hours of searching. An event that never left the host disappears along with it: a pod restart, autoscaling, a node failure — and the incident context is lost forever.
It's even worse when the logs exist, but every server has its own time and its own format, and stitching them into a single timeline by hand is impossible.
Centralized logging via Graylog solves this at the collection layer: an agent on each machine picks up events from files, journals, and container streams and sends them over the network to a single storage point.
From there, the server side takes over — it receives the stream, parses it into fields using templates, distributes it across indices, and retains exactly as much as is needed for incident investigation and audit requirements.
The key here isn't "collecting for the sake of collecting," but what happens afterward. A single storage point turns searching from a tour of servers into a single query: by service, level, request ID, or time window.
The engineer sees the entire chain — from the incoming request to the error in the database — as a continuous sequence of events, not as a set of scattered fragments.
The question of loss is addressed separately. The buffer on the agent side survives server unavailability, while on the receiver side, queues and drop rules are configured: under load, noise is dropped, not meaningful events.
This is the difference between "the logs are somewhere" and "the logs can be used to investigate an incident."
We break down how this is solved: from choosing the collection scheme and field tagging to retention policies and data access.
What centralized log collection provides
Log search instead of scanning servers one by one
When logs live on dozens of hosts, investigating an incident turns into a manual walkthrough: connect to each one, remember the file path, grep by time.
A unified Graylog storage eliminates this step — a query across all sources at once, filtered by service, node, and time window. What used to take an hour of coordinating "who should look where" comes down to a single search string in seconds.
A complete picture and a unified format
Different applications write in their own way: some in JSON, some in free text, some stay silent about errors altogether. We bring streams to a common format through parsers and extract fields — level, service, request ID.
As a result, an event scattered across several nodes is assembled into a single chain, and the cause of a failure is visible in full, not as a fragment.
Less manual work and predictable rollback
Configured alerts remove the "eyes on logs" duty: a rule triggers on an anomaly and sends context rather than a raw stream.
Rotation and retention are set by policy — the registry doesn't grow uncontrollably, while history for the required period remains available for investigation. Engineers spend time on remediation, not on finding where things are stored.
What this changes for the team
Search speed, event completeness, and a unified format together deliver the main thing — a predictable time to answer the question "what happened." Let's work out how this is covered on your infrastructure.
Methods of delivering logs to Graylog
The delivery method determines what you will actually see in Graylog. An agent on the host reads files and application buffers, the system journal collects everything the operating system writes, and sending from the application produces a structured event with ready-made fields.
The choice affects the completeness of the picture, the load on hosts, and the speed of search during incident analysis.
| Option | When it fits | Limitations |
|---|---|---|
| Collection agent on the host | Heterogeneous server fleet, logs in files, collection from hundreds of machines is needed | Requires installation and updates on every host, buffer control is needed when the connection drops |
| Collection via the system journal | Quick start, infrastructure events, a unified format from the operating system | A lot of unstructured text, some fields are lost, weak linkage to business events |
| Sending from the application | Critical services where request context, user ID, and tracing matter | Requires code changes and a unified event format, otherwise fields across different services will not match |
| Direct network sending from the logging library | Containers and short-lived tasks where the agent does not have time to start | No local buffer: if the receiver is unavailable, events are lost irrevocably |
In practice, the options are combined: the agent covers the system level and files, while the application sends business events directly. Then both infrastructure and application data converge in Graylog, and incident search proceeds by a single request ID.
We tag sources so that every event carries the service name, environment, and version — without this, analysis turns into manual trial and error.
Separately, we decide what to do when the connection to the receiver drops: we configure a disk buffer and a limit on its size so that the host does not go into overflow.
Describe your fleet and critical services — we will come back with an assessment and a plan for the delivery scheme.
How we implement centralized logging
Implementation starts not with installing Graylog, but with inventorying what already generates events: applications, load balancers, databases, the orchestrator, network equipment.
At this stage we record the formats (plain text, JSON, GELF), daily volumes, and peak intensity — the topology of inputs and the storage size depend on them. Without inventory, some sources are lost, and gaps in the logs are discovered only during incident analysis.
- Surveying sources. We compile a list of systems, formats, and message volumes, and determine where an agent is needed and where sending over the network is sufficient. We separately note sources with a non-standard format — we prepare parsing rules for them.
- Designing the flow schema. We distribute messages across streams and indices: application, security, infrastructure. We configure routing rules and retention policies — we keep hot data fast, and the archive goes to cheap storage.
- Configuring ingestion. We bring up inputs for the required protocols, configure field extraction, time normalization, and enrichment (service name, environment, version). This is what turns raw text into data that queries and alerts are built on.
- Verifying completeness. We check that everything arrives: we compare counters at the source and in the system, catch lost messages and duplicates, and check behavior under peak load and connection loss.
- Alerts and dashboards. We configure rules for anomalies and thresholds, and assemble panels for key services so that the on-call engineer sees the picture without manual queries.
- Regulations and access. We delimit rights by team, and describe retention periods and the procedure for adding new sources.
- Handover to operations. We train the on-call staff, hand over the documentation, and transfer the system to support.
Each stage ends with a verifiable result, not a report. Describe your infrastructure — we will come back with an assessment and an implementation plan.
What the customer receives
We deploy Graylog so that the system lives without us: streams, indices, parsing rules, and access are described in configuration, not "in the engineer's head." Below is exactly what stays with your team after handover.
- Configurations of all nodes: log ingestion entry points, binding of streams to indices, storage and rotation parameters, replication settings — as version-controlled files, not manual edits in the web interface.
- Stream and routing scheme: which source goes where, by what rules a message is routed to the right stream, how priorities are configured, and what happens to unparsed events.
- Parsing and field extraction rules: rule sets for each source type — parsing dates, levels, request identifiers, normalization to a unified field schema so that search across different systems works the same way.
- Dashboards and alerts: ready-made panels for key services and rules that trigger on anomalies — a rise in the error rate, a missing log stream, a spike in failures.
- Operations regulations: the procedure for adding a new source, index rotation, updating parsing rules, and recovering after a node failure — step by step, with responsible parties.
- Documentation: architecture diagram, stream descriptions, field dictionary, and instructions for the on-call engineer — so that incident investigation doesn't start with the question "where do I look for this."
- Access and permission separation: roles for teams, separate views for their services, auditing of actions in the system.
We hand over every artifact in working order and verify it on your infrastructure. Describe your task — we'll come back with an estimate and a plan.
Case study: how incident triage time was cut
What it was like before implementation
A client came to us with a classic picture: three environments — production, pre-production, and internal services — each with its own way of looking at logs. Some applications wrote to files on local disks, some sent their stream to journald, and some to container stdout.
When triaging an incident, the on-call engineer first had to figure out which node the problematic service lived on, then go to that node, then remember what format events were written in there.
The average triage of a single incident took 40-60 minutes, and most of that time went not into analysis but into collecting data.
Worse still: when a container restarted, the logs disappeared along with it, and the picture of the failure could only be reconstructed from indirect signs. Different time formats across services added confusion — correlating events from two systems by time was a task in itself.
What was changed
We deployed Graylog as a single collection point: sidecar agents on the nodes, GELF at the input, one index per environment.
The key was not the collection itself, but bringing events to a common schema: mandatory fields for service, environment, and request ID, plus a unified time format.
Then we set up field extraction from unstructured messages and streams with separate retention rules for each environment.
Access was addressed separately: the on-call engineer sees only their own streams, while a developer sees their service across all environments.
Retention is configured so that hot data lives in a fast index, while everything older moves to an archive — and search across the archive remains available from the same interface.
What it became
| Metric | Before | After |
|---|---|---|
| Average incident triage time | 40-60 min | 8-12 min |
| Share of incidents triaged without pulling in a second engineer | ~30% | ~80% |
| Reconstructing the picture after a service restart | impossible | full history |
| Searching events over a month | manual traversal of nodes | a single query |
The main gain came not from log collection as such, but from the unified field schema: searching by request ID immediately shows the entire chain — from the load balancer to the database. The engineer stopped being a courier between systems and got down to actual diagnostics.
How does Graylog differ from collecting logs on each server?
With local storage, each server keeps its own log, and this works right up until the first incident investigation.
A failure affected three nodes — you open three consoles, manually correlate timestamps, and some of the logs have already been overwritten by rotation or lost along with the disk of the crashed machine.
The larger the fleet, the more expensive each such investigation: engineers' time goes into collecting data rather than fixing the cause.
Centralized storage changes the very scheme: agents on servers send a stream of events to a shared repository, where records are indexed and live by uniform rules.
Graylog receives events over standard collection protocols, sorts them into streams, and stores them in a search index — the question "what was happening across all nodes between two timestamps" is answered with a single query instead of walking through machines.
The local log remains as a buffer in case of a connection break, but it stops being the only source of truth.
The difference shows up in three things. First — speed: searching across a distributed incident takes minutes, not hours of manual correlation.
Second — retention: the storage period is set by policy on the storage side, not by the disk size of a specific server, so events don't disappear along with a node.
Third — a unified context: parsing rules, masking of sensitive fields, and alerts are configured once and applied to all sources, rather than duplicated on every machine.
There is also a downside worth stating honestly. A centralized scheme adds a point that must be maintained: the storage, indexes, access rights, and the network channel between agents and the receiver.
If the event stream isn't filtered at the input, the volume grows faster than the value of search. That's why implementation starts with rules for selecting and structuring records, and only then expands coverage.
The practical conclusion is simple: local logs are good for debugging a single service here and now, while centralized storage is for operating a fleet where it matters to recover after a failure and prove what happened.
Describe how many nodes you have and which sources need to be brought together — we'll come back with an estimate and an implementation plan.
Common technical concerns
Concerns about centralized logging usually come down to three things: whether log collection will overload production servers, whether storage will grow to indecent sizes, and whether a single entry point to logs will become a security hole. Below is how each of these is addressed at the engineering level.
Will log collection overload production servers?
The agent reads files incrementally and sends data asynchronously, with an on-disk buffer and a throughput limit. If the receiver is unavailable, the agent does not block the application and does not lose events — they wait in a queue.
In addition, each host has a limit set on the share of CPU and memory, so even a peak burst of logs does not displace working processes. In practice, overhead stays within a few percent rather than "eating half the server."
Will storage grow and become unmanageable?
The volume is determined not "by eye" but by a retention policy: hot indices are kept for N days for fast search, after which the data goes to cold archive or is deleted by a retention rule.
Separately, rotation is configured by index size and by the number of shards — the registry stops growing indefinitely.
Some of the noise (health-check, routine heartbeat) is filtered out at the input through filters, and only what is actually needed for incident investigation ends up in storage.
Is a single log collection point a security risk?
Access to the interface and to the API is role-based: a developer sees only their own streams, an on-call engineer sees the entire production environment, and audit has read-only access.
Traffic between agents and the receiver is encrypted, and the receiver itself lives in a separate network segment and is not exposed to the outside.
All actions in the system are logged — who, when, and what request was performed — so the platform itself does not become a blind spot.
What happens if the receiver is temporarily unavailable?
Agents buffer events locally and resend them after connectivity is restored, so an unavailability window does not turn into a gap in the logs. For critical streams, a backup receiver or a second delivery path is configured. This is a standard part of the design, not an emergency scenario that has to be put out manually.
Describe your infrastructure and log volumes — we will come back with an assessment and an implementation plan.
Let's discuss your logging task
To come back with an estimate and a plan rather than generalities, we need to understand your current log collection setup.
Describe where events come from (applications, load balancers, databases, network devices), where they are currently written, and what is already configured — that is enough to discuss the task concretely.
Next, we will compare your picture against a typical Graylog architecture and show exactly where the flows diverge:
- sources and delivery methods: agents on hosts, network ingestion, collection from containers and cloud services;
- parsing and normalization: extracting fields from unstructured text, a unified schema for heterogeneous sources;
- routing: streams and rules by which events reach the right indices and storage;
- storage and rotation: index lifecycles, size-based splitting, old data cleanup policy;
- search and alerts: saved queries, dashboards, triggers on anomalies and thresholds;
- access: roles, separation by teams and projects, auditing of actions in the system;
- operations: configuration backup, upgrades, monitoring ingestion load.
For each item, we will say what we do ourselves and where your side is needed — access, network diagram, list of sources.
This way the estimate and plan are based on real infrastructure rather than assumptions, and you can see the scope of work and the implementation order in advance.
Describe your current setup and what you are unhappy with right now — we will come back with an estimate and a plan.