Setting up centralized logging with Fluentd / Fluent Bit

Why Fluentd loses logs under peak load

Peak traffic breaks log collection not at the moment of writing, but at the moment of sending. The Fluentd agent on a node receives events from applications faster than it can deliver them to storage and places them in a buffer — first in memory, then on disk.

When the space allocated for the queue runs out, the overflow policy triggers: the agent begins discarding the oldest chunks. That is why logs disappear exactly when incident investigation is underway and they are needed most.

The collection bottleneck is rarely just one.

Different log formats are processed by different filters: structured JSON is parsed quickly, while multiline traces and an irregular stream from the web server require regular expressions and timeout-based joining — this costs CPU time on every line. The more lines per unit of time, the slower the filters and the faster the queue grows.

During incident investigation the picture repeats: the queue on the node grew to its limit in a matter of minutes, the buffer flush interval cannot drain it fast enough, and the queue's disk storage is full.

Some events reached central storage, some remained in chunks that have already been deleted during rotation. It is impossible to reconstruct the chronology from truncated data — what disappeared is precisely the failure boundary.

We examine how this is addressed: we calculate the actual throughput of the agent on the node, set buffering with disk headroom, choose an overflow policy that preserves fresh events, and bring different log formats to a unified schema before sending.

What centralized log collection provides and where the gain is

When logs reside separately on each node, incident investigation begins with walking through machines rather than with a hypothesis.

The engineer cross-checks hosts, time zones, and versions, loses context between containers of a single request — log search time grows with the number of nodes, while downtime cost is calculated before a diagnosis even exists.

Centralized logging removes that stage. The Fluentd agent collects container output and system logs, joins multiline messages into a single event, adds metadata — node, service, environment — and routes streams by tags so application noise does not mix with the kernel log.

The disk buffer survives receiver unavailability: when communication breaks, the agent slows down reading and delivers the accumulated data later instead of losing records.

Then the gain becomes measurable. Log search turns into a single query by metadata, and incident investigation into viewing the event chain of a single request by correlation ID.

The share of recovered events grows: nothing is trimmed at the collection stage, so the picture is assembled even for failures that were previously attributed to "no data."

The class of failures also changes. The scenario "a failure was noticed, but the cause was not found" goes away — records remain with exact time, node, and build version.

Retention is set by policy per tag: voluminous technical messages go to cold storage, while errors and access events remain at hand longer.

Fluentd operating modes: agent on a node, service, gateway

Fluentd is not only an agent on every node. The same collector is deployed in three modes, and the choice determines where the buffer accumulates, how many resources are consumed by the application, and what happens if the channel to the receiver breaks. Below are the options that are actually chosen between.

The buffering mode is selected according to the fault-tolerance requirements for collection.

An in-memory buffer gives maximum speed but is lost when the process restarts; a disk buffer with limits on size and number of chunks survives agent crashes and loss of connection to the receiver.

The scheme of "lightweight agent on the node plus a collection gateway with disk buffer" offloads applications: the gateway, not the service itself, handles the network and retries.

Stages of implementing centralized logging

The order of work is built so that each step relies on data from the previous one.

We start not with installing agents, but with inventory: as long as it is unknown how many sources there are and what formats they use, routing turns into guesswork and resurfaces as lost records during the very first incident investigation.

1. Source inventory. A list of services, files, system logs, and containers: record format (structured JSON, unstructured text, multiline traces), growth rate, owner. The output is a table with onboarding priority: who is included in the first wave, who is deliberately deferred.

2. Stream schema. The route "agent on node → aggregator → storage," with disk buffering and defined behavior when the receiver is unavailable. Tags and labels (service, environment, node) are fixed in advance — they are later used to build queries and alerts.

3. Collection and parsing setup. Fluentd configurations — parsers, filters, buffer sizes, retries on errors — are kept in a repository and changed through review, not by editing on the server.

4. Pilot group. Three to five services with different log formats. We check losses when the network breaks, the load from the agent on the node, and the correctness of parsing multiline records.

5. Log completeness check. We compare the number of records at the source and in storage and look for gaps under peak load. The acceptance criterion is a discrepancy within specified limits, not "everything seems to arrive."

6. Handover to operations. Instructions for typical failures, alerts for delivery lag and buffer overflow, and a procedure for connecting a new source — so that services can be added without our involvement.

What the customer receives: Fluentd configurations and procedures

The work ends not with a running service, but with a set of artifacts that your team maintains on its own.

We hand over each item in a form suitable for changes without us: configurations are in a repository, routing decisions are described in words, not only in the engineer's head.

• Collection configurations — descriptions of sources, parsing filters, and output routes as a single set of files in a repository. Changes are tested in a test environment: the collector starts with the new set, runs reference events, and only then goes to the production environment.
• Stream schemas — where each type of event comes from and goes to, where the disk buffer is located, where events are enriched, and where they are discarded as duplicates. The schema lets you understand in a minute why a record from a specific service did not reach storage.
• Access policies — which teams see which log sets, which fields are masked before writing, and how collector tokens are issued and revoked. We separately document what to do when an employee leaves or a contractor changes.
• Retention and cleanup rules — how long events live in fast access, what goes to the archive, and at what age the rest is deleted. Storage stops growing uncontrollably, while the incident investigation window remains sufficient.
• On-call procedure — who receives a signal and in which channel when a collector fails or a stream is lost, how quickly a response is needed, what is considered an incident, and what is normal degradation.
• Metrics of the pipeline itself — delivery latency, buffer fill level, and share of dropped records. They show the problem before development reports it.
• Documentation — step-by-step instructions "how to connect a new source" and "how to troubleshoot typical failures," with examples of ready-made configuration blocks.

We perform acceptance testing on a live scenario: we shut down one collection node and check that events are not lost and, after recovery, arrive in full.

Case study: incident investigation from logs instead of hours of searching

Before: logs existed, but there was no picture of the incident

The service ran on ten nodes, and each wrote to a local file with size-based rotation.

During a failure, the engineer went through machines via terminal and manually matched timestamps — and still did not see the chain as a whole: some events were lost during rotation, some remained on a node that had already been removed from balancing by the time of investigation.

Log completeness hovered around 60%, searching for the needed event took 40-60 minutes, and service recovery stretched into hours.

What was changed

We assembled a unified pipeline: Fluentd for aggregation and a lightweight agent on nodes with a disk buffer that survives connection loss, agent restart, and machine reboot.

Every record was tagged with metadata — service, environment, release version, request ID — and brought to a unified format at ingestion, so search works by fields rather than by text.

Retention policy was tied to data value: frequent technical events live for days, errors and change traces for months.

After: the incident is investigated with a single query

Log completeness increased to 99%: losses from rotation and connection breaks are gone, and missed sends are delivered by the agent from the buffer after the channel is restored.

Event search takes 2-3 minutes instead of an hour, and service recovery time decreased by roughly four times — the on-call engineer sees the sequence from the first failure to the cascade instead of assembling it from fragments.

The load on nodes also decreased: a single worker thread with a configurable memory limit turned out to be lighter than the previous heavy collector.

Later, metrics and traces were connected on the same pipeline — failure points are now correlated with logs by shared metadata, and investigation does not start from scratch for each new tool.

How Fluentd differs from Fluent Bit and what to choose

These are not two competing products, but two levels of a single collection scheme. Fluent Bit is installed on each node as a lightweight agent: it reads container stdout and log files, adds metadata (pod name, namespace, labels), and sends them onward.

Fluentd takes the place of the central aggregator, where full-fledged routing, filtering, and distribution to multiple receivers are needed.

The difference is visible in resource consumption. Fluent Bit is a single statically built executable with no external dependencies, so dozens of nodes do not create noticeable load and the agent does not compete with working services for memory.

Fluentd is an environment with a large set of plugins and flexible disk buffers; it is more demanding and is deployed where resources are known to be sufficient.

The practical criterion is simple. If the task is to deliver logs from a node without loss and without increased consumption, Fluent Bit is sufficient with disk buffer configuration and retries when the receiver is unavailable.

If stream branching by labels, field masking, deduplication, multiline record joining, and distribution to multiple systems at once appear — that is a job for Fluentd.

A mixed scheme satisfies both requirements: a cheap agent remains on the nodes, while all logic moves to the aggregation layer. Then changing routing does not require restarting agents on every machine, and buffers at two levels smooth over storage unavailability.

Next, we examine how this scheme is configured and which buffer parameters determine whether records will be lost when a node restarts.

Technical doubts before implementing logging

At the evaluation stage, the same set of doubts almost always comes up: applications write logs differently, personal data ends up in lines, it is unclear how long to store all this and whether collection will slow down working services. Below is how we address each of them with configuration rather than promises.

What about personal data in logs?

Masking is performed on the agent side before sending: filters cut out values by field names and patterns, and everything else is filtered out by an allowlist.

Storage receives an already depersonalized stream, so retention no longer depends on personal data protection requirements.

How long to store data, and will the registry grow indefinitely?

The policy is set by data class: fresh logs reside in fast storage, then automatically move to a cheap archive and are deleted on a schedule.

Application logs and access journals have different retention periods — the registry does not grow indefinitely, while the incident investigation window remains sufficient.

Will logging degrade working services, and how to roll it out in stages?

The agent runs as a separate process, writes the buffer to disk, and is limited by CPU and memory. If the receiver is unavailable, the buffer survives restart, and the application continues to respond — log collection does not enter the request's critical path.

We start with a pilot pipeline: verify completeness and losses on one group of services, then roll out to the rest, with the ability to roll back the agent configuration.

Let's discuss your logging task

To make the estimate substantive rather than a rough guess, a few inputs are enough: which sources write logs (applications, system services, containers, load balancers), examples of real records from one to two hours, and the current path of those records — where they go and in what volume.

From the lines, it is immediately clear where field parsing and enrichment are needed and where delivery alone is sufficient, and this changes both the configuration and the timeline.

Then we examine the scheme as a whole: agents on nodes or a set of daemons in the cluster, disk buffering when the receiver is unavailable, labels and routing to multiple storages, and limiting index growth.

After the review, we return with a plan, timeline estimate, and scope of work — without abstract wording.

What the review includes and what you get as a result:

• A list of sources and labeling: which fields we extract from lines and what we leave in its original form.
• A delivery scheme: where agents are located, how queues are configured, and what happens when communication with the receiver breaks.
• A staged implementation plan: a pilot on one pipeline, completeness and loss verification, then rollout to the rest.
• A timeline and effort estimate for each stage with control points and readiness criteria.
• Fluentd configurations, stream schema, and retention rules — with a calculation of how quickly the volume grows.
• A post-launch support procedure: response to delivery failures, configuration updates, and on-call duty.
• Instructions for your team: how to add a new source without breaking parsing for existing ones.

Describe the task to the extent you have it on hand — we will return with an implementation plan, timeline estimate, and a clear area of responsibility after launch.