Centralized logging in Kubernetes: where it breaks
Logs of a container live exactly as long as the container itself lives. The application writes to standard output, the file layer disappears on restart — and if the collector did not manage to pick up the data, for incident analysis those lines no longer exist.
The error that caused the crash is visible from a fragment, while the cause remains in the lost part. Worse, when collection goes directly to storage: during a network hiccup there is nowhere to flush the buffer, and it is not one pod that is lost, but the entire node.
Second failure point — time. Clocks on nodes diverge, timestamps come from different sources and in different formats, so events of a single failure are arranged in the wrong order.
It is impossible to reconstruct the sequence of "what broke after what" from such a sample, and the on-call engineer again goes to restart the service at random.
Third — noise and duplicates. The same request leaves a trace in the application, at the ingress, and in the system log; there is no deduplication, no parsing or enrichment either.
The volume of unprocessed data grows faster than the team can read it: useful lines in the stream are a few percent, the rest are repeats and service noise. It is at this stage that log collection in the cluster stops being used and teams return to manually reading from nodes.
In practice, all three problems are solved by one layer between the application and storage — Vector is deployed in the cluster precisely for this. Let's break down how this is arranged and what needs to be provided for.
What a unified log collection and storage loop provides
An agent on each node collects the container stream and sends it to a single receiver — instead of logging into a specific machine and reading local files, the on-call engineer opens one entry point.
The share of incidents where the cause is found on the first attempt grows from a third to the majority: context for all services involved in the request lies nearby, not in four systems.
Deduplication and stream compression solve the second problem — volume. The same stack trace, multiplied across dozens of pods, collapses into a record with a repeat counter, and the structured format packs more densely than raw text.
Log retention grows without expanding disks: what previously lived for only a few days now lives for months, and analyzing an incident from a week ago no longer hits a void.
Routing by teams is built on labels: namespace, service, environment. Service owners receive their own stream and alerts for their own errors, not the full cluster noise.
Time to first response on an incident drops from tens of minutes to single digits — the needed fragment is retrieved with one query, not by iterating through hosts.
Separately, the class of failures "logs exist, but not where needed" and "the pod was recreated — the trace was lost" goes away: buffering and retry sending keep the stream during restarts.
The Vector collector describes the entire loop declaratively — transformation rules and routes are versioned together with the cluster configuration, so a new team is onboarded by editing a file, not by setting up a separate instance.
Centralized logging schemes: comparison of approaches
The collection scheme determines at what point the log stream is intercepted and what happens to it when storage is unavailable.
In a cluster, teams realistically choose between four options: they differ in the interception point, overhead on nodes, and behavior during a connection break.
| Option | When it fits | Limitations |
|---|---|---|
| Log collection agent on the node (Vector as a node service) | Many services on a node, unified processing rules | One process reads files from all containers: overhead on nodes grows with the stream, and an agent failure affects all pods on the node at once |
| Sidecar container for logs | Heterogeneous formats, separate configuration per service is needed | Multiplies the number of containers in a pod, adds memory consumption for each instance, complicates configuration updates |
| Direct write from the application | Several services, strict requirements for record structure | Sending logic lives in the code; when storage is unavailable, the buffer is limited by process memory, and the application itself performs retries |
| Hybrid collection scheme | Dense nodes with heterogeneous services | A policy is needed: what is given to the agent, what is moved to a separate container, and where disk buffering is enabled |
Buffering when storage is unavailable is a separate design decision, not a side effect of the scheme.
Disk buffering on a node survives agent restart, but when it overflows it evicts the oldest records — we agree on this class of loss and the order of catch-up sending after a long break in advance, before rollout.
Implementation stages: from inventorying streams to handover
Centralized logging does not start with installing an agent — first, they record what generates events, at what rate, and in what format. Otherwise the loop will not withstand the peak or will store noise that cannot be used to find the cause of a failure. Each subsequent stage relies on the data from the previous one.
- Inventory of log sources. We go through workloads: whether the application writes to stdout or to a file, whether there are non-standard formats and multiline error stacks, what the cluster audit provides. The output is a list of streams with an estimate of daily volume and peak rate.
- Stream routing scheme. We decide what goes to hot storage for search, what goes to a cheap archive, and what goes to alerts; streams receive tags, retention periods, and duplicate discard rules.
- Collection and transformation. The agent is deployed on each node, disk buffering on the local disk is enabled, structured records are parsed into fields, multiline messages are stitched together, and sensitive data is masked before sending.
- Load testing of the loop. We run a stream above the calculated peak: does the buffer grow, can storage keep up, how does delivery latency behave.
- Testing during storage failure. We disable the receiver and make sure the agent retains events on disk, and after recovery sends them without gaps or loss of order.
- Policies and alerts. Delivery latency monitoring, alerts for buffer overflow, access roles, retention periods, and the procedure for export on request.
- Handover of the loop to operations. We hand over configurations, the routing scheme, dashboards, and instructions to the on-call engineer, and go through typical incidents with your team.
Configurations, schemes, and policies: what remains with you
Implementation ends not with a working cluster, but with a set of artifacts that can be read, edited, and handed over to new people.
Everything lies in Git alongside manifests and goes through review, so the collection configuration does not remain as the only copy on someone's machine.
- Collection agent configuration. Sources, transformations, routing by receivers, disk buffering, and behavior when storage is unavailable — with comments explaining why these particular parameters were chosen.
- Log stream scheme. Where data comes from, where it is enriched with namespace and pod labels, where it is filtered, which receiver it goes to, and at what point it is lost first under overload.
- Log access policies. Separation by roles and namespaces, separate accounts for the on-call shift and incident analysis, masking of sensitive fields at the collection stage.
- Rotation and retention policy. Retention periods by log type, segment size and count, the rule for moving to cold storage, and the list of what is deleted permanently.
- Instructions for the on-call engineer. Step-by-step scenarios: how to find logs for a specific pod, how to check that the agent is sending data, what to look at when the receiver lags, and when to escalate.
- Change policy. How to connect a new application to collection, who approves routing changes, and how to roll back the configuration if the stream disappears after rollout.
Before handover, the artifacts are checked on your staging environment: we run typical operations together with the on-call shift and adjust the wording for your process.
Case: how the time to find the cause of a failure was reduced
What it was before
The application wrote logs to standard output, files on the node were rotated, and rotation overwrote exactly the interval needed for analysis.
Each service had its own time format and its own scale of levels, so the on-call engineer assembled the picture manually: walked around nodes, matched timestamps, and blindly looked for correlation between services.
Average time to find the cause of a failure was 45 minutes; on cascading incidents it reached up to two hours, and some episodes could not be reconstructed at all.
What was changed
We deployed Vector as a set of agents on each node: they read both container files and standard output, parse formats on the agent side, and bring time to a single time zone.
Labels for namespace, pod, service, and build version are added to each record — these are exactly what the query is later built on. Noise is cut off at the same time: liveness checks, repeating messages, debug levels.
Next, routing: critical levels go to hot storage with fast search, the rest — to cold storage with long retention. Reduction in the stored stream volume — approximately 6x, without losing significant events.
A single point for log search appeared: one query by service and interval instead of walking through nodes.
What it became
Time to find the cause of a failure — 4–6 minutes, this is the 95th percentile of incidents. The eliminated class of failures — log loss during pod restart and time desynchronization between services, because of which event correlation was built incorrectly. Incident analysis now starts with a query by label, not with data collection.
Log collection by a node agent or a sidecar container?
Both schemes solve one task — delivering logs to the centralized logging system — but differ in where the collection process lives and whose resources it consumes.
A node agent runs one per node and reads container logs through a mounted volume; a sidecar container starts next to the application in the same pod and sees only its output stream.
A node agent is cheaper: one process, one buffer, one update point for the entire node. The price — a shared resource pool: a spike in logs from one service clogs the queue, and neighboring streams lose events.
Isolation of log streams rests not on resources, but on routing rules by pod and namespace labels, so a configuration error affects the entire node at once.
A sidecar container provides maximum isolation: its own process, its own limits, its own parsing, and its own receiver — overflow of one sidecar does not affect its neighbors.
On the other hand, log collection overhead grows linearly with the number of pods: a copy of the process for each instance, separate memory quotas, and multiplied maintenance work when updating the fleet.
Vector supports both topologies with one configuration: the agent is deployed as a set of daemons on nodes, the sidecar — as a separate container in the pod, and read positions are recorded in checkpoints, so a restart does not resend the entire file.
In practice, a hybrid is more often chosen: the base stream is collected by a node agent, and a sidecar is deployed selectively — for services with special requirements for access, parsing, or destination.
Answers to technical questions about log collection
Logs are needed not when everything works, but when something has failed — therefore questions about centralized logging concern not the setup itself, but behavior in abnormal situations.
Below are the topics most often discussed before starting: safety when storage is unavailable, container eviction, access between teams, and connecting an existing storage system.
What happens to logs if storage is unavailable?
The agent buffers the stream on the node's disk and retries sending with increasing delay, so during a brief failure data is not lost.
The disk queue has a limit, and the overflow policy is chosen by criticality: for audit we set a blocking mode, for debug streams — dropping the oldest records.
A container is evicted — will the last logs be preserved?
The agent runs separately from the application, so restart or eviction of a pod does not interrupt collection: output is read to the end, including lines written before the process terminated. When a node is decommissioned, the buffer has time to flush before the pod is stopped.
How to separate access between teams?
Routing is built on metadata — namespace, pod name, labels. Each team gets its own index in storage and rights only to it, so product development does not see logs of the payment loop, while the on-call engineer sees everything needed for failure analysis.
Can our existing storage be connected?
Yes, if it has a compatible receiver: Elasticsearch, Loki, object storage, or a queue as an intermediate buffer. We adjust the field and index scheme for your dashboards so that old queries and alerts continue to work.
Let's discuss the task and come back with an estimate and plan
To come back with an estimate of implementation work and a plan for deploying the loop, we need input about your infrastructure — they determine the agent architecture, routing scheme, and effort.
Below is the list of what is worth preparing: if some data is unknown, we will gather it together on a call.
- Load profile and stream volume. Number of nodes and workloads, average and peak event rate, daily growth, whether there are sharp spikes during incidents. Disk buffering and throughput headroom depend on this.
- Log sources and format. Application output, system logs of nodes, cluster audit log, data layer logs. It is important whether this is structured JSON or free text — it determines the set of parsing rules and enrichment with cluster metadata.
- Storage requirements. Retention period for each log type, whether full-text search is needed, deletion and archiving rules, expected registry volume in the future.
- Loop scheme. One cluster or several, own servers or cloud, restrictions on outbound traffic, where the collector may be placed.
- Accesses and security requirements. Service accounts with permissions to read metadata, channel encryption, need to mask personal data before writing.
- Current state. Which collection agents are already deployed and where they write — so as not to raise a second parallel loop and not pay for it twice.
- Rollout order and rollback points. Whether it is acceptable to enable collection in stages, starting with one workload, and how quickly the previous scheme needs to be restored.
Describe the task within these boundaries — we will come back with an estimate, a work plan, and a Vector deployment scheme for your loop.