Configuring OpenTelemetry Collector for telemetry aggregation in Kubernetes

OpenTelemetry Collector in Kubernetes: where telemetry collection breaks

When configuring OpenTelemetry Collector in Kubernetes, the first working variant is almost always assembled on a single instance: receiving signals, batching, sending to storage.

While there are dozens of services, the scheme holds the flow, but at hundreds of pods and several nodes it begins to lose data — imperceptibly, because no one monitors the observability layer itself.

The first point of failure is the sending queue. If the storage responds slower than the flow arrives, the buffer fills up and the collector begins to drop signals according to the defined policy.

There is a counter for dropped signals, but no one reads it: gaps in observability coincide in time with service degradation, meaning they occur exactly when data is needed most.

Second — duplication. A sidecar next to the application, an agent on each node, and a shared aggregation layer can send the same data multiple times.

Counters and histograms are summed twice, dashboards show inflated load, and alerts trigger on numbers that do not exist in reality.

Third — blind spots for some services: somewhere an old agent remains, somewhere instrumentation was not added, and trace collection covers only part of the request paths. A failure in such a service is not visible in the overall picture until users report it.

A separate cost item is the load on applications. Synchronous sending from the process, processing and sampling on the service side take resources away from business logic, and during a traffic spike the queue inside the application grows faster than it can drain.

We examine where to place ingestion and aggregation, how to move processing into a separate layer, and how to configure memory limits so that priority signals get through even under load.

What a unified telemetry pipeline provides: effect in numbers

A unified telemetry pipeline is when metrics, logs, and traces go through a single layer with shared rules for filtering, enrichment, and routing.

Previously, each service wrote through its own agent to a separate storage, and the question “what happened” turned into manually traversing several systems.

Now the OpenTelemetry Collector receives signals, normalizes them to common attributes, and distributes them to their destinations.

At acceptance, we record measurable effects:

• Time to find the root cause of a failure: it used to be hours comparing dashboards, now it is minutes, because a single request identifier shows the entire chain from the ingress node to the database.
• Telemetry loss: the share of lost measurements drops from percentages to tenths of a percent due to buffering on the agent and retries when storage is unavailable.
• Volume of stored data: reduced severalfold — service metrics and extra attributes are cut off at ingestion, not after writing.
• Failure classes: connection drops, latency degradation, and errors that were swallowed by retries stop being invisible.
• Coverage: new services are connected via a configuration template, so the share of observable services grows without a separate project.

Separately, we verify that the pipeline has not become a single point of failure: when the receiver is unavailable, agents write to a local buffer and after recovery catch up with the flow without manual actions.

OpenTelemetry Collector: deployment options in a cluster

Deploying the collector in a cluster is primarily a choice of the point at which telemetry is enriched and filtered.

The same OpenTelemetry Collector is installed as an agent on each node, as a sidecar container next to the application, or as a shared gateway — or not installed at all, and the service sends data directly.

The working scheme is almost always hybrid: node-level collection provides a cheap stream of logs and metrics, while the centralized gateway takes on the heavy lifting — tail sampling, masking sensitive fields, routing telemetry streams across backends.

The choice of point determines where data is lost during a failure and who is responsible for its delivery.

What to choose for the task

The options are combined: the agent and gateway connect via OTLP, and the sidecar container is kept where the service needs its own transformations before sending.

Implementation order: from source inventory to handover

We structure the work order so that before configuration changes the full traffic picture is visible, and after — it is clear exactly what changed.

We deploy OpenTelemetry Collector not “as in the documentation”: we select the set of receivers, processors, and exporters for the specific fleet of services.

1. Audit of telemetry sources. Inventory: which services and agents emit metrics, logs, and traces, in what format and how often. We separately record gaps — areas where data is not collected at all or is duplicated through different paths.
2. Data flow scheme. We design routes from receivers through processors to storages: where batching, where retries, where disk buffering. We agree on the scheme before configuration — otherwise the configuration grows chaotically.
3. Processing and filtering rules. We cut off noisy metrics and service attributes, set trace sampling, and split streams across different storages and teams. The rules live in the repository, not edited by hand on a running service.
4. Collection validation under load. We run real traffic, not test data: we look at the share of lost records, delivery latency, memory consumption, and behavior when storage is unavailable.
5. Fault tolerance and limits. We limit queues and memory, configure retries so that a short-term receiver failure does not turn into data loss.
6. Handover and training. We hand over configurations, the scheme, the change procedure, and analysis of typical failures: the on-call team adds a new source using a ready template.

Configurations, routing schemes, and procedures at the output

At the end of the work, the customer's team is left not with a set of one-off edits on the stand, but with a working telemetry pipeline: collector configuration, routing, processing rules, and the procedure by which all of this continues to live.

The files are stored in a repository under version control — changes go through review rather than being edited by hand on the cluster.

• Collector configuration — receiving data from applications and nodes, processing, export to storages; separate pipelines for metrics, logs, and traces so that the load of one stream does not delay the others.
• Data routing schemes — what goes where, by what rules streams are separated, and how delivery is configured when the receiver is unavailable.
• Filtering and enrichment rules — cutting off noisy events, sampling, adding environment and service attributes so that the source can be found from a record.
• Data access policies — which teams see which streams and how access to sensitive attributes is restricted.
• Operations procedure — the order of configuration updates, monitoring collector health, response to buffer overflow and delivery degradation.
• Telemetry documentation — description of streams, naming conventions, typical queries, and extension points for new services.

We separately record a flow map and hand it over along with the configuration: it shows which service sends what and which rules are applied to it. New sources are connected using a ready template — without revising the entire scheme.

Case study: how telemetry volume and failure investigation time were reduced

In one project, telemetry went to storage indiscriminately: all spans were saved, including health checks every few seconds, metrics with request identifiers in attributes, and duplicate logs from service replicas.

The OpenTelemetry Collector was running in simple forwarding mode, so useful records were about a quarter — the rest did not participate in alerts or failure analysis, but grew in volume and slowed down queries.

What was changed in the pipeline

We separated ingestion, processing, and export into distinct pipelines. Health-check spans are dropped at ingestion, and request and session identifiers were removed from attributes — they provided the main cardinality.

Infrastructure metrics are sampled down to the service level, traces are stored selectively: all requests with errors, retries, and anomalous duration remain in full. On top — batch buffering and a memory limiter so that data is not lost at the receiver during a traffic spike.

Result in numbers

The volume of telemetry in storage decreased by 4.5 times: 78% of the incoming stream turned out to be redundant. The time to find the root cause of a failure dropped from 90 minutes to 12–15 — an engineer opens the needed trace instead of sorting through matches by service name.

Measurement losses during peaks are gone: previously, queue overflow lost up to 5% of points, now not a single span with an error disappears. The rule “drop only what no one reads” worked: business-level metrics and everything involved in alerts are retained in full.

What to do if some services already send telemetry

Partially configured collection is a normal state: some services already emit metrics and traces directly, some are scraped, and the rest are not observable at all.

The danger is not fragmentation, but trying to switch everything at once — a pause in telemetry produces blind alerts at the moment they are most needed.

We start with an inventory: which collection schemes are working now, where data is duplicated, and which dashboards depend on them. Then OpenTelemetry Collector is installed alongside and becomes a second delivery address — it does not replace the working paths.

Services that already send telemetry over an open protocol change only the receiver address: code and libraries are not touched.

Those that exposed metrics via scraping are connected through service discovery by pod annotations — the new collection runs in parallel with the old one.

Connection proceeds by namespace or by team, not all at once: both schemes coexist until parity is confirmed. At each step we compare metric cardinality, resource attributes, and the absence of duplicates — this shows that the same data is arriving through the collector.

There is no gap in observability: buffering is enabled on the collector, and when the backend is unavailable, data catches up in the queue rather than being lost. Rollback is also cheap — we return the previous delivery address, which we agree on before migration begins.

Common technical concerns before implementation

Before implementing OpenTelemetry Collector, teams ask almost the same questions: what data collection costs the application, what happens if the gateway fails, how to keep volume under control, and how to separate streams between teams. Below are short answers to each.

What happens if the gateway is unavailable?

The collector runs as replicas behind a load balancer, so losing one instance does not break the stream. Agents keep a local buffer on disk and retry sending with increasing delay: data arrives with a delay but is not lost. During a prolonged outage, volume limiting is enabled to avoid overflowing the node's disk.

How to limit the volume of collected data?

Volume is set on the collector side: filtering rules cut off service and noisy metrics, and sampling reduces scrape frequency where dynamics matter more than precision. Data grows with the number of services only where this is intentionally built in.

Can access be separated between teams?

Yes. Each team is issued its own set of keys and rules by which the collector distributes data to different storages: the team sees only its own services. Shared rules are changed through review, so an accidental edit will not expose someone else's data.

Let's discuss the task: what to send for a preliminary assessment

To make the conversation about the telemetry pipeline substantive rather than a retelling of documentation, send the source data — based on it we will return with an estimate and work plan, without two weeks of back-and-forth. Below is what usually removes most of the uncertainty before the start.

• Clusters and responsibility: how many environments, where they are located, managed or self-hosted, who owns the nodes and their updates.
• Data sources: which services and languages, what is already instrumented, which metric, log, and trace collectors are working now.
• Aggregator configuration: working OpenTelemetry Collector files or charts — which receivers, processors, and exports are used, what was changed last.
• Volumes: number of active metric series, daily log flow, share of retained traces — an order of magnitude is enough, precision to the percent is not needed.
• Telemetry receivers: where data is written today, retention periods, security and environment isolation requirements.
• Access: roles and service accounts, network policies, restrictions on outbound connections.
• Reliability requirements: acceptable delivery latency, behavior under overload, who receives alerts and how they respond.

Describe the task in your own words and attach what you have: a complete set is not required. Our engineers will review the pipeline, show bottlenecks, and return with an estimate and work plan — before any changes in production.