Why tracing is lost: what breaks without Jaeger
In a distributed system, a single request passes through a gateway, several services, a queue, and a database.
But there is no common thread between them: the request ID is not passed further down the chain, timestamps on different nodes diverge, and some calls go asynchronously and do not appear in the caller's log at all. It is impossible to assemble the request path from this — one can only guess where it broke.
Then hypothesis guessing begins. An engineer looks at metrics, pulls up service logs one by one, manually compares time windows — and still does not see at which exact transition the request degraded.
Lost traces mean that a rare but regular failure reproduces intermittently, and the cause remains an indirect guess.
The cost of such failure diagnostics is not only the on-call engineer's hours. While the source is not found, latency grows, errors accumulate on the client side, and changes are made at random: a fix that 'was supposed to help' closes the symptom and masks the cause.
End-to-end tracing addresses this differently: each request receives a single identifier that is passed along the entire call chain, and Jaeger builds a tree of operations with the duration of each from this data. Below we discuss how this is deployed and what it provides during a real incident.
What end-to-end tracing of a distributed system provides
In a distributed system, a single request passes through several services, and during a failure logs show only fragments: you can see that a service responded slowly, but not why.
Observability without end-to-end tracing comes down to manually checking services one by one, and every minute of searching is a minute of downtime.
End-to-end tracing collects the entire request path into a single chain with the duration of each step. Jaeger stores such chains and breaks the request down into intervals: which call took most of the time, where a queue accumulated, which dependency responded with latency.
An engineer opens a specific slow request and sees the bottleneck by time and number of calls — without manually going through services.
The practical effect is measurable: finding the cause of a failure is reduced from hours of hypothesis testing to minutes, and the share of failures that cannot be explained drops — the chain either shows the problematic area or proves that the issue is outside the system.
The class of failures being investigated also changes. Instead of 'sometimes everything is slow,' there is a specific area with excess time, tied to a version, configuration, and load volume. Recovery time is now determined not by finding someone to blame, but by fixing it.
Tracing complements metrics and logs rather than replacing them: metrics say what got worse, logs say what happened, chains say where exactly. We enable it in stages so that the trace flow does not grow uncontrollably and does not affect service response time.
Jaeger deployment modes: comparison of options
The choice of deployment mode determines not only how quickly a trace appears in the UI, but also how many resources to allocate in the cluster and how to separate environments.
Service instrumentation itself does not depend on this, but trace storage, fault tolerance, and scaling depend entirely on it. Below are the options that teams actually choose between at the start and as the span flow grows.
| Option | When it fits | Limitations |
|---|---|---|
| Single process with in-memory storage | Local debugging, verifying instrumentation correctness, demos | Traces disappear on restart, no horizontal scaling, not suitable for a shared environment |
| Collector and Elasticsearch storage inside the working cluster | Medium-sized team, retention depth of 14–30 days, environments separated by namespaces | Resources are shared with production services, retention settings and replica count are mandatory |
| Separate tracing environment on Cassandra | High span flow, months-long retention, separate operations environment | More expensive to maintain, requires expertise in schema and upgrades, harder to deploy environments |
| Collector as a buffer (Kafka) in front of external storage | Peak loads, storage degradation should not bring down trace ingestion | Traces appear with delay, failure points are added, queue monitoring is needed |
| Managed tracing service | Small team without a dedicated storage on-call | Less control over schema and retention, vendor lock-in |
The collector does not store state: its instances scale horizontally behind a load balancer, and the agent on each node buffers spans and survives a restart. Fault tolerance rests on the storage — replicas and quorum are configured on the database side, not on the tracing side.
Environment isolation is best done with separate collector instances and separate indices per environment; otherwise, test traffic blurs the sample and inflates storage volume.
The sampling rate is set separately for each environment: in development we write everything, in production — a share with guaranteed retention of slow and erroneous requests, otherwise incident investigation is left without data.
Jaeger implementation: stages of engineering work
We start with inventory, not deployment: until it is clear where a request originates and which calls cross service boundaries, the tracing scheme will be incomplete. The order of work below is what we go through on every implementation, from assessment to handover of policies.
-
Service and entry point assessment. We record the list of services, communication protocols, and places where context is lost: queues, background jobs, external calls. The output is a map showing which parts of the path are currently not observable.
-
Trace flow scheme. We agree on a unified format for passing context between services, and rules for naming operations and attributes. Without this, traces end up as disjointed fragments and do not show the full response time.
-
Jaeger deployment. Ingestion collector, storage, search UI: we separate ingestion and writing, set trace retention and cleanup of old ones, and decide where to place agents — on nodes or next to the application.
-
Service instrumentation. We enable automatic collection on standard libraries, mark critical areas manually so that business attributes appear in the trace. Here we also set sampling rules: continuous collection at peak inflates storage, sparse collection loses anomalies.
-
Load testing. We run traffic close to production and look at three things: the share of joined traces, overhead on the service side, and storage behavior at peak. We adjust sampling rules based on actual results, not calculations.
-
Handover of documentation and policies. We hand over flow diagrams, configuration, monitoring dashboards, alert rules, and the procedure for degradation. The team adds a new service to observability without us.
After that, only edits remain: connecting a service or changing storage is a configuration change, not a system rebuild.
Handover package: tracing schemes, sampling policies, and procedures
We hand over the package not as an archive of settings, but as a working system: each document has an owner and a use case. Below is what remains in your infrastructure along with the deployed Jaeger and how to use it without us.
- Collection configurations — trace ingestion and processing files with sampling rules: which services are traced in full, which by request share, which are excluded due to volume.
- Trace flow scheme — a diagram of the data path from service to storage and search UI, with marked failure points and places where a trace is lost.
- Access policies — roles and visibility scopes: who sees all services, who sees only their own, how rights are granted and revoked, where keys are stored.
- Retention rules — trace lifetimes by type, early cleanup conditions, index partitioning, and compression; separately, a long retention period for traces of investigated incidents.
- On-call shift instructions — how to check in a minute that tracing is alive, what to do when ingestion is delayed, and how to roll back a configuration change.
- Scheduled check procedure — reconciling storage volume, monitoring the share of dropped traces, end-to-end test after service updates.
- Instrumentation map — services and libraries with versions, so that updates do not lose compatibility and do not leave you without traces.
- Change log — who changed sampling rules and why; without it, in six months no one will be able to explain the disappearance of traces for a specific service.
Case study: how the time to find the cause of a failure was reduced
A platform of several dozen services, with some communication via queues. During a failure, the on-call engineer saw a spike of errors at the entry point, but the cause was often in the third or fourth service down the chain, and each service's logs lived separately. There was no single request identifier — events were linked manually, by time and indirect signs.
Localizing the cause took 3-4 hours, and another similar amount to confirm the hypothesis. About a third of incidents were closed without an explained cause: the symptom was removed, the root cause was not found, and a few weeks later it appeared elsewhere. Average recovery time remained at several hours.
What we changed
We implemented Jaeger and an end-to-end request identifier that is passed through all calls, including asynchronous ones. We instrumented the places where time was most often lost: database calls, external calls, enqueueing and processing messages in queues.
Sampling was configured adaptively — slow and rare requests are retained in full, while the stream of routine ones does not inflate storage.
Results
A link to the trace is attached to the incident ticket — investigation starts with a fact, not with enumerating versions. Localization was reduced to 10-15 minutes, recovery time — several times over, and the share of incidents without an explained cause fell to isolated cases.
Recurring failures became visible in the overall call picture, and they are closed during review, not after the third report.
How Jaeger differs from log and metric collection
Logs and numeric indicators answer different questions, but both look at the system in fragments. A metric shows that latency increased or the share of successful responses fell — but does not say at which step this happened.
A log records an event with the text and context of an individual service, and in a distributed environment it has to be manually matched with neighbors' records by time.
Tracing in Jaeger is built around request context: it is assigned an identifier that is passed through all calls, and each step leaves a record with duration and a link to the parent step.
The result is a tree — the request path between services in full, including asynchronous branches through queues and retries.
The boundary of applicability is simple. Logs are indispensable when you need the exact error text or field value. Numeric indicators — when trends, alerts, and storage cost matter: they are compact and cheap over a long horizon.
The request path is needed where time losses arise at the junction of services: a metric finds the spike, tracing points to the specific step and its share in the total delay, and the log by the same identifier gives the cause.
They complement each other rather than replace: the combination 'spike → specific step → error message' reduces finding the cause from hours to minutes.
At the same time, there is no need to trace everything — input sampling sets the share of recorded requests, and filtering by errors and slow responses leaves exactly those worth investigating.
Technical concerns: load, data volume, compatibility
Concerns about load and storage volume arise when tracing is already enabled in several services and it is visible how much data each operation brings. Below are answers to questions engineers ask before connecting the remaining services.
How much will response time increase?
The additional work is recording intervals and passing context between services. When sampling a share of the request flow, the increase stays within a few percent, and on critical paths we set separate priority rules. We take measurements before and after enabling in production, rather than estimating by eye.
How quickly does storage volume grow?
Volume is determined by three things: the number of instrumented operations, the sampling coefficient, and retention period. We keep raw intervals for a few days, aggregates — longer, and configure a deletion policy for each service name.
The registry stops growing uncontrollably, and investigating an incident from last week remains possible.
Is this compatible with our environments and stacks?
Collection uses a unified format for containers, virtual machines, and serverless functions: the agent is placed next to the application, so moving between environments does not change the data schema.
The stack is not critical either — instrumentation libraries exist for major languages, and we cover non-standard wrappers with manual intervals or an intermediary gateway.
What to do with partially instrumented services?
We connect them in stages: the edge points — load balancer, gateway, database — provide an end-to-end picture immediately, and intermediate links are added as code changes are made.
A gap in the chain appears as a separate interval and is marked in the schemes as an uncovered area, but this does not hinder finding the bottleneck.
Can collection be reduced without rebuilding services?
Yes. Sampling and ingestion are managed on the collector side: under abnormal load, it is enough to reduce the coefficient or temporarily disable the source — applications continue to work. Returning to the previous agent configuration also does not require a release.
Let's discuss the task: what will be needed from your side
The agent collects traces and sends them to Jaeger, but to deploy the collector and connect it to your services, we need access to the infrastructure and an understanding of how the environments are structured.
Below is what we request at the start of work: this is enough to come back with a plan and estimate, without clarifying rounds in the middle of implementation.
- Access: accounts to the cluster and image registry, rights to create objects in a separate namespace — without rights to modify production workloads.
- List of services: which applications we instrument, what languages and libraries they are written in, whether there is already trace export in any form.
- Environments: how many environments — development, pre-production, production; whether Jaeger is needed in each or one is enough with separation by department.
- Retention requirements: how many days traces must remain available for incident investigation and who accesses them.
- Change approval procedure: when service configuration edits can be rolled out, who confirms, whether a one-step rollback is needed.
- Service connection scheme: whether there are proxies and load balancers to account for so that context is not lost.
- Acceptance owner: an engineer who will verify that traces are collected end-to-end and can be found by request identifier.
Describe the task and send this data — we will come back with a work plan: stages, boundaries of our responsibility, rollout order by environment, and points where your confirmation is needed.