Distributed tracing: what breaks without Tempo and Grafana
While a system consists of a single service, the question “where exactly is it slow” is solved by looking at the application log.
As soon as a gateway, several services and a message queue appear, metrics show that response time has increased, but not where in the call chain this happened.
The answer is spread across logs on all servers, and the connection between them is visible only by request ID, which is not always propagated to the end.
Incident investigation turns into manual assembly: an engineer unloads logs from several nodes, correlates timestamps, reconstructs the order of calls — and still does not see calls that left no records.
With a stream of failures, this is hours of work by several people, and the cause is not always found: more often, a segment is found that simply ends up under suspicion more often than others.
Monitoring blind spots are usually where a call is not synchronous: background jobs, publishing to a queue, retries after timeout. One failing node produces a series of retries, and by metrics it looks like an overall increase in load rather than a specific failure.
Without distributed tracing, such cascades are diagnosed after the fact, when degradation has already affected users.
Distributed tracing closes this: the call chain is assembled across all services, including asynchronous transitions, and incident investigation starts from a specific slow segment rather than from going through logs.
Next, we’ll look at how this is arranged on Tempo and Grafana and what is needed for visibility between services to appear not just in theory.
What end-to-end tracing provides: response time and failure investigation
End-to-end tracing links one user request to all calls it generated: from the ingress gateway to the queue, cache and database. Instead of a dozen disparate logs, you have one route with the duration of each segment.
Tempo together with Grafana stores these routes and allows searching through them — by response code, operation name, duration.
The first measurable effect is time to localize the cause. While investigation is done through logs, the engineer goes through services manually, and an incident takes about an hour.
Tracing shows the problematic segment immediately: you can see at which step the request lost time or failed, and investigation fits into 5–10 minutes.
At the same time, the share of detected failures grows — some failures leave no record in application logs, but remain in the trace as a broken segment.
The second effect is reduced downtime. The faster the culprit is found, the shorter the window when users receive errors.
On a typical incident, the difference between manual search and working from a trace is tens of minutes of unavailability: rollback or switching to the standby branch is started while the picture is still fresh, rather than after a long investigation.
The third effect is a whole class of failures that is otherwise almost impossible to catch: rare slow queries, degradation under load, errors at service boundaries. By individual metrics they look normal, but in the trace you can see that one segment consistently adds latency.
Investigation stops being guesswork, and such failures enter the statistics instead of resurfacing repeatedly.
Connecting Tempo to Grafana: comparing deployment options
Grafana connects to Tempo as a standard HTTP data source, and the integration itself looks the same in all scenarios. The fork is elsewhere: where trace blocks physically reside, who maintains the storage and for what period it is designed. This determines retention, load on services and how environments are separated.
Options for placing trace storage
| Option | When it fits | Limitations |
|---|---|---|
| Local disk on a single node | Sandbox, debugging, small flow | No replication, retention limited by disk size |
| Object storage in monolithic mode | Medium flow, full control over data needed | One process handles both ingestion and block compression |
| Distributed mode on top of object storage | High flow, long-term storage | More components — scaling and maintenance are more complex |
| Managed service | No in-house operations team | Data outside your perimeter, separation is set by the provider |
Retention is set by a policy at the block level: a background component deletes blocks older than the specified window, and rules can be tied to different tenants.
This way, product traces live for months, while the debug flow from the test environment is cleaned out within days — without a separate cluster for each.
Load on services is determined not by the storage option, but by the volume of spans sent: the ingestion buffer smooths peaks, but it is not infinite, so sampling and per-tenant limits are set.
Environment separation is done either with separate Tempo instances for prod and stage, or with one instance using different tenant IDs — the second path is cheaper to maintain, but requires discipline in Grafana access rights.
Tracing implementation stages: from service audit to handover
Before connecting data collection, you need to understand how the system is actually structured: for most teams, the call map diverges from reality, and some services have never been instrumented. The work proceeds in five steps — from inventory to handover of a ready maintenance setup.
- Audit of services and connections. We collect a list of services, queues and external dependencies, note who calls whom and where calls are synchronous. The output is a list of services without instrumentation and “blind spots” on the critical path.
- Request flow diagram. We draw the end-to-end path of a request from entry point to storage, mark key measurement points. We agree the diagram with service owners; otherwise the trace map will diverge from real connections.
- Setting up trace collection. We deploy ingestion and storage (Grafana Tempo), connect applications, set common service and operation names, retention and sampling rate. Without shared naming rules, segments of one request do not merge into a trace.
- Validation on a pilot group of services. We take several services on the critical path and look at overhead, completeness of connections, and time to find the cause of a failure. We close gaps before rolling out to the entire system.
- Maintenance regulations and handover. We fix naming rules, retention periods, typical on-call queries and the procedure for connecting new services. The customer’s team receives documentation and a breakdown on live incidents.
Handover package: configurations, diagrams and regulations
Tracing brings value as long as the team understands how it is assembled and why it behaves this way as load grows.
We hand over not only a working setup, but also a package with which your engineers can change sampling themselves, add services to routes and investigate incidents without our support.
- Receiver, storage and collector configuration files — with comments on what each parameter is responsible for and which values can be safely changed without rebuilding the setup.
- Request route diagrams: how a trace goes from service to storage, where it is lost under overload, which sampling rules apply to which services.
- Access policies: roles of developers and on-call engineers, separation by namespaces, procedure for issuing and revoking keys when an employee leaves.
- Retention regulations: trace lifetime, what is cleaned on schedule and how the registry volume grows when load doubles.
- Instructions for the on-call shift: where to look when traces disappear, how to distinguish storage overflow from collector failure and which actions are safe at night.
- Checklist before acceptance: what confirms that a trace reaches from service to query, and how to reproduce each check manually.
- Map of services and owners: whom to contact when changing the tag scheme or adding a new service to a route.
The package remains alive: we update it together with changes to the setup and review it with your team in a separate session. After it, the on-call engineer resolves a typical incident on their own, without waiting for us.
Case study: incident investigation reduced from two hours to ten minutes
Night incident during peak load: the gateway started returning errors, an alert fired, the on-call engineer opened the logs. What followed was almost two hours of finding the cause of the failure.
Some requests did not reach the application log, some were lost on the cache side, metrics showed growing latency but did not answer which call was creating it.
The cause was sought by iterating through hypotheses: one internal service was waiting for a response from a second, and that one was queued on a connection pool. The fix itself took about twenty minutes; the rest was spent understanding where to look.
What was changed
Enabled distributed tracing on Tempo: services send spans through a shared collector, the request ID passes through all calls, including asynchronous ones.
Logs and metrics were linked to traces by this ID — instead of switching between three panels, the engineer opens one trace and sees at which segment time increased. Alerting rules provide a link to slow traces for the same interval, so investigation does not start from scratch.
Metrics before and after
| Metric | Before | After |
|---|---|---|
| Finding the cause of a failure | ~100 minutes | 8–10 minutes |
| Investigation without escalation to a senior | isolated cases | most incidents |
| Picture for a request | logs and metrics separately | trace, logs and metrics on one axis |
A separate class disappeared from the set of failures — intermittent latencies that did not reproduce outside load and had previously been handled by guesswork. Now such an incident is confirmed by a trace within minutes, rather than remaining in the list of unresolved ones.
What to do if tracing is already partially deployed?
Partial implementation is more a rule than an exception: tracing is enabled on one or two services to validate the approach, and then the setup becomes overgrown with segments without a common scheme.
Data flows, but the picture does not add up: some services send detailed traces, others only errors, while still others lose the ID when passing the request further along the chain.
There is no need to rebuild everything from scratch — it is important to find the gaps and complete what already works.
We start with inventory: which services send data, in what way and with what detail, where it ends up.
If there are several receivers — part of the flow goes to Grafana Tempo, part to another collector — we do not run both setups in parallel, but consolidate the data into one so that search by ID returns the whole chain, not a fragment of it.
Consolidation rests on three things: a single service name across all sources, end-to-end context propagation through queues and background jobs, identical sampling rules and retention periods.
Incomplete traces most often occur precisely at asynchronous transitions — where context is not propagated or is broken at an intermediate handler; we find and close such places pointwise, without rewriting the service.
We move settings without stopping collection: sampling rules, retention periods and labeling are transferred from the already working setup rather than configured anew.
After that, refinement of the existing setup comes down to connecting missing services and checking connectivity — after it, every trace is readable from the entry point to the last call, and by time you can see which segment holds the latency.
Technical doubts: retention, load and access
Before starting, the same questions come up — not about tracing itself, but about how it will coexist with the infrastructure. We answer briefly to remove them before estimation.
How much trace volume should be stored, and will it grow too much?
Volume is determined by the sampling rate and the number of requests. The rest is decided by the retention policy: hot-tier period, thinning of old blocks, offloading to cold storage. This way, volume stops growing along with the number of services.
Will collection add load to the services themselves?
The agent sends data in batches and, when the channel is insufficient, drops part of it rather than slowing down request processing. Overhead is fractions of a percent of CPU time. For latency-sensitive services, we enable collection only for slow and erroneous requests.
Will all services be included in the trace picture?
Coverage depends on libraries in the application: current stacks are instrumented automatically, without code changes. Some systems we connect pointwise — we draw up a coverage plan at the start and immediately note what remains out of scope.
What to do with legacy systems without tracing support?
For them, we install a proxy that restores context from incoming request headers. The legacy component becomes a link in the chain, not a hole in it. The rule is simple: if it accepts a request, context can be passed further.
How to separate access between teams?
Through roles in the observability system: a team sees its services and its data sources, the on-call engineer sees the whole picture, but without the right to change collection rules. Data from different teams is not mixed, and the request history remains under control.
Let's discuss the task: what is needed to start work
To come back with an estimate and work plan, rather than clarifying questions, it is enough to send five things. Access to the working environment is not needed at this step — a description and diagram are enough.
- Task description. What hurts: where context is lost between services, how much time is spent finding the cause of a failure, which questions about the system remain unanswered today.
- List of services. Which components participate in request processing, what they are written in, how they communicate with each other — direct calls, queues, background jobs. Separately note which ones already emit metrics and logs.
- Environment diagram. How many environments there are and how they differ in service composition and load. Usually tracing is implemented in the test environment, and production is connected after validation.
- Investigation priorities. What to investigate first: errors on external calls, slow processing segments, retrospective incident investigation or monitoring operation execution deadlines.
- Constraints and access. Security policy, data storage requirements, where service configuration cannot be changed, who approves rights. This affects the collection scheme more than data volume.
- Current tooling. What is already in place and how to integrate tracing via Tempo and Grafana alongside it, without duplicating collection.
The first step on your side is a short conversation on these points, usually 20–30 minutes. Then we come back with a plan: trace collection scheme, data volume and resource requirements, work stages and what you will see after each stage.