End-to-end tracing breaks at service boundaries
Tracing holds exactly until the first boundary between services: an HTTP call inside the cluster is visible in full, but as soon as the request goes to a queue, a background job, or an external gateway, the context is lost.
Spans are not linked into a single trace, and a fragment of the path remains in the panel — up to the moment when the service passed the work on. End-to-end tracing turns into a set of disconnected segments, and they have to be matched manually by logs.
It costs the most during incident analysis. The on-call engineer opens logs of four or five services, compares timestamps by eye, and discovers that each component has its own request identifier.
The bottleneck is found by brute force: first the database comes under suspicion, then the connection pool, then the cache — and only by the end of the hour it turns out that the path was slowed down by one external call with retries.
Broken service observability does not let you prove the cause, so the degradation manages to repeat.
To keep the trace from breaking, context needs to be propagated through all transports that exist in the system, not only through synchronous calls.
New Relic APM collects distributed tracing across services, but the picture becomes complete only when queue handlers, background job schedulers, and external integration clients are instrumented as well.
A separate part of the work is linking log records to traces by a common identifier so that moving from a span to a log line takes one click.
We break down how this is closed: what to configure in the agent, how to tag services, and what checks to put in place so that the trace is collected in full, and not up to the first boundary.
What end-to-end observability across all services gives
End-to-end observability is when, from a single complaint, you can trace the entire path of a request: from the entry service through queues and external calls to a specific database query.
New Relic APM collects distributed traces, application metrics, and environment data into one picture, so the cause of a failure is found not by brute-forcing logs across five systems, but by the dependency graph. Below is what exactly changes in the team's work when such a link is already configured.
- Time to localize the cause. Tracing links segments by a single request identifier — instead of hours spent agreeing on "whose segment this is," the cause is found in minutes.
- Service bottlenecks. The top of the longest external calls and storage queries is visible, broken down by response time — it is clear what to optimize first and what does not affect the result.
- False alerts. A rule is built on a combination of signals (error rate + duration + throughput), not on a single threshold — triggers without a real deviation almost disappear.
- Class of hidden failures. Degradation in which everything is "green" but one segment responds three times longer is detected before users contact you.
- Recovery time. Each deployment is automatically tied to the release point: before-and-after comparison and rollback rely on facts, not guesses.
- Dependency map. Connections between services are built from actual traffic, not from documentation that has fallen behind reality.
- Unified context for analysis. Logs, metrics, and traces open side by side, so incident analysis does not turn into manually collecting exports from different systems.
The point is not a monitoring dashboard, but a change in process: we configure alerting rules, an analysis procedure, and service tagging so that observability works for the on-call team rather than existing on its own.
Ways to collect telemetry: agent, instrumentation, gateway
The method of collecting telemetry is chosen by one criterion: where we are ready to make changes — in node configuration, in application code, or in a separate service between the application and storage.
A monitoring agent is installed on a host or in a container, requires no code changes, and immediately provides requests, errors, and external call time; the price is keeping and updating it on every node.
Code instrumentation gives precise boundaries of business operations, but ties us to the language and to the development team's discipline.
A separate metrics collector is a compromise for a distributed data collection architecture: agents and libraries send data to it, and it decides what goes further, how it is enriched and filtered.
In New Relic, these options complement each other, so the choice is not "either/or," but "in what order."
| Option | When it fits | Limitations |
|---|---|---|
| Agent on a host or in a container | Telemetry is needed without code changes and for several languages | Access to nodes, agent versions must be kept in sync |
| Library inside the application | Business metrics and precise operation boundaries are needed | Code changes, language dependency, rebuild and release |
| Separate collector/gateway | Many sources, a single intake and filtering are needed | It is itself a service: scaling, updates, fault tolerance |
| Agent on a node plus a library | Both infrastructure and application data are needed | Metric duplication, name normalization is required |
The boundaries between options blur: an agent on a node solves the task in a day, while a library adds what is not visible from the outside. Next we break down how to combine them without duplicates in reporting.
How implementation goes: from schema to handover
Implementing monitoring rarely comes down to installing an agent: first you need to understand which services are connected to each other and where the request context is lost.
Therefore we go in stages — from surveying the environment to handing over procedures, so that the observability schema works without us.
- Environment survey. We inventory services, environments, and entry points, and record what already provides metrics and what will have to be covered manually.
- Observability schema design. We agree on what we consider a separate service, how we link a request between them, and how long we store data — the retention period is chosen for failure analysis.
- Agent configuration. We install the New Relic agent next to the application, enable default collection, and selectively add manual measurements where automation does not see a business operation.
- Distributed tracing. We propagate context through services and queues and configure the share of retained requests: the sample remains representative, while data volume does not grow uncontrollably.
- Dashboards and alerting rules. We build dashboards for the on-call engineer, tech lead, and business unit manager, and configure rules for deviation from the norm rather than for every spike.
- Load testing. We run a profile close to production and make sure that data recording does not eat up throughput and does not add latency noticeable to the user.
- Operations procedure handover. We document who reacts to an alert and how, how changes are rolled back, and how a new service is connected to the schema.
Access policies, roles, and data separation
Access separation in a tracing system is not bureaucracy, but a condition under which investigation results can be trusted.
One shared account with full rights in New Relic nullifies the audit: it is impossible to understand who changed an alert rule and who exported traces with request parameters.
We build observer roles from the team structure: a developer sees their services, the on-call engineer sees the whole perimeter, and an external contractor sees only an agreed area.
The customer receives:
- a role matrix tied to teams and services: who views metrics, who views traces, who manages rules;
- data separation and masking policies: request parameters are cut before recording, not after;
- a data flow diagram from the metrics collector to storage, indicating anonymization points;
- an on-call and escalation procedure: response times, shift handover order, criteria for a nighttime call;
- operations documentation: access key rotation, rights review, connecting a new service to tracing;
- an action audit log and an incident analysis instruction with links to specific dashboards and queries;
- an acceptance checklist by which the configuration can be checked without us: queries, filters, visibility zones.
Setting up alerts without noise: thresholds and groups
By default, teams set static thresholds "by eye": response time above a conventional value means an alert.
During peak hours, such a threshold triggers dozens of times, the on-call engineer gets used to ignoring the channel, and a real failure arrives together with the noise.
This is treated not by disabling rules, but by separating conditions by the nature of the metric — in New Relic this is done at the level of policies and conditions.
Resource metrics — disk utilization, connection pool exhaustion, queue growth — we keep on static thresholds: they have a physical limit, and deviation is always significant.
Traffic-dependent ones — response time, error rate, throughput — we move to baseline thresholds with a smoothing window of several days and a requirement for the deviation to persist longer than one measurement. This removes triggers on a single spike and leaves those confirmed by neighboring service metrics.
The second source of noise is one failure generating forty alerts.
Grouping incidents by entity and key attributes collects related triggers into one incident with a shared context, and separation into critical and warning levels prevents background events from drowning out critical ones.
We deliberately set the warning threshold below the critical one — it works as an early signal, not as a reason for a nighttime call.
Alert routing relies on tags: each entity has a service owner, and delivery rules distribute triggers across on-call shifts — critical ones go to the on-call channel with escalation, warnings to the team channel, and the nighttime background stream is rolled up into a morning digest.
Noise suppression includes muting rules during planned maintenance and release deployment, so that an update does not look like an outage. The result is that the share of alerts that actually lead to action grows, while time to response falls.
How does request tracing differ from metrics?
A metric answers the question "what and how often," a trace answers "where exactly and why." Choosing "metrics or traces" is a false fork: these are sequential tools, not competitors.
Aggregated metrics show that a problem exists and in what time window, while request tracing leads to the specific path segment that creates it.
Aggregated metrics are calculated across all requests at once: error rate, 95th percentile response time, requests per second, connection pool saturation.
They are cheap in volume and good for observation — they show that after a deployment the failure rate rose or the latency percentile dropped.
What they lack is a link to an individual request: a metric will say that it is slow, but will not show at which processing step hundreds of milliseconds were lost.
A trace is a record of one path: from the entry point through internal services to the database and external calls, with the duration of each segment and the link between them.
It preserves error context — which handler failed, with what parameters the request arrived, what was happening in neighboring services. For failure diagnosis, you first narrow the window by metrics, and then open traces for exactly that interval.
In New Relic APM, both layers are linked in one picture: distributed tracing glues together path segments between services, and metrics for them land in the same charts used for on-call duty.
Analysis goes top-down — from "the error rate on payments rose" to "the external gateway call stopped fitting into the timeout."
Case: analyzing degradation of a payment service
The payment service processed payments through a chain of four segments: request intake, anti-fraud check, call to an external provider, and result recording.
The average response time at the entry point stayed normal, but for some operations it jumped several times over — this was not visible on the summary chart.
Before: tracing broke in the middle of the chain
Tracing was collected only on the first two segments: further on, the request context was lost because the intermediate layer did not pass headers.
New Relic received disconnected transactions without a link to a specific payment, and analysis of each case turned into manually matching logs by timestamps.
Time to find the cause reached 4–6 hours, and almost half of the analyses ended with the conclusion "could not reproduce." Changes to anti-fraud were made blindly: it was not visible what added latency — the check or waiting for the provider's response.
After: end-to-end linking by payment identifier
We passed the payment identifier through all segments and enabled distributed tracing on the intermediate layer, and split errors by type and segment — provider failure stopped being mixed with an internal response parsing error.
- Time to find the root cause per incident: 4–6 hours → 10–20 minutes.
- Share of cases with the cause found during the first analysis: about half → nine out of ten.
- Anti-fraud changes: by hypotheses → by breakdown of latencies by segment.
The effect came not from telemetry itself, but from trace continuity: a break in one segment devalued the data from all the others.
Technical doubts before starting work
Objections before implementing end-to-end tracing are usually not about capabilities, but about the cost of operation: overhead, compatibility, and data privacy. We break down four that most often stop a project.
Will data collection overload the services themselves?
The agent takes metrics continuously, while distributed tracing is enabled selectively — on critical paths and with sampling. We keep overhead within a few percent of response time, measuring it before and after on the same load profile. For latency-sensitive services, we narrow tracing to individual operations.
What should be done with legacy services?
Rewriting them is not necessary: compatibility is ensured at the environment level — collection through a proxy and logs, not through an agent inside the application. This way old services get into the overall tracing picture even if no one touches their code.
How should data privacy be handled?
We configure masking on the agent side: personal fields, identifiers, and request bodies are cut before sending, while operation names and timings go out.
Masking rules live in the configuration and are protected by checks — when the data schema changes, they do not diverge from reality.
How long are traces stored, and how do you limit volume?
We set retention by policies: raw traces live for a limited period, aggregated metrics for longer, because they take up less space and are needed for trends. After enabling sampling and age-based cleanup, storage volume stops growing linearly.
Let's discuss your observability task
It is worth starting with a description of the task: which services participate in request processing, where context is currently lost between them, and what question needs to be closed — finding the source of errors, analyzing slow segments, or controlling response time along the entire path.
For a preliminary review, a list of services, their runtime environments, and the environment schema are enough: separately the development, testing, and production perimeters.
The estimate of work is not built by eye. We count the number of entry points, the depth of the call chain, the volume of request flow, and the required share of covered routes.
If observability is already partially deployed, we look at where data collectors are installed and what context gaps remain — this shows whether additional configuration is enough or whether the logic will need to be moved.
Based on the review, we come back with an implementation plan and environment schema: the order of connecting services, stages, areas of responsibility, acceptance criteria, and rollback points.
What is usually needed at the input to move to the start of work:
- a list of services and connections between them, including background jobs and queues;
- an environment schema indicating which perimeter is where and how they communicate with each other;
- access to observability systems and log storage — service accounts with read-only rights;
- runtime environments and versions of libraries that form the trace;
- a list of already connected data sources and configured alerts;
- expectations for trace retention depth and search speed;
- an employee who will confirm acceptance criteria on the customer side.
Describe the task in whatever way is convenient — a short email or a meeting. We will come back with an estimate, a plan, and a clear scope for the first stage.