AI agent tracing: where control over the session is lost
An agent responds to a client incorrectly — and it is only discovered from a complaint.
The interface shows the final text, the application log shows a successful service response, and between them there is emptiness: it is not recorded which steps the agent performed, which tools it called, and at which intermediate decision it went wrong.
Incident investigation starts from scratch and often ends with the phrase "could not reproduce."
Without AI agent tracing, the agent's call chain is not saved as a single whole. A search query was sent with one wording, an outdated document entered the context, the model asked itself twice — each step lives in its own log or nowhere.
They can only be correlated by time manually, and only if request IDs match, which usually do not exist.
This leads to two consequences. An unexplained failure cannot be distinguished from an external service failure: whether search, the prompt, or the task formulation itself is at fault is unclear.
And session reproducibility is zero: on repeat, the agent responds differently because neither the prompt version, nor the set of passed documents, nor the order of calls is fixed.
As long as this data is not recorded, any change to agent behavior is guesswork: whether it got better or just got lucky on a different set of examples.
Next we explain how this is closed — what exactly goes into the session trace, how the chain is reconstructed from it, and how to distinguish model degradation from a tool error.
What observability of tool call chains provides
Without tracing, an agentic system behaves like a black box: the user received an incorrect answer, inside there are a dozen external service calls, and which one failed is unclear. Reproducing the situation also fails: on the same request the agent may behave differently.
Observability of agentic chains records every step: which tool was called, with what arguments, what was returned, how long the step took, and where the chain broke.
Incident investigation turns into reading a single trace instead of iterating through hypotheses across logs from different services — investigation time drops from hours to minutes.
The share of explained failures is the second measurable value. Instead of "the model made a mistake", a cause appears: external service unavailability, truncated context, wrong tool selection, exhausted step limit. As this share grows, the team fixes classes of failures rather than individual cases.
Run repeatability makes results comparable. The same set of requests is run on an unchanged configuration, so discrepancies are explained by changes in the system rather than randomness — versions before and after the change are compared on the same slice.
From this grows answer quality control: a regression set of requests with expected results, the share of correct answers, the number of steps per request.
A change to the system instruction or connecting a new tool is tested on the set before rollout — degradation is visible in advance.
AI agent tracing: comparison of data collection modes
The data collection mode determines what will go into the trace at all, and it is chosen before implementation, not after the first incident investigation. Below are the options engineering teams actually choose between, with their applicability and limitations.
| Mode | When suitable | Limitations of the approach |
|---|---|---|
| Interception of calls in agent code | Own code: there is access to wrappers around models, tools, and memory accesses | Sees only instrumented areas; during refactoring, wrappers must be maintained, otherwise steps fall out of the trace |
| Boundary layer at the perimeter | Closed and third-party services: an independent trail is needed without modifying their code | Records only network exchange — internal steps, memory state, and intermediate decisions remain outside the log |
| Parsing application logs | Legacy and services without access to code, where there is no other source | Delay between action and record; links between steps are reconstructed by guesses, and parsing rules break when the format changes |
According to postmortems of major failures Google SRE, more than half of incidents are related to changes and incomplete observability — therefore in practice call interception and the boundary layer are combined: the first provides the agent's steps from the inside, the second provides independent verification of what went beyond the perimeter.
Log parsing is left as a fallback for services that cannot otherwise be inspected, and as a second source during reconciliation.
The main limitation of any mode is incompleteness: if a step is not recorded, it cannot be reconstructed after the fact, so the data composition is fixed at the trace design stage.
Implementation stages: from session audit to handover of regulations
AI agent tracing is not turned on with a single switch: events are born in different places — in model calls, tool accesses, and user sessions. Therefore implementation proceeds in stages, and each ends with an artifact that can be verified.
- System survey. We inventory services, queues, and external calls, and find places where the session ID is lost. The output is a flow map and a list of points that need to be instrumented.
- Collection schema. We agree on the event structure: required fields, correlation by session ID, what we sample and what we store in full. Separately, we decide where personal data is de-identified — before recording, not after.
- Setup. We bring up the event receiver, connect agents and tools, set retention period and access control. We check that recording does not lengthen the agent's response.
- Validation on real scenarios. We run typical and failure chains: tool failure, timeout, retry, reasoning loop. Each failure must be readable in the trace in full, not as a fragment.
- Incident investigation together with the team. We show how to go from a metric anomaly to a specific agent step in minutes. This is the check that the data is suitable for work.
- Handover of regulations. We hand over field descriptions, naming rules, retention policy, and incident investigation procedure. After that, the team runs the system itself, and we get involved when the schema changes.
Configuration and artifacts that remain with the customer
We hand over not a report but a working configuration: tracing lives in your repository and is deployed by your own delivery tools. Otherwise, in six months, you will be unable to investigate an incident or change recording detail without an external team.
- Collection configurations. What the agent writes to the trace — tool calls, arguments, responses, branches, errors; level of detail and sampling rules: some interactions are stored in full, the rest only as metadata.
- Data flow schema. From agent call to storage: where events are written, where they are aggregated, how long they live, what goes to the external perimeter and what does not leave yours.
- Access rules. Who sees raw traces in full, who sees only aggregates and metrics; accounts are taken from your user directory, no separate list is created.
- Field masking. Personal and sensitive data is cut out at the collection stage, before recording, not cleaned up after the fact: what did not get into the trace cannot leak.
- Team instructions. How to read a trace and find the root cause of a failure, how to add a new agent step to tracing, how to investigate an incident step by step.
- Regulations and alerts. Threshold values for error rate and response time, monitoring dashboards, and alerting rules are stored as configuration, not as manual settings in the interface.
All of the above remains in your environment, under your access control, and with documentation in Russian: any of your engineers will be able to develop tracing, not only the one who implemented it.
Case study: incident investigation in hours instead of weeks
The customer is a product team with a conversational agent that handles customer requests and itself calls internal services: billing, delivery, stock checks.
Eight out of ten dialogues proceeded normally, but investigating the remaining ones turned into manual log sifting and correspondence between support and development.
Symptoms came as "the agent answered incorrectly" or "the order hung". The failure could be reproduced in about one out of five cases: neither input data, nor the sequence of calls, nor external service responses were saved.
Localizing the cause took working days, and support turned each request into a manual reconstruction of the dialogue.
What was changed
We linked every agent step — plan, tool call, model response — into a single trace and tied it to the request ID. The trace storage was deployed on Langfuse: search by dialogue, input parameters, and error code appeared immediately, without a custom investigation interface.
Around the traces, we built replay — the incident is replayed with the same inputs on an isolated environment, including recorded external service responses.
Metrics before and after
- Share of reproducible failures: 20% → 85%.
- Time to localize the cause: 2-4 working days → 2-4 hours.
- Manual support work decreased threefold: typical requests are closed using the trace, without a developer.
- Escalations to engineers decreased fourfold — only those requiring a code or prompt change remain.
Separately, we kept recording of failed runs: a new defect is now tested against accumulated traces as a regression set, so the reappearance of the same class of failure is visible at the rollout stage.
What to do if the system already works without control?
A working system is not rewritten for observability. The first step is to set up capture at the boundaries: at request input and response output, where services already communicate over the network.
Such an intermediary records the request ID, input, result, and processing time without interfering with the handler itself — the system behavior for the user remains the same.
Next, we choose critical scenarios rather than covering everything at once. These are paths where failure is most costly: order placement, charging, document issuance, support request. We connect them first, and the rest as the data flow stabilizes and retention rules are configured.
Activation goes in two stages. First, shadow mode: events are only written and do not affect anything — this checks that the context is fully labeled and the call chain between services does not break on abnormal branches.
Then part of the traffic goes into collection, metrics are reconciled with application logs, and the share grows only when the discrepancy is within normal limits.
Migration without downtime: nodes are connected one by one, the activation switch lives in configuration, not in code. Rollback is turning off collection, not deploying a previous build; if the data receiver is unavailable, requests continue as usual.
The procedure is: inventory of calls, selection of critical paths, capture at boundaries, reconciliation in shadow mode, expansion of coverage to the remaining scenarios.
Technical concerns: storage, load, access
Trace collection should not become a bottleneck in the dialogue with the agent: we separate the hot response path from telemetry recording.
Events go to a local buffer and are sent over a separate channel without blocking generation; if the receiver is unavailable, the buffer survives restart and catches up on recording later, while the dialogue continues as usual.
History storage, de-identification, and access are described by policies rather than edits in the agent code — rules change without rebuilding. Below are answers to what engineering teams most often clarify before starting.
Will data collection slow down agent responses?
Recording is asynchronous: the event is placed in a non-blocking buffer, and the sender sends them in batches outside the response path.
We separately measure the contribution to response time on the customer's load profile before production launch — if it is noticeable, we adjust the configuration rather than put up with it.
How is personal data de-identified?
Masking happens on the agent side, before recording: names, contacts, and documents are replaced with identifiers according to a mapping dictionary. The dictionary is stored separately, with its own permissions and lifetime. The trace retains the dialogue structure and agent decisions, but not sensitive content.
How long is history stored and can retention periods be changed?
Retention periods are set by policies by data class: full traces with content live for a limited window, aggregates and quality metrics live longer. The policy is changed without rebuilding the agent, and after the period expires, data is deleted automatically.
How is access to traces segmented?
Roles determine the depth of output: an engineer investigates the full trace, an analyst works with aggregates, a manager works with the summary dashboard. Any trace read is recorded in the access log, and sensitive fields are hidden at the output level.
What if collection needs to be paused or a policy rolled back?
Tracing is enabled by a configuration flag: collection is disabled without releasing a new version of the agent, the agent works as usual, and history is simply not written.
Rollback is returning to the previous profile; already recorded data remains accessible under the same rules.
Let's discuss the task: collection schema and implementation plan
To estimate the scope of work, we need initial data about your agent: what model and external tool calls it makes, where logs currently settle, and how they are linked to each other. Without this, the tracing collection schema will be speculative and the estimate rough.
At the first meeting, we review the event flow, mark the points that need to be instrumented, and agree on what we consider a successful run.
On our side, we bring a draft collection schema: what events we record, what we put into the step context, how we link calls by session ID, where we store them, and how we provide them for audit. Along with it, a staged implementation plan so the order and dependencies are visible.
What is useful to prepare before the conversation:
- list of agent scenarios and its external calls — they are used to build the tracing map;
- samples of current logs or exports, even incomplete ones: they show the format and gaps;
- requirements for retention period and depth: what is needed for incident investigation and what is needed for reporting;
- roles and access rights — who reads traces, who can export, who administers;
- environment constraints: closed perimeter, own storage, prohibition on data export;
- integration points: queues, notifications, incident and on-call systems;
- acceptance criteria — how you will know that transparency has actually appeared.
The next step is a 40–60 minute meeting: we review your inputs, show the draft schema, and return with an estimate of the scope of work and a staged implementation plan. Describe the task in two or three paragraphs — that will be enough for us to come prepared.
