AI observability: why model responses degrade in production
A failure that is invisible on dashboards is the most expensive one.
The service responds, response time is normal, error rate is zero, while answer quality quietly drifts: the model more often misses conditions from the instruction, answers contain unsourced facts, and for some requests the context of previous steps is lost.
Infrastructure monitoring will not catch this — it looks at availability and latency, not at the meaning of what was returned to the user.
There are three typical places where context is lost. First — a failure of one step in the call chain: search returned an empty result, the normalization step produced a truncated fragment, and then the model completed the answer by guessing.
Second — an unnoticed behavior change after a model or prompt update: formally the same contract, but the answer length, link format, and willingness to follow constraints have changed. Third — unsourced answers, when the document link is either missing or points to the wrong place.
Such failures are usually caught from users: a complaint about a specific answer arrives, and then incident investigation begins from logs.
Diagnostics run into the fact that the request and response are stored as a single line, without step-by-step breakdown, without the input data of each call, without the prompt and model version.
The situation cannot be reproduced — the input data has already changed, and there is no evaluation build or regression set.
This is exactly what AI service observability addresses: tracing the entire chain while preserving intermediate results, linking to prompt and model versions, regular evaluation of answers against a set of checks, and alerts on quality drift, not only on errors. Below we break down how this is structured and which AI observability tools we put to work.
What AI observability delivers: measurable changes
Tracing model calls provides the link “request — retrieved context — response — evaluation”.
Metrics are collected on the application side, so you can see not only the fact of a failure but also the step where it occurred: prompt preparation, knowledge base search, tool call, post-processing.
Platforms of this class — from the open-source Langfuse to cloud services — accept such traces without rewriting application logic. The effect of observability is measured in verifiable quantities.
Incident investigation time
The complaint “the model answers incorrectly” used to be investigated manually: reproduce the dialogue, find the prompt version, compare with the release.
With tracing, the incident is localized in minutes — input data, triggered tools, and the full response text are visible, so there is no need to guess about the context.
Share of detected deviations
Automated evaluations calculate the share of answers that passed checks for relevance, format correctness, and absence of fabricated facts. Deviations are visible across the entire sample, not only in the dialogues someone happened to open.
Volume of manual reviews
Manual review narrows to a slice: the system itself selects answers with a low score, long response time, or unusual token usage. The volume of model calls is visible for each scenario — it is clear which prompts and knowledge base sections carry the load and which are unused.
Options for connecting observability: comparison
Telemetry is captured in two ways: from inside the application — with the platform's client library, or from outside — with a gateway between the application and the model.
The first sees the entire chain of steps, knowledge base search, and tool calls; the second requires no code changes but records only the exchange with the model. Below are five telemetry collection modes and the boundaries of each.
| Option | When it fits | Limitations |
|---|---|---|
| Instrumentation inside the application | Need spans for chain steps, knowledge base search, and tool calls | Code changes and library version maintenance |
| External telemetry gateway in front of the model | Quick start without code changes, a single point for all services | Sees only the exchange with the model, does not reveal internal steps |
| Cloud observability service — LangSmith, Helicone | No own storage, need ready-made dashboards and error analysis | Data goes to the service provider, there are volume limits |
| Open-source tracing platform in your own environment — Langfuse | Requirement to store prompts and responses locally | Deployment, updates, and backups on your side |
| Self-managed build: telemetry collector and your own storage | Non-standard metrics and your own storage schema | Development and support entirely on your side |
Modes are usually combined: the gateway collects the network layer across all services at once, while the library adds detail for critical scenarios.
If prompts and responses cannot leave your environment, the cloud option is out — leaving an open-source platform or a self-managed build.
Implementation procedure: from audit to handover
Observability of an LLM application does not appear just by connecting a library: first you need to understand which calls occur and what needs to be reconstructed from them during incident investigation.
Below is the sequence of work we follow from inventory to handover of procedures to the client's team.
Audit and inventory of calls
We build a map of all model call points: user scenarios, background tasks, agent chains, service and retry requests. For each point we define the trace composition — session identifier, prompt version, input and output, token usage, latency, status. A separate list covers what must not be stored: personal data and keys are masked before recording.
Flow diagram and data storage
We design the telemetry path to storage: recording from the application or through a proxy layer, behavior when the receiver is unavailable, thinning of routine calls so that volume does not grow linearly.
Ready-made platforms like Langfuse cover visualization, but data placement and retention are dictated by the client's requirements.
- Inventory of calls and agreement on the field list.
- Collection scheme: capture points, event format, buffer on the application side.
- Storage setup: cleanup policy, record retention, masking.
- Validation in a test environment: running scenarios, comparison with logs, walkthrough of a training incident.
- Assessment of volume growth and behavior under load.
- Handover of procedures: reading traces, alerts, actions when collection fails.
Output artifacts: what the team retains
We hand over not a report on completed work but a working observability loop that the team then maintains itself.
All materials are stored in the repository next to the code: changes go through review, the change history is preserved, and dependencies on external services are described explicitly.
Configurations and diagrams
- Data collection configuration: which model calls are recorded in full, what goes into sampled recording, where personal data is removed — before storage, not after.
- Telemetry flow diagram: capture points, buffers, trace storage, retention period, and system behavior when the receiver is unavailable.
- Configured project in Langfuse with dashboards: response latency, error rate, token usage by models and scenarios.
- Metrics and labeling reference: what counts as an error, how scenarios are labeled, and along which dimensions slices are built.
Access policies and procedures
- Data access policies: roles, who can see full traces, who sees only aggregates, retention periods, and the deletion procedure on request.
- Deviation alerts: thresholds for error rate, latency, and usage, notification channels, and on-call procedure.
- Incident investigation procedure: who opens the investigation, within what time, what is recorded in the conclusion, and how the fix is verified.
After handover, engineers add new metrics and change thresholds without us — this is a single configuration change, not a separate project.
Case study: how incident investigation time was reduced
The client is a service with an AI assistant: the answer is assembled from several model calls and internal service calls.
They approached us after a series of complaints — responses arrived empty, truncated, or off-topic, and the failure could not be reproduced from application logs.
What it was like before
Diagnostics relied on text logs: the fact of the request and the response code were visible, but not which step of the chain corrupted the result. Context was assembled manually, and this consumed most of the time.
- Failure localization — 5–6 hours, if the context could be assembled at all.
- Root cause was found in 40% of cases; the rest were closed with a restart — and the complaint returned.
- Repeat contacts about the same issue — about one third of users.
- Up to 25 incidents per month, some noticed only by the client.
What changed after
We deployed tracing on Langfuse: each request is a tree of steps with input, output, latency, and token count, so the failing step is visible without manual assembly.
On top — a set of reference questions with automated answer checking and alerts on a drop in the share of successful answers.
- Failure localization — 15–20 minutes including confirmation of the cause.
- Root cause is found in 9 out of 10 investigations; disputed cases are closed with a link to the trace.
- Repeat contacts decreased by 60%: investigation ends with a fix, not a restart.
- Incidents — from 25 to 8 per month; the rest are caught by the regression set before release.
Each investigation adds to the set of checks, so a similar failure surfaces during testing, not at the client.
How tracing differs from request logging
Logs are a flat stream: a line for an incoming request, a line for the response, a separate one for an error.
While there is only one request, the picture is readable; under parallel load, events from different users are interleaved, and the cause of a failure is reconstructed from timestamps — with errors at step boundaries.
Tracing describes a request as a tree of steps with a common end-to-end identifier. In a single trace, you can see how many fragments were retrieved from storage, with what parameters the model was called, which tools were invoked, and how long each step took.
Parent-child relationships show chains of model calls, including nested and repeated calls, which in the log look like a set of unrelated lines.
The difference is noticeable when investigating degradation. From logs you can see that response time increased; from the trace — that it is specifically knowledge base search that is slow, while generation remained the same.
Correlation of processing steps allows comparing two runs of the same request after a prompt or model change and seeing where the results diverge, instead of guessing from the final text.
Logging is not canceled by this: logs are broader and cheaper, they cover infrastructure, failures, access restrictions — everything that happens outside the processing of a specific request.
The context of the request and response is stored at its own trace level and does not mix with the general stream, while the link is maintained by the end-to-end request identifier in each record: from an anomaly in a metric you move to the trace and to the needed log lines without manual correlation by time. Observability is assembled from this linkage — when metrics, traces, and logs reference each other.
Limitations: data storage, privacy, latency
Traces grow linearly with traffic, and telemetry storage volume quickly becomes the main cost item of maintaining the loop. Therefore, retention is set by policy: detailed traces with all steps live for a short window, aggregated quality metrics — for a long one.
Sampling is also controlled: full traces are recorded for a share of requests, while the rest provide metrics without prompt and response bodies.
Privacy of request data is addressed at the input, not by cleanup after the fact. Sensitive fields — personal data, credentials, access keys — are identified by a list of rules and covered by masking or hashing on the application side, before being written to storage.
The trace retains the call structure and metrics sufficient for investigation, but not text that must not be shown outside the environment. The list of rules is maintained together with code changes — otherwise a new field in a form will again start leaking into telemetry.
The impact of collection on response latency is eliminated by exporting outside the critical path: writing to a local buffer, sending in batches, no synchronous network calls between the model and the user.
If policy prohibits data export, traces remain in your environment — Langfuse is deployed inside your network, and not a single fragment of the request leaves it.
Will trace collection slow down user responses?
No, if it is moved outside the critical path. The application writes an event to a local buffer, and a separate process sends batches to storage. If storage is unavailable, the buffer is size-limited and drops old events rather than blocking the model request.
Let's discuss the task: what we clarify in the first review
Such a review takes less than an hour and does not require prepared infrastructure — it is enough to have a person on your side who knows the environment and access to a description of how model calls are currently organized.
We review the observability task, record what is already being collected, and come back with an estimate and work plan.
- list of source data: which services call models, where prompts live, what current logging looks like, and in what form traces arrive;
- volumes: approximate number of requests per day, context length, share of responses currently investigated manually;
- access procedure to the environment: who issues keys to the environment, whether approvals are needed, and in what mode access is allowed — metrics read-only or collector deployment;
- privacy requirements: which fields must not leave the perimeter, what we mask before sending, where history is stored;
- questions the system must answer: where time is lost, which requests are more expensive than others, why some responses are rejected;
- roles: who on the client side accepts the result, and what timelines of the first stage we meet.
After the review, you receive a breakdown of stages with the sequence of work, control points, and an understanding of what will be needed from your team at each step. Describe the task — we will come back with an estimate and plan.
