Quality control for AI agents: where the process breaks
An agent that answers confidently in a demo breaks on the tenth real request. The first failure point is instability: for two identical inquiries, the model produces different structures, loses required fields, and interprets the same instruction differently.
In a production environment, such divergence turns into an incident: the customer gets an incorrect answer, the operator gets extra manual editing, and support gets the fallout.
The second point is degradation after a model update. Changing weights, context, or system instruction alters behavior in edge scenarios, while baseline answers look the same. Without fixed benchmarks, regression is noticed weeks later — from accumulated failure statistics.
The third is that there are no reproducible benchmarks, and there is no way to verify the reproducibility of the agent's answers.
Labeling reference dialogues captures expected steps, acceptable wording variations, and failure criteria: a conversation of "it seems to have gotten worse" turns into a verifiable comparison.
Quality control for AI agents rests on running such sets before deployment, not on selective log review.
The fourth is errors in the production environment, where the agent works with real data, integrations, and the right to act. Here the cost of an error is higher than in chat: an incorrect call remains in the system.
We'll break down how this is addressed — which benchmarks we collect, how we catch regression, and how we embed checks into deployment.
How we measure agent behavior: metrics and benchmark sets
As long as agent quality is assessed on a few illustrative examples, conclusions rest on impressions. We fix a benchmark set of requests — real inquiries from product logs, labeled manually: typical scenarios, rare wording, and edge cases where the agent most often errs.
The set is versioned together with prompts, so iterations are compared on the same data, and a metric increase cannot be explained by a "lucky" new sample.
We describe labeling rules before the first run: what we consider a correct answer, where different wording is acceptable, how to resolve disputed cases.
Two labelers go through the sample independently, an arbiter resolves discrepancies — the benchmark does not "drift" with the reviewer's mood. Metric calculation is automated on LangSmith: manual runs are needed only when the set itself is updated.
Checklist by which an agent is allowed into the product:
- Share of correct answers — the base metric: how many requests in the benchmark set are resolved correctly. We look not only at the average but also at the breakdown by groups: typical, rare, edge.
- Behavior stability — we run the same request repeatedly. If the meaning of the answer changes from run to run, the agent is not ready for production even with a high average share of correct answers.
- Grounding in the source — the answer contains no invented facts or references to data the agent did not receive.
- Correct refusal — outside its domain, the agent declines the task instead of giving a confident incorrect answer.
- Labeling stability — the sample is periodically relabeled blindly: increased disagreement means the benchmark is outdated and it is time to update it.
Metrics are captured at every iteration before release, so regression is visible in numbers before the user notices it.
Comparing agent validation modes: runs, labeling, hybrid
The validation mode is chosen not by convenience but by the class of errors that need to be caught. Automated runs on a benchmark set of scenarios cover regression: recurring failures are caught immediately after every change to the prompt, model, or tool-call logic.
What runs do not see are new wording, ambiguous requests, and tone errors for which there is no benchmark yet.
Manual expert review provides analysis where the criterion "correct" requires a human: disputed topics, source quality, coherence of a multi-step answer.
Its limitation is labor intensity and variance in assessments between experts: without a labeling protocol and calibration, assessments of the same dialogue diverge. A hybrid scheme removes this by distributing the flow by risk level.
| Mode | When it fits | What it provides | Limitations |
|---|---|---|---|
| Automated runs | There are benchmark answers and metrics, scenarios repeat | Validation on every release, reproducibility, early regression detection | Catches only what is already described in checks; does not assess meaning and tone |
| Manual expert review | New directions, disputed topics, benchmark labeling | Depth: analysis of logic, source evaluation, benchmark creation | Expert labor intensity, assessment variance, does not scale to the entire flow |
| Hybrid validation scheme | Agents in production, request flow growing | Automation covers the entire flow, expert review covers a sample by risk and new cases | Labeling and sample selection setup are required, otherwise the scheme does not work |
The choice of control mode is not a one-time decision: as analyzed failures accumulate, manual cases are transferred into automated checks, and expert time goes to labeling the next layer.
Implementation stages: from audit of current checks to handover
We fix the order of stages before the start: each is closed with an artifact that can be verified, not a work report. Iterations within a stage are normal, but moving to the next one happens only after the result is accepted by both sides.
- Audit of current checks. We analyze what already works: manual reviews, logs, metrics, disputed cases. We calculate what share of deviations is currently caught and where automated checking diverges from a human decision.
- Quality control scheme. We define metrics and their distribution: automated checks, sample manual review, escalation. We fix thresholds and run-stop rules so the result does not depend on the duty engineer's mood.
- Run setup. We assemble benchmark sets, label edge cases, run on historical data, and calibrate thresholds. The output is a reproducible run: the same input data yields the same verdict.
- Trial run. We launch it in shadow mode alongside current control and compare verdicts. We analyze discrepancies: some are closed by adjusting rules, others by additional training on disputed examples.
- System acceptance. We go through the criteria agreed in advance: share of caught deviations, false positive rate, time to analyze one case.
- Documentation handover. We provide regulations, metric descriptions, instructions for duty engineers, the procedure for updating benchmark sets, and access. After that, we support in an agreed mode until the team reaches an independent cycle.
What remains with you: configurations, schemes, and validation regulations
We hand over not a report on work performed but a working validation system that your team runs further without us.
All artifacts are stored in your repository in text form, undergo review, and are versioned together with agent code — changing a threshold or rule does not require rebuilding the entire system.
- Validation configurations — where thresholds, release stop conditions, and agent version comparison rules are set. They are changed in one place and apply to all flows at once.
- Data flow schemes — the path from a user request to the validator's verdict: where answers are stored, where the benchmark is, at which step manual review is connected.
- Labeling regulations — what counts as a correct answer, how to resolve disputed cases, and how many reviewers examine one sample. Without it, inter-labeler agreement drops already in the second week.
- Benchmark sets — fixed examples with agreed assessments. Versions are reproducibly compared on them: what improved and what degraded is visible.
- Metric documentation — how each value is calculated, where the data comes from, and which deviation is considered a trigger rather than noise.
- Team instructions — what the reviewer does in case of a discrepancy, who has the right to change a rule, and where an exception is recorded.
- Incident log and analysis procedure — cases where a check missed an error are analyzed and turned into a new rule or an example in the benchmark set.
- New version release procedure — a run on the benchmark set, metric acceptance, and the procedure for rolling back to the previous configuration.
A new engineer gets up to speed using these documents, not the memory of the previous team. This is the maturity check: if the validation system cannot be handed over, it rests on people, not on process.
Case: share of agent errors before and after regression runs
A customer came to us with a first-line assistant: the agent answered questions about orders, timelines, and returns. At baseline measurements, the share of answers matching the benchmark stayed around 78%; the rest was handled manually by operators.
The error was floating — the same question in different wording received a different answer, and in the logs it looked like noise rather than a pattern.
What pre-implementation measurements showed
We labeled the request flow by classes and saw that more than half of the discrepancies had one cause: the agent substituted a general return period while ignoring the product category.
The second group was a breakdown at clarification: the agent did not ask for the order number and went into guessing instead of asking. Traces and metric slices were collected in Langfuse so that every discrepancy could be traced to a specific dialogue step.
What we changed and what the result was
We assembled 240 benchmark scenarios — for each, a benchmark answer and formal pass criteria. The regression run is launched for every change to the agent instruction and rule set, takes about 12 minutes, and blocks deployment if the share of passes falls below the threshold.
Separately, we added a mandatory clarification step: the agent does not answer until it has received the order number.
The share of correct answers rose from 78% to 94%, and the "incorrect return period" class disappeared completely — 0 hits across 240 scenarios and 0 in the request flow over the following month.
Dialogue branch coverage was brought to full according to labeling, and the breakdown at clarification stopped reproducing. Regression runs do not eliminate errors forever: a new instruction can break a neighboring scenario. But now such a rollback is visible before deployment, not in a customer complaint.
How does a shadow run differ from a regression test?
These are two different validation tools, and confusing them is costly. A shadow run launches a new version of the agent alongside the current one on a live request flow: answers are not sent to users, but how the model would have answered in a real situation is recorded. A regression test works on a fixed set of scenarios with known benchmark answers in advance.
A shadow run catches what is in no prepared set: new wording, topic drift, rare but regular request types. The price for this is the absence of a benchmark: the verdict has to be derived from divergence from production version answers and sample labeling by an engineer.
A regression test is strong in another way: it is reproducible and cheap to run. The same fixed set is run after a model, prompt, or call chain change — and any shift is visible immediately, down to the specific scenario that broke.
Comparing the modes comes down to three questions: do we have a benchmark, do we need repeatability, and what are we validating — known requirements or unknown risks. A shadow run covers the second, regression covers the first; they do not replace each other.
Therefore, regression runs on every release, and a shadow run runs on schedule and after major changes. Everything a shadow check finds for the first time becomes a new scenario in the fixed set: coverage grows, and old defects do not return.
Common technical doubts before starting work
Before starting, engineering teams ask the same questions: will the quality control layer fit into an already deployed environment, will the closed perimeter be sufficient, and where will validation data come from. Below are four technical doubts that most often slow down the decision to start work.
We have our own model — will it have to be changed?
Evaluation is built on top of the model, not instead of it: answers pass through a common call interface and are checked against rules and benchmark examples.
Your model remains operational; additions concern only the validation set — from answer format to task completion completeness.
Infrastructure access is limited, the environment is closed
Then checks are deployed inside your perimeter, and only anonymized metrics go outside — without requests and answers. If even that is unacceptable, we hand over configurations and schemes, and your team performs launch and updates according to the description.
We already have a deployed environment — won't you break it?
We integrate as a separate evaluation layer and do not touch routing, storage, or request processing logic. Such a layer is disabled with one setting, so rolling back to previous behavior takes minutes and does not require rebuilding the entire environment.
There is no labeling — should benchmark sets be built from scratch?
We start with formal rules and agreement among several models with one another, and grow the benchmark set on disputed cases as work proceeds.
Your specialists label only those answers where automation does not give an unambiguous decision — usually a small share of the overall flow.
Let's discuss your task and come back with an estimate and plan
The first step is not a commercial proposal but an analysis of the task. We look at which agents and models are already in operation, how the request flow is structured, and exactly where answer quality is lost.
This is enough to come back with an estimate of the scope of work and an implementation plan: stages, checkpoints, and areas of responsibility on both sides. You make the decision to start after the estimate — we do not fix the scope in advance.
For the estimate to be to the point rather than "based on feelings," a minimal set of inputs is needed:
- Task and context: what is already automated, how many requests per day pass through AI agents, and which errors you consider unacceptable.
- Validation data: a dialogue export for a period, benchmark answers if available, and a log of operator edits after auto-replies.
- Access: to the environment where agents operate, to logs, and to the request queue — read access is enough; nothing needs to be changed.
- Quality criteria: how you formulate a "good" answer and which topics must necessarily go to a human review.
- Constraints: data storage requirements, access separation, environment isolation, prohibition on transferring sensitive information.
- Scope estimate: on this basis, we calculate the number of scenarios, the share of answers under automated validation, and the labor costs for labeling.
Next, we assemble an implementation plan: a pilot on one scenario, measurement of metrics before and after, and expansion based on the result. Describe the task — we will come back with an estimate and plan, and we will schedule the start of work at a pace convenient for you.
