KPI for AI agents: where quality control breaks
An agent has been launched into production: it responds to requests, closes some tickets without human involvement.
Acceptance at the same time usually amounts to a dozen questions — they were asked before launch, the answers were satisfactory, and after that no one measures quality.
The real flow of requests differs from the test one, so the first signal of a problem comes from dissatisfied users, not from the system.
The degradation of answers almost never looks like a breakdown. You updated the system instruction for a new scenario, expanded the knowledge base, changed the model for load — and on some requests the agent begins to answer confidently without relying on a source.
Each change individually looks harmless, but the accumulated effect is visible only in comparison: with the same question asked before and after.
KPI for AI agents work when it is a set of measurable quantities, not a single aggregate of "satisfied or not." The share of requests closed without transfer to a human; the share of answers with a confirmed source; the share of repeat requests on the same issue; time to a useful answer; the share of dialogues where the user asked again.
These quantities are captured in two ways — on a regular run of a reference set of requests and on a live sample from dialogues.
Evaluating an agent's work without a reference set runs into manual labeling, and without a live sample — into self-deception: test questions stop reflecting real formulations.
We explain how to assemble both parts, link metrics to configuration changes, and obtain agent quality control in which agent implementation problems are caught before complaints, and a decision is rolled back based on a specific indicator, not a feeling.
What a metrics system for AI agents provides in operation
AI agent metrics begin to work when they are calculated on every dialogue, not after a complaint.
The base value is the share of correct answers: a fixed set of real requests with labels is run on every change of prompts, tools, and model, and the result is compared with the previous version.
A discrepancy above the threshold — the update does not go into production but is returned for revision.
The second loop is reaction time to deviation. While quality is monitored by eye, a day passes between a drop and its detection, and all that time the agent confidently answers incorrectly.
The metric turns the timeframe into minutes: a drop in the share of correct answers or an increase in tool call failures raises a signal immediately, the on-call engineer opens the specific dialogue and sees the cause, not just the fact.
Reducing escalations is an integral indicator, so we break them down by class: the agent did not find data, incorrectly determined intent, went beyond the regulations, missed the response deadline.
Each class has its own metric and its own owner in the customer's team, so the cause is eliminated, not an individual case from the queue.
This is assembled on standard telemetry: dialogue events, tool calls, delays, and outcomes enter a common observability loop, dashboards in Grafana update without manual exports.
A drop becomes visible before the client writes about it, and rolling back a configuration version takes minutes.
Measurement approaches: dialogue review and automated checks
Evaluating an agent with a single number is pointless: methods differ in cost, coverage, and which errors come into view at all. Below are four working options and their blind spots: from fully machine runs to manual review.
| Method | When it fits | Where it runs into limitations |
|---|---|---|
| Rule-based automated checks | Response format, mandatory steps, correctness of internal function calls | A formally correct but useless answer passes as success |
| Automated check by a rubric-based judge model | Mass runs, regression after every edit of agent instructions | Inherits the model's blind spots, requires calibration on humans |
| Reference dialogue sets | Comparing agent versions with each other, protection against degradation | Become outdated, cover only known scenarios |
| Expert dialogue review | Disputed cases, launching a new line, reviewing failures | Expensive in expert time, small sample |
The methods do not compete but divide roles. We fix the reference set as a regression loop: an edit of instructions is run against it, and it is visible what exactly became worse.
Automated checks run on every message of the flow, expert review — on a sample of discrepancies and failures.
The combination holds on calibration: as long as the judge's score and the expert's score diverge within a few percent, machine runs can be considered a metric.
As soon as the divergence grows, we revise the rubric — otherwise KPI for agents begins to measure not the quality of answers but their smoothness.
How we build KPI for AI agents: work stages
KPI for AI agents begins not with charts but with an audit: until every scenario is described and it is clear where exactly the agent makes mistakes, any number in the report is disputable.
Therefore, the first days of work go into analyzing the flow of requests and identifying points where quality can be measured objectively.
- Scenario audit. We analyze all paths along which the agent conducts a dialogue: typical requests, rare branches, transfers to an operator. For each, we record what we consider success and what we consider failure.
- Data collection. We configure logging so that the request, the agent's response, the sources used, and the user's action are saved. Without this, there will be nothing against which to check thresholds.
- Metric alignment. We align technical indicators with business goals: the share of requests closed without an operator, accuracy of answers against the knowledge base, time to answer, share of failures.
- Threshold setup. For each metric, we set not a single value but a corridor: normal, warning, emergency level. We calculate the boundaries based on accumulated statistics, not by eye.
- Metric validation. We run the system on a holdout sample and deliberately break scenarios: if the metric does not notice this, it does not work.
- Implementation and handover. We bring the indicators into general monitoring, describe the reaction regulations for deviations, and teach the customer's team to change thresholds independently.
After handover, the regulations live without us — the team adds new scenarios and threshold edits itself.
What the customer receives at the end of the project
A project on metrics for AI employees ends not with a presentation but with a working kit: the customer's team uses each artifact without our involvement.
We hand over everything with an indication of owner, version, and change procedure — so that in six months no one has to reconstruct the logic of a metric from chat history.
- Metric catalog. For each: what exactly it measures, at what level it is calculated (individual dialogue, scenario, performer), the target corridor, and what is considered a deviation.
- Metric calculation schemes. Formulas, data sources, aggregation window, rules for handling gaps and repeat requests. The same metric yields the same result when recalculated for another period.
- Monitoring dashboard. Slices by scenarios and performers, dynamics over time, and a separate deviations block — the on-call engineer sees the problem earlier than the user notices it.
- Data access policies. Who and on what grounds sees dialogues, who sees only aggregates, where anonymization occurs during export, and how the request log is maintained.
- Review regulations. Threshold values that trigger a review, responsible roles, reaction deadlines, and the procedure for recording the conclusion on an incident.
- Documentation and handover. Instructions for adding a new metric, a scheme of its connection to the department's business indicators, and a recording of a training session for your engineers and operators.
Such a kit survives a change of contractor and growth in the number of AI employees: metrics are added according to the described procedure, rather than being reinvented for each new scenario.
Case study: agent answer quality before and after implementation
Baseline
A consulting agent answered inaccurately on approximately 4 out of 10 requests: it missed a condition from the knowledge base or inserted outdated service data.
Quality was measured selectively — 30 dialogues per week manually, and reviewing one deviation took an average of 40 minutes: the dialogue log lay separately from tool calls and the fragments the agent pulled from the base.
Repeat requests on the same topic accounted for 23% — the client returned because the first answer was formally correct but did not close the task.
What was changed
We introduced an answer labeling scheme: correct, incomplete, fabricated fact, refusal. The call log was joined with the dialogue log in Langfuse — on one screen the entire chain is visible: request, extracted fragments, called tool, final answer. Reviewing a deviation stopped being archaeology across scattered files.
We assembled a reference set of 240 questions with expected answers and attached a run to every release of agent instructions: a drop in the share of correct answers below the threshold blocks the rollout.
The share of correct answers grew from 61% to 89%, deviation review time decreased from 40 to 8 minutes, repeat requests on the same topic fell from 23% to 9%.
The metrics sit in the monitoring dashboard next to the number of requests — a drop in accuracy after a knowledge base edit is visible the same day, not a month later via complaints.
How quality metrics differ from load metrics
Load metrics count what is easy to take from infrastructure: number of requests, time to first token, total response latency, share of technical errors, volume of consumed tokens.
They answer the question "is the service alive and coping with the flow," but say nothing about whether the agent solved the user's task. A flat latency chart looks equally good for a working assistant and for a polite auto-responder that retells an instruction.
Quality metrics measure the result of the dialogue, not its servicing.
The key ones: the share of scenarios closed without transfer to a human; the share of repeat requests on the same topic within one session; the share of answers supported by a source rather than generated from memory; the share of dialogues where the user reformulated the request twice or more — this is a direct sign that the usefulness of the answer is low.
The divergence between the two sets manifests quickly. We speed up generation by shortening the verification chain — latency halves, while the share of escalations to an operator grows, because the agent more often answers confidently and incorrectly.
Load metrics at the same time look better than before the changes, and it is easy to make an erroneous decision.
Quality can only be calculated on a labeled sample of real dialogues: reference sets with an expected outcome, periodic manual labeling, tracing of the agent's steps.
For this, we deploy an observability loop — for example, LangSmith — where each run is compared with the previous one by the share of solved tasks, not by speed.
The practical conclusion is simple: load is the background and condition for operability, quality is the acceptance criterion. KPI for AI agents are built on the latter; otherwise optimization leads the system toward fast but useless answers.
Doubts about agent metrics: we answer substantively
Objections about metrics are usually the same: there are no reference answers, dialogue logs arrive incomplete, and the agent passes checks formally and behaves differently on live requests.
None of these doubts cancels measurement — it changes the scheme: where to get support instead of a reference, what to consider log completeness, and how to distinguish real quality from fitting to the check.
We have no reference answers — what should we compare against then?
A reference is needed only where the answer is unambiguous. For dialogue scenarios, the support is different: scoring rubrics by criteria, pairwise comparisons of two agent versions with each other, and labeling a sample by humans with calibration of divergences.
Such a set is more stable than a single "correct" answer and shows quality movement from version to version.
Dialogue logs are incomplete — will the assessment be unreliable?
Log completeness is itself a metric. We record how many steps are saved: tool calls, input and output data, the fact of transfer to a human.
If intermediate steps are lost in the record, this is a separate class of failures — it shows where context disappears, and it is closed before calculating task indicators.
The agent formally passes checks — how is this caught?
Formal passing is a sign that the checks have become predictable. Therefore, we keep some sets hidden, shuffle tasks and do not show them in sequence, and we make the readiness decision based on work on the flow of real requests, not on internal runs.
We also measure the divergence between the check and live work: it shows where the agent learned a template instead of the task.
Let's discuss the task: we'll analyze scenarios and assemble metrics
So that the review does not turn into an exchange of general considerations, three things are enough for the first conversation: a list of scenarios where the AI employee already works or should work, examples of its answers over recent weeks, and an understanding of who in the company decides "the answer is correct or not." This is enough to outline KPI for AI agents and not guess blindly.
What we analyze at the start:
- which scenarios the agent closes and where the boundary of its responsibility lies;
- what a correct and incorrect answer looks like in each scenario — labeling grows out of this;
- which metrics are already collected and which will have to be added to logging;
- where automated control is needed and where a sample with manual review is needed;
- who on the customer's side confirms reference answers and with what frequency;
- what data and access restrictions the agent team has.
After the review, we return with a work plan: a list of metrics with definitions and thresholds, a data collection scheme, implementation stages with checkpoints.
There too — an estimate of timelines for each stage and the team composition: who is responsible for labeling, who for quality assessment, who for refining agent instructions and rollback rules. Describe the task — we'll discuss scenarios and return with an estimate and a plan.
