Why LLM quality assessment breaks after release
Offline sets of examples and real traffic diverge almost always. On a closed sample, metrics look smooth, but in prod the distribution of requests is different: short clarifications, mixed languages, context snippets, attachments.
LLM quality assessment by average score doesn't catch this — the model answers confidently where it previously honestly said "I don't know."
The worst part is that changes come one at a time. A wording edit in the system prompt, a model version change, a new tool in the call chain — and regressions after prompt updates manifest not in the scenario that was touched, but in the neighboring one.
No one rechecks the entire scenario matrix after each such edit, so hallucinations in prod surface days later.
Model response drift accumulates more slowly, but just as unnoticed: an external provider updates the model on their side, and wording, response format, and willingness to follow instruction change without a single edit on our side.
Manual spot checks don't save here — an engineer physically reviews dozens of dialogs, while there are thousands per day, and the sample is biased toward those requests that have already been written to support.
As a result, the first to notice a failure is the client. They get a wrong link, a fabricated number, or a refusal where previously there was an answer, and lose trust in the product entirely, not just in one scenario.
We look at how this is addressed: continuous evaluation that catches discrepancies before they reach the user.
Automated response evaluation instead of manual runs
Manual runs scale linearly: checking a new prompt version on two hundred dialogs is a day of expert work, and by the time of analysis some answers no longer correspond to the current configuration.
A regular check loop removes this linearity: automation takes on some decisions, and a human reviews only disputed cases.
How much manual work leaves the process
Automation of response checks is built on unambiguous criteria: the response relies on the source, doesn't invent facts, maintains the given tone, doesn't reveal internal instructions.
Formulated once, they apply to every run identically — different experts stop interpreting correctness in their own way.
Reduction of manual labeling comes from cutting off obvious cases: automation itself marks responses with high similarity to the reference and surfaces those where the discrepancy is significant. The human is left with a tenth of the sample — borderline and conflicting dialogs.
How quickly regression is caught
Braintrust stores run history, so a new set of responses is compared with the previous one immediately after assembly: metrics by unified correctness criteria are calculated automatically, and the drop is visible in the report without manual comparison.
Regression detection speed is measured in minutes: a prompt edit or model change is run on historical dialogs and gives an answer before rollout, not after a week of operation.
Discrepancies are recorded for specific dialogs — the reason for rollback is visible without repeated investigation.
Options for the check loop: comparison of approaches
A check loop is metrics, datasets, and labeling rules that answer whether response quality got better after a model or query wording edit. It is assembled in three ways, and the choice depends not on team size, but on how often scenarios change and who participates in labeling.
| Option | When suitable | Limitations |
|---|---|---|
| Custom loop on open-source evaluation libraries | Standard metrics, labeling by team | Comparison of runs and dataset versions lives in utility scripts, maintenance grows |
| External response evaluation platform | Need run history, reproducibility, collaboration | Part of data goes to external loop, need to export metrics back |
| Hybrid check scheme | Automation at scale, expert labeling pointwise | Requires unified dataset format and rules version |
When open-source libraries are enough
If tasks are uniform — classification, field extraction, response format check — open-source libraries are enough. Metrics are calculated locally, the dataset lies next to the code, and a run is launched on every change.
The loop is transparent: you can see which function calculates which metric. The limit comes at the second labeled sample, when comparing outputs of two runs manually is already inconvenient.
When a separate external loop is needed
A separate loop is needed when reproducibility and labeling by several people are important. Braintrust stores versions of datasets and metrics, so the dispute "did it get worse or did it seem so" is closed by a link to a specific run, not by rerunning scripts.
Hybrid removes the compromise: automatic checks remain in the repository, and expert evaluations and version comparison — in the external loop.
Implementation stages: from scenario audit to regulation
We build the check loop so that every prompt or model edit passes the same set of checks and the result is comparable with the previous run. Below is the order of work: from scenario analysis to regulation according to which the customer continues on their own.
Audit and criteria definition
We start with an audit of assistant scenarios: export real dialogs and label where the response went wrong — loss of context, fabricated facts, wrong tone.
From this list grow criteria for a correct response: for each scenario we fix what we consider correct, what is an acceptable deviation, and what is a defect.
- Analyze scenarios and classify errors by type — get a list of what needs to be checked at all.
- Formulate correctness criteria: mandatory facts, acceptable wording, prohibitions in the response.
- Assemble a check set — questions with reference answers and automatic checks for format and mandatory blocks.
Testing on historical data and rollout
The check set cannot be taken on faith, so we run it on historical data and compare with manual labeling. If automatic evaluation diverges from expert evaluation more often than the agreed share of cases, we rewrite the rule.
- Run the loop on accumulated dialogs and compare with manual labeling.
- Set up regular runs on Braintrust for every prompt or model change; the report contains dynamics by each criterion.
- Roll out the check loop into working mode and hand over the regulation: how to add checks, how to read the report, at what drop to roll back a change.
What we hand over: tests, schemes, regulations
We hand over not a folder with settings, but a working quality evaluation loop: an engineer launches a run and reads the report without understanding someone else's logic. The package includes:
- Check sets by scenario — support dialog, summarization, field extraction; each has its own set of reference examples and the metric by which quality is calculated.
- Scheme of the check loop: where references are stored, how the run proceeds, how results get into reports and what data is needed for that.
- Labeling rules for disputed cases: what we consider correct for an ambiguous response, who resolves labeler conflict and how the decision is recorded.
- Reports and alerts on metric drop: thresholds, notification channel, on-call procedure and false alarm sign.
- Model update regulation: mandatory run on a fixed set before version change, acceptance criteria and rollback conditions.
- Access and role matrix: who edits references, who edits labeling rules, who sees only reports.
Documentation and labeling rules
We describe the loop so that a new engineer can assemble a run according to instructions: purpose of each set, reference format, analysis of typical labeler errors.
Disputed cases are consolidated into a table "situation — decision — rationale": it removes most questions without requiring contacting us.
What is already configured in your loop
The loop is assembled on Braintrust and connected to your repository: a run is launched from a branch, the result is compared with the previous release, and a metric drop below threshold goes to the team channel.
A new scenario is added with one reference file and one entry in configuration — without edits to check code.
Case: reducing errors in a support assistant
The first-line assistant answered typical requests using the knowledge base and passed complex cases to an operator.
Instructions for the model and the model itself were updated every few weeks, and quality was checked manually: operators read a sample of dialogs and made a verdict.
At the volume of requests this stopped working — the sample covered a small part of scenarios, and different reviewers evaluated disputed answers differently.
What it was before implementation
The share of incorrect answers held around 12%: the assistant confused return conditions, referenced outdated knowledge base sections, and answered confidently where transfer to an operator was needed.
Manual release check took 3-4 days — reference dialogs were reviewed by eye, and the update waited for a verdict. In a quarter, two regressions went to production: they were noticed by complaints when the share of escalations to operators grew.
What changed after
We assembled a check loop on Braintrust: a reference set of dialogs with labeling, automatic evaluations by criteria — answer completeness, reliance on source, escalation correctness, absence of invented conditions.
A run is launched on every change of instructions or model, and the quality threshold blocks release if the score by criterion dropped; only cases where automation is not confident go to manual review.
The share of incorrect answers decreased from 12% to 3%, release check decreased from 3-4 days to 40 minutes, missed regressions per quarter — zero.
In support this gave fewer escalations and relieved operators on complex requests: the assistant closes typical questions itself, and the team sees which criterion pulls quality down after another change.
How offline labeling differs from prod checks
Offline labeling and prod checks answer different questions. A set on historical data shows whether the model got better on known cases; live traffic — whether it breaks on what was not in the set.
Prepared set on historical dialogs
Here real requests are collected, a reference answer or rating scale is fixed, and every model version is run through.
The property of the set is repeatability: same inputs, same methodology, so results are comparable between builds and you can see what gave improvement and what rolled back quality.
A platform like Braintrust keeps such sets together with evaluations and automatically compares runs, showing discrepancies for each example.
There is exactly one blind spot: the set reflects past request distribution. New wording, topics, and edge cases do not get into it until added manually.
Control on live traffic
Online response quality control works on what arrives now: share of refusals and repeat questions, analysis of dialogs where the user left, collection of user feedback directly in the interface. This is the only way to see topic shift and rare scenarios.
But there is no reference here — evaluation is noisy, depends on audience behavior, and part of feedback simply doesn't arrive.
The difference between check methods is that offline is responsible for regression, and online — for detection. Working scheme: a set on historical dialogs stands as a gate before release, and anomalies from prod are analyzed and turned into new examples of the set.
Common technical questions before integration
Before integration, engineering teams ask almost the same questions — and all of them are about how checks coexist with an already working product.
Below are answers to typical doubts: from discrepancies in labeling to choosing between your own check loop and an external platform.
The general principle is one: checks are embedded into the current release process, without stopping the service, and start with an offline run on historical requests.
Labelers diverge in evaluations — can checks be trusted?
We calculate agreement on the overlapping part of the sample: disputed examples go to arbitration with a senior labeler, and the rule is recorded in the instructions.
The agreement metric goes into the report next to model quality — otherwise labeling noise is mistaken for degradation.
The model is already in prod — will it have to be stopped?
No. First, a shadow run on historical requests, without affecting live answers. Changes in checks and rules are rolled out behind a toggle that is removed with one action — rollback takes minutes.
How much historical data is needed to start?
Usually several hundred labeled examples per scenario, collected from the real request stream — then thresholds are not random. If there is little data, we start on synthetic examples and actively collect disputed cases.
How to connect new scenarios to checks?
A new scenario gets its own reference answers and thresholds, and common checks (format, completeness, safety) are reused. Before rollout, a run on old sets is mandatory — regressions are visible before release.
Own check loop or external platform?
Own loop keeps sensitive requests within the perimeter, but requires maintenance of storage and versioning of sets. External platform is faster at start: on Braintrust labeling and runs are assembled quickly. Often a hybrid is chosen — sensitive locally, the rest outside.
Let's discuss the task: what data is needed to start
Evaluation accuracy depends not on the tool, but on how much the check set reflects your real scenarios. To propose a working check loop on Braintrust, not an abstract scheme, we need to see how the system behaves now and where exactly it errs.
- Description of scenarios: what tasks the model solves, who the user is and what is considered a correct answer.
- 30–50 examples of successful answers — the reference and base for repeat runs are assembled from them.
- Examples of unsuccessful answers, including borderline cases: invented facts, refusal, broken format.
- Current metrics and request logs: error share, response time, user labeling, if it is maintained.
- Timeline constraints: when the first result is needed and which releases cannot be stopped.
- Data handling rules: where requests may be processed, what to anonymize, whether an isolated loop is needed.
- Who on the customer side approves quality criteria and analyzes disputed examples.
On this data we will assemble an implementation plan for the check loop: test sets with fixed references, automatic runs on every prompt or model change, thresholds at which the release is blocked, and reporting on quality dynamics from version to version.
Describe the task and send what you already have — we will return with a timeline estimate, scope of work, and implementation plan.