A/B Testing LLMs: Where Fine-Tuning Breaks
A/B testing of LLMs breaks not at the launch stage, but earlier — when the team chooses a metric that does not reflect the task.
The classic failure: fine-tuning of a model is evaluated by average response length or by a "like/dislike" rating, even though the business goal is the share of correctly resolved tickets.
The metric starts to grow, while the quality of responses in production drops: the model has learned to write longer and more confidently, but not more accurately.
The second source of errors is sampling bias. If the test set is assembled from the same dialogues that went into fine-tuning, or only from "convenient" scenarios, comparing versions shows nothing.
The model simply reproduces what it memorized, and on rare but costly cases — returns, complaints, non-standard requests — it degrades imperceptibly. Hence the rule: the holdout set is assembled before fine-tuning starts, stratified by request types, and frozen.
The third failure is the lack of prompt version control. When the system instruction, template, and generation parameters change between runs, you are comparing not models but a random set of conditions.
The difference in responses is attributed to fine-tuning, even though it was caused by the new prompt. Without pinning the prompt versions, dataset, and run configuration, the result of A/B testing of LLMs cannot be reproduced — and therefore no decision can be made based on it.
We break down how this is addressed: which metrics actually distinguish versions, how to build an unbiased sample, and how to organize version control so that the comparison withstands scrutiny.
What a correct model test delivers: the effect in numbers
A properly designed A/B test turns fine-tuning from a debate about tastes into a measurable engineering result.
We compare two versions of the model on the same stream of real requests, record quality metrics before and after, and see exactly where the new version wins and where it degrades.
Without such a comparison, any "it got better" remains an opinion rather than a fact you can rely on when rolling out.
The main indicator is the share of relevant answers: it grows predictably when the training set is collected from real requests rather than synthetic ones.
The second most important effect is the reduction in manual review: the more accurately the model holds context and format, the smaller the share of answers an operator has to read through.
In practice, this frees up the team's time and removes the bottleneck that previously slowed down the entire process.
Separately, we monitor the class of errors, not just the average score. A model can raise overall accuracy and at the same time start confidently making mistakes in critical scenarios — such a trade-off is unacceptable.
That is why in the test we record which types of failures have disappeared, which have appeared, and how dangerous they are, and only after that do we make the decision to roll out.
The speed of evaluation is also part of the result. When the comparison run is automated and repeatable, the next fine-tuning iteration is verified in hours rather than weeks of manual analysis.
This changes the pace of work: the team manages to test more hypotheses and abandons dead ends faster without spending resources on them.
The outcome of a correct test is not a single number but a set of quality metrics that show the direction of movement.
Growth in the share of relevant answers, a drop in the share of manual review, and a reduction in the dangerous class of errors together provide confidence that the model has improved in substance, not just on paper.
Evaluation metrics: precision, recall, stability
The metric is chosen for the task, not "by default." For entity extraction and classification, precision (the share of correct answers among all answers) and recall (the share of found items among all that were in the reference set) are calculated — they pull in different directions, and the balance is set by the threshold.
For generative answers, human or model-as-judge evaluation against a rubric is added: factual consistency, completeness of the answer, absence of fabrications.
Stability is a separate axis.
The same query is run several times at different temperatures and the spread is observed: if a version answers correctly sometimes and incorrectly other times, the average precision is deceptively high, while in production the user encounters exactly the bad answer.
Therefore, alongside precision, the share of unstable cases and the spread across subsamples are tracked.
False conclusions most often arise from three places: a small sample (dozens of examples produce a difference within the noise), dataset bias (tests are assembled from easy queries), and fitting to the metric.
The defense is a fixed held-out set, stratification by query types, and confidence intervals: if the gap between versions is smaller than the interval, the conclusion "it got better" is not made.
Comparing versions is done correctly on the same set, with identical generation settings and a single evaluation rubric. Then fine-tuning yields a comparable result: it is clear where the gain is real and where it is the effect of a different prompt or random spread.
Approaches to running an A/B test: comparison of methods
The choice of comparison scheme depends on how costly a model error in production is and whether you have enough traffic for statistics.
An offline test runs both versions on a fixed set of queries with reference answers — it is fast and repeatable, but it will not show how the model behaves on live user phrasings.
An online test splits real traffic between the variants and measures the target metric, but it requires a sufficient volume of queries, otherwise the difference will drown in noise.
A hybrid scheme removes the limitations of both: first, offline filters out obviously weak variants, then the winner is validated on live traffic. Below is a comparison by applicability and limitations.
| Approach | When it fits | Limitations |
|---|---|---|
| Offline test | Many labeled examples, fast iteration needed, risk of error is high | Does not catch phrasing drift, the metric may diverge from real-world value |
| Online test | There is a stream of queries, the actual metric matters (share of resolved tasks, returns) | Requires traffic volume for significance, takes longer, affects users |
| Hybrid scheme | A critical product where the cost of an error is high and traffic is barely enough | Harder to organize: two evaluation loops and a consistent metric between them |
How to choose for the task
If there are few queries and the cost of an error is high, an offline test on an extended set of examples gives an answer faster and more safely.
When there is traffic and the metric is unambiguous, an online test shows what offline misses: unexpected phrasings, shifts in behavior across segments, regressions on rare scenarios.
In practice, we more often go hybrid: the offline loop filters out variants, and online confirms the winner on a limited share of traffic. This way the decision relies on actual data rather than on a single metric.
Stages of LLM testing implementation
The process is built around one principle: A/B testing of LLMs only makes sense when metrics, a query corpus, and acceptance rules are fixed. Otherwise, comparing two versions turns into an argument about taste based on ten examples. Below is the sequence we go through on every fine-tuning project.
How the work is structured
- Audit of the current model. We collect real queries from product logs, label them by task type, and identify the share of failures: where the model hallucinates, where it deviates from the format, where it loses context. The output is a baseline against which we will compare everything that follows.
- Test design. We define the control and test groups, the share of traffic for each, the set of metrics (answer accuracy, format adherence, completeness, robustness to paraphrasing), and the threshold at which the new version is considered better. We fix the threshold before launch — otherwise the result is tailored to what we want.
- Data preparation. We clean the corpus, remove duplicates and leaks between the training and validation sets, and balance rare scenarios. We keep a separate holdout set that does not participate in fine-tuning at any step.
- Setup and launch. We train the model, run both variants on the same set of queries, calculate metrics, and review discrepancies manually — automatic evaluation misses semantic errors that a human can see.
- Robustness check. We look at behavior on edge-case phrasings, long dialogues, and off-topic queries. We verify that the quality gain is not bought by degradation on rare but important scenarios.
- Handover of the protocol. We deliver a document: how to run the test, which metrics to calculate, where to store the labels, and under what conditions to roll back the version. The client's team repeats the cycle without us.
- Re-run. After every change to prompts, data, or the base model, the test is run again — otherwise accumulated edits once again turn into unverified hypotheses.
The key result is not a one-time conclusion that "version B is better," but a reproducible procedure that answers this question on every subsequent release.
What the client receives
We hand over not a "model that works" but a reproducible loop: your team can use it to repeat the test, explain the result, and roll back a change without us. Everything below consists of working artifacts, not a presentation.
- Experiment configurations — files with training and inference parameters, pinned versions of the base model and adapters, generation seed. The same run is reproduced both at your end and at ours.
- Test designs — splitting traffic into control and test groups, the share of traffic per variant, stopping criteria, a set of metrics and the threshold at which a variant is deemed better or worse.
- Access policies — which employees can see prompts and logs, where data is stored, what is masked before writing, how export rights are delineated.
- Data documentation — the provenance of the training and validation samples, labeling rules, the share of rejected examples, known biases.
- Rollout regulations — the procedure for switching traffic, conditions for automatic rollback to the previous version, the person responsible for the decision.
- Monitoring regulations — which quality and cost metrics are collected continuously, how often the model is reviewed, who responds to degradation.
- Experiment report — the result of the A/B test with confidence intervals, an analysis of discrepancies between groups, and an explicit list of what the test did not prove.
Each artifact lives in your repository and is updated together with the model. If tomorrow you change vendors or build the loop on your own infrastructure, the documentation and regulations carry over as is — you won't have to rewrite the experiment logic.
How does an offline test differ from an online LLM test?
An offline test is a run of the model on a pre-assembled set of queries with known reference answers. You fix the dataset, calculate metrics for each "question-answer" pair, and compare the base model with the fine-tuned one on the same data.
Speed here is determined by the size of the set: a thousand dialogues run in minutes, and a repeat run after tweaking hyperparameters costs the same.
The risk lies in the disconnect from reality: the dataset quickly becomes outdated, and the metrics don't capture what matters to the user.
An online test runs both versions on live traffic and distributes queries between them. Here you see not a judge's score but behavior: the share of accepted answers, repetitions, dialogue length, refusals.
The speed of getting a signal is lower — you need to accumulate a statistically significant volume on each branch, otherwise the difference will turn out to be random.
The cost is higher: two models are running, the load on inference grows, and part of the queries goes to manual labeling.
The main difference is in the cost of an error. An offline test is safe: you discard an unsuccessful version before rollout, and the user never sees it.
An online test risks reputation — if the fine-tuned model starts hallucinating on live customers, the rollback must be prepared in advance: keep a traffic switch and be able to return the base branch in minutes, not hours.
That is why the approaches don't replace but complement each other. An offline test filters out obviously weak options and is cheap, while an online test confirms the effect on real scenarios.
A reasonable sequence is to first run a dozen configurations offline, keep the two or three strongest, and bring them onto live traffic with a limited share of users.
Case study: how we reduced the share of erroneous answers
The task was typical for support: the assistant answered questions about internal regulations, and for some requests it gave a confident but incorrect answer — it referenced an outdated version of a document or mixed up clauses from different sections.
Erroneous answers accounted for a noticeable share of requests, and operators spent time manually checking each answer before sending it to the client.
We started by establishing a baseline: we ran a set of real questions with reference answers through the assistant and labeled exactly where the model made mistakes — source substitution, loss of a condition, incorrect interpretation of wording.
It turned out that most of the errors came from a few types of questions rather than being spread evenly.
What we changed
Fine-tuning was done on "question — correct answer with source reference" pairs collected from the same regulations as in the production loop. We separately added examples where the model had to recognize an outdated version and not use it.
The settings were not chosen by eye: version comparison was done via an A/B test on the same set of questions, with identical conditions and a fixed set of metrics — the share of erroneous answers, the share of answers referencing the wrong source, and the share of refusals to answer.
Results in metrics
| Metric | Before | After |
|---|---|---|
| Share of erroneous answers | baseline | reduced by roughly a factor of three |
| Answers referencing an outdated version | regularly | isolated cases |
| Time to evaluate one model version | manual review of the corpus | automatic run over the labeled set |
The acceleration of evaluation turned out to be no less important than the reduction in errors itself: previously each new version went through manual review, now comparison takes hours instead of days, and the decision to roll out is made based on numbers rather than an impression from a few examples.
This loop is reproducible — the metrics are calculated on the same set, so the next fine-tuning is compared with the previous one fairly.
Let's discuss your LLM testing task
To make the conversation concrete, rather than "tell us in a nutshell what you want," we start with a short brief. That way we immediately see exactly where testing breaks down in your process, and come back with an estimate and a plan, not general considerations.
What's worth having on hand for the first meeting:
- Which model is currently in use and on which branch — the base one or one already fine-tuned on your data.
- What exactly we're optimizing: answer accuracy, format adherence, tone, refusal of undesirable topics.
- Whether there's a reference set of prompts and answers that can be used to compute a metric, or whether it needs to be assembled.
- How two versions of the model are currently compared — manually, by the eyes of several experts, or with automated checks.
- Which constraints on response latency and inference cost are critical for the product.
- Who on the client side decides "this version is better" and by what criteria.
- Whether there are requirements for reproducibility: whether you need to return to test results a month later.
We tailor the formats of work to the task. It could be a one-off audit of the testing methodology, where we examine your current approach and show where it produces a biased result.
Or a full cycle: we prepare a set of test prompts, set up automated evaluation, run a comparison of versions, and deliver a report with metrics. Or a regular process — you get a pipeline in which every new version of the model goes through the same check before rollout.
Describe the task in free form: what the product is, which model, what's unsatisfactory about the current testing. We'll come back with a timeline estimate, a work plan, and what data we'll need from you at the start.
