Why RAG evaluation breaks on real data
On a demo set of twenty questions, the assistant answers without errors, but after a week of use employees stop using it.
The reason is not the model, but that real traffic is structured differently: queries are short and contain typos, wording does not match the text of regulations, and some questions require data from two documents at once.
The first layer where quality is lost is knowledge base retrieval.
The needed fragment either does not make it into the top results or comes along with noise: document splitting does not match semantic blocks, metadata does not work as a filter, and similar wording from different versions of the same document compete with each other.
Here we measure retrieval recall and precision — what share of relevant fragments the system retrieves and how much extra it pulls into the context.
The second layer is generation. Even with correct context, the assistant adds “common knowledge,” merges two fragments into one, and substitutes numbers or deadlines.
The answer sounds confident but diverges from the knowledge base, and trust breaks down at that point: an unverifiable answer is worse than no answer.
Hence the third layer is source faithfulness: what share of claims in the answer can be confirmed by the cited fragment. We separate metrics by layer — taking the framework from the open-source RAGAS — and see exactly where the quality of the assistant's answers drops. We analyze how this is addressed at each layer.
What changes after implementing regular checks
A regular run on a frozen set of questions turns answer quality from a feeling into a metric.
We record reference pairs “question → expected answer and source” and after every change to the prompt, model, or search index calculate faithfulness, precision, and recall via RAGAS — results are comparable between releases, rather than “this time it seems to answer better.” What changes, point by point:
- The share of refusals drops. Answers whose faithfulness is below the threshold or that lack support from a retrieved fragment are filtered out before rollout: the version that goes to production has a share of unreliable answers that is several times lower, not “about the same as before.”
- Index updates ship faster. Rebuilding and validation happen in a single run, so an edit to the knowledge base takes hours instead of days of manually reviewing dozens of dialogs.
- Regressions are visible at the release threshold. Comparison with the previous run shows which groups of questions declined after a model change or document chunking.
- Disputed answers surface before release. The list of questions with low recall shows gaps in the knowledge base — they are filled with content, not another prompt tweak.
- Analysis became concrete. Any answer has a trace: which fragments were retrieved, what entered the context, and at which step the needed fact was lost.
The result: answer quality control stops being a one-off acceptance check and becomes part of the release cycle — with thresholds, metric history, and a shared team criterion for what counts as a degradation.
Evaluation methods: from manual labeling to automated tests
The quality of a RAG system can be checked in three ways, and in practice they are not chosen as either/or — they cover different classes of errors.
Manual labeling provides a reference: an expert goes through a test set of questions with known correct fragments and marks whether the answer relies on the source and whether the needed paragraph was lost.
A synthetic set is assembled from the document corpus and scales to hundreds of cases, but it reproduces the wording of the documents themselves, not live user speech.
Automated runs calculate metrics without a human: the share of answers supported by a source, whether the needed fragment made it into the results, and recall relative to the reference.
| Option | When it fits | Limitations |
|---|---|---|
| Manual labeling | Project start, disputed and rare questions, quality threshold tuning | Expensive in expert time, scales poorly, subjective between annotators |
| Synthetic test set | Fast coverage of hundreds of questions, checking indexing completeness | Inherits document wording, does not catch conversational paraphrases and typos |
| Automated metric runs | Regular checks on every change to the index or prompt | The metric does not understand meaning: a correct answer with different wording is counted as a miss |
The working scheme is combined: a reference from manual labeling (usually one to two hundred questions, including rare scenarios), synthetic data for volume, and an automated run on every change.
Open metric suites like RAGAS cover basic measurements, but we compare their scores with manual ones — a discrepancy immediately shows where the metric is lying, and then the weights are revisited.
The test set lives together with the system: new user requests become reference questions, so regressions are caught before release, not after complaints.
Stages of implementing a quality control loop
A quality control loop does not appear together with the first version of answers: it is assembled in stages, leaving at each step an artifact that will survive a model change and corpus reindexing.
Below is the order of work from data inventory to the procedure by which runs are launched without our involvement.
- Data audit — inventory of sources, checking freshness and duplicates, topic labeling: what the corpus covers densely and what is assembled from several fragments.
- Reference questions — a simple fact, a summary across several documents, a question with no answer in the corpus. For each, expected source fragments, not just the correct wording.
- Metrics and baseline measurement — retrieval precision, result recall, answer faithfulness to supporting fragments. The first run provides a baseline for all subsequent changes.
- Acceptance thresholds — separately for each metric: what is sufficient for release and what blocks the release. Fluent wording must not mask a failure in recall.
- Regression runs — on a model change, chunking scheme, or corpus edit; we analyze discrepancies with the baseline before release, not after complaints.
- Error analysis — classification of failures: fragment not found, an outdated one found, answer assembled not from sources. The share of groups shows what to fix — retrieval or the corpus.
- Handover of the procedure — the question set, thresholds, and run order go to your team as a document it launches itself.
The procedure describes what to run after each corpus edit and how to interpret discrepancies — quality stops depending on who is on release duty today. As new scenarios appear, the reference question set expands.
What we hand over: test sets, thresholds, procedures
Along with the working system, the client receives a kit by which answer quality can be measured after handover — without us and without our infrastructure. The run is launched on your side, on a fixed index, and gives comparable results from release to release.
- Test set — questions collected from real requests, labeled by type: retrieval, clarifying, multi-step, and deliberately without an answer in the knowledge base.
- Reference answers — for each question, what is considered correct is recorded, and the source fragment confirming that answer is specified.
- Metrics report — retrieval precision and recall, the share of answers confirmed by a source, and the share of correct refusals “no data in the base.”
- Slices by group — metrics are calculated separately by topic and query type: an average score across the entire sample hides a failure on a narrow but important segment.
- Acceptance rules — threshold values for each metric and conditions under which a release is blocked, rather than “improved later.”
- Run configuration — fragment size, document chunking scheme, search parameters, and index version: without this, two reports cannot be compared.
- Maintenance procedure — the order for replenishing the test set after each failure analysis and the rule for who updates the references when the knowledge base changes.
A model change, reindexing, or knowledge base expansion is run in minutes and immediately shows what exactly has declined — retrieval or generation. You make the rollout decision based on numbers, not on the feeling from a few manual checks.
Case study: RAG evaluation on a stream of customer requests
A RAG assistant on a stream of customer requests answered quickly, but no one measured quality: changes to the prompt and document chunking were made based on gut feeling.
The answers looked convincing but relied on the wrong fragments — the discrepancy was noticed from operator complaints, and analyzing a single case took days.
Before
They tested on ten convenient questions where the system answered well: real wording with typos, mixed terminology, and clarifications in a long dialog did not make it into the sample.
There were no accuracy metrics — only a subjective “it got better.” Because of this, a change could neither be accepted with justification nor rolled back: it was unclear what exactly improved and whether something broke in adjacent scenarios.
After
We assembled a set from live requests with reference fragments and answers and connected regular runs on RAGAS: answer faithfulness to the source, context precision, and context recall.
Now every change to retrieval or the prompt goes through a run before rollout, and engineers analyze disputed cases manually.
Context precision grew from 0.61 to 0.87, answer faithfulness from 0.72 to 0.94, and context recall from 0.58 to 0.83. The share of answers built on an irrelevant fragment decreased by about three times — incorrect advice and repeat requests on the same topic went away.
Changes stopped being a lottery: degradation is visible in the run before rollout, not in feedback a week later, and quality analysis takes hours instead of days.
How retrieval evaluation differs from generation evaluation
A failure in a RAG system looks the same: the user gets a confident but incorrect answer. The cause, however, can lie in two different places — at the retrieval stage or at the generation stage — and the metrics for these loops are different.
Retrieval evaluation answers the question of whether the needed fragment made it into the context at all.
Here we measure context recall — the share of questions for which the relevant piece ended up in the top results, result precision — what portion of that set consists of genuinely useful fragments, and the position of the first match: if the needed paragraph is seventh, the generator will not always reach it.
Generation evaluation works with the finished answer and the context passed to it. The main metric is faithfulness to context: every claim in the answer must rely on the provided fragments, not on the model's memory.
Alongside this, we measure answer completeness relative to the question and the share of honest refusals “this is not in the context” — such an “I don't know” is better than a plausible fabrication.
Separate metrics are needed because these loops are fixed differently. Low context recall is treated with chunking, semantic and keyword search, reranking, and metadata filters.
If the context is in place but the answer diverges from it, we fix the prompt and answer schema, require source references, and verify claims after generation.
An overall score without separation gives the engineer nothing: a drop of ten percent looks the same with a broken index and with an unsuccessful prompt edit.
We calculate both loops on the same set of questions with labeled fragments, so it is clear which layer declined and what to do about it.
Common technical questions and limitations
Before starting, we define the measurement boundaries: metrics like faithfulness, precision, and recall describe the system's behavior on a prepared question set, not its quality in general.
Therefore we always add error analysis to the numbers — otherwise a rise in faithfulness masks answers that formally rely on context but do not solve the user's task.
How do you deal with labeling noise?
The reference set is labeled independently by two people, and discrepancies go to arbitration by a third specialist.
We keep the disagreement rate below 10% and bring disputed examples to consensus: without this, precision and recall fluctuate more than the effect of changes in the system itself.
What are the limitations of the metrics themselves?
No metric catches an incomplete answer: it may be correct but omit an important part of the context. We supplement automated measurement with a set of “pain point” questions and manual review of a random sample — this way degradation is visible where an averaged score hides it.
How often should evaluation be run and when should the set be updated?
A full run — for every search index release or generative model change; a quick metric slice — for every change to the wording of model requests.
We revisit the test set when the corpus is updated and new scenarios appear: outdated questions inflate the score because the system answers them from memory rather than from documents.
Let's discuss the task and come back with a work plan
To make the analysis concrete rather than general, send the inputs for the current solution. They show where the bottleneck is — in generation quality, retrieval, or reference labeling — and what makes sense to change first.
- scenario: what questions users ask and what answer is considered correct;
- 20–50 real queries with reference answers and a link to the source fragment;
- current retrieval configuration: how documents are chunked, which embedding model is used, how many fragments are supplied to the context;
- document corpus or its anonymized slice — we run checks on your data, not on a demo set;
- metrics, if you already calculate them: precision, recall, faithfulness, share of answers without context support;
- observed failures: fabricated facts, outdated fragments, loss of information from the middle of a document;
- data constraints: where the loop can be deployed, which external services are allowed.
In response, we will come back with a work plan: stages, a set of metrics with threshold values, a regression run scheme on a fixed question set, and the start order.
Separately, we will record which evaluation procedures we hand over to your team so that checks remain reproducible without us.
Describe the task — we will come back with an estimate and a plan.
