Why LLM dataset validation breaks at scale
On a sample of a thousand examples, annotation discrepancies are almost invisible: they are cleaned up manually and considered random noise.
At a scale of hundreds of thousands, the picture changes — dozens of annotators interpret the instruction differently, and the same question receives either a detailed answer, or a one-word answer, or a refusal.
A 5-8% difference in annotation between annotators does not surface in the report, but it is exactly what sets the ceiling above which the fine-tuned model cannot rise.
The second layer of the problem is duplicates and leakage. Similar phrasings of the same question end up in both the training and control sets at the same time, and validation metrics show growth that does not exist on real requests.
As long as the dataset is deduplicated by exact match, semantic repetitions remain unnoticed: a paraphrase of two or three words passes the filter and doubles the weight of the topic in training.
Noise in the training set and annotation errors in the dataset are dangerous because they are amplified by the model, not dampened by it.
The team spends weeks selecting training recipes and fixing prompts, although the bottleneck in data preparation lies earlier — in the dataset itself.
LLM dataset validation is needed not as a one-time cleanup before launch, but as regular control: annotation consistency rules, discrepancy thresholds between annotators, and checks for overlaps between sets.
We break down how this is addressed: what checks are put in place at the input, how disputed cases are annotated, and what the discrepancy shows before and after cleanup.
What gets fixed before training: duplicates, noise, annotation, leakage
Before fine-tuning, the dataset is checked by defect classes, and the model learns each of them along with the useful signal. Fixing this is cheapest on the data: after training starts, the cost of fixing grows with the volume of annotation. Below is exactly what we check and how we measure the result.
- Duplicates and near-duplicates. We find exact repetitions and paraphrased pairs by n-gram overlap with a configurable similarity threshold. After cleanup, the training part is 15–40% shorter without losing meaning — the model stops overestimating frequent phrasings.
- Noise and garbage. Recognition fragments, service insertions, broken characters, and off-topic examples are filtered out by rules and length statistics. The share of garbage drops below 1%: the model's capacity is not spent memorizing artifacts.
- Annotation consistency. The same example receives different labels from different annotators. We calculate the discrepancy rate on a control subsample, review disputed cases and record the rule in the instructions, while suspicious labels are ranked by Cleanlab. Discrepancies below 5% mean that the class boundary is annotated consistently.
- Leakage between sets. Similar examples end up in both training and validation, which inflates metrics on the holdout part. We group related examples by source and document and split by groups — the overlap rate is zero, and the estimate is honest.
- Class imbalance. A rare class drowns in a frequent one: we recalculate weights in the loss function and select examples by strata instead of synthetic data. The quality curve on the rare class comes out of the noise.
Cleanup modes: rules, model annotation, manual review
No single cleanup mode works on its own. Filtering rules are cheap, reproducible, and predictable, but blind to meaning: they will not distinguish a coherent answer from a plausible set of phrases.
A judge model catches semantic defects — contradictions, non-compliance with instructions, harmful phrasings — but inherits the biases of the base model and requires calibration against a reference.
Two things determine the choice of mode: the size of the corpus and the cost of an error at the output.
That is why we compare approaches along three axes: what exactly it catches, at what scale it pays off, and what it requires from the team. Using the table below, we choose a single mode or build a cascade.
| Option | When it fits | Limitations |
|---|---|---|
| Filtering rules | Large corpora, coarse defects: duplicates, fragments, broken annotation, prohibited fragments | Does not see semantic garbage or leakage; rules must be maintained when the source changes |
| Judge model | Semantic criteria: coherence, instruction following, toxicity, topical relevance | Biases of the base model, rubric drift; calibration and repeatability control are needed |
| Manual sample review | Threshold calibration, reference annotation, review of disputed example groups | Does not scale to the whole corpus; requires annotator instructions and agreement measurement |
| Cascade of rules, model, and manual review | Mixed corpora with high cleanliness requirements | Harder to reproduce: rules, rubric, and thresholds are versioned together |
Separately, we calibrate the example cutoff threshold: it always balances between recall and precision.
A too-low threshold leaves noise and contradictions that the model will later learn as the norm; a too-high one cuts out rare but valid examples — long dialogues, rare vocabulary, non-standard phrasings.
We select the threshold on a reference set annotated manually and fix it together with the rubric version — otherwise a repeated run produces a different dataset and reproducibility is lost.
Stages of dataset audit and cleanup: from inventory to handover
Data audit is not a one-time file cleanup, but a sequence where each step relies on the result of the previous one.
Order matters: without inventory, profiling counts duplicates as separate records, and without a cleanup plan, filtering cuts out rare classes that the model is supposed to learn.
- Source inventory. We collect all corpora and exports, record provenance, usage rights, collection date, and overlaps between sources — some records are usually duplicated across two or three datasets.
- Dataset profiling. We calculate distributions: record length, language, share of empty and truncated examples, label spread. Skew is revealed here — before it enters training.
- Annotation schema. We formalize target classes, annotation rules, and the procedure for resolving discrepancies between annotators; otherwise identical examples will receive different labels.
- Cleanup plan. We describe exact and near-duplicate deduplication, garbage removal, normalization, class balancing, and splitting into training, validation, and holdout sets without data leakage.
- Filtering and re-annotation. We run the rules, review disputed cases manually, and measure annotation consistency before and after.
- Control run. We train on the cleaned and original slices, compare metrics on the holdout set — this shows what the cleanup actually achieved, rather than a feeling.
- Handover. We deliver a versioned dataset, a data card describing limitations, and scripts that reproduce the assembly of the slices.
Artifacts: annotation schemas, filtering rules, reports
LLM dataset validation ends not with a verdict of “data is suitable,” but with a set of artifacts that the client's team can use to repeat the cleanup on a new slice without us. Below is what the handover includes and why each document is needed.
- Annotation schema: record fields, value types, requiredness, allowed labels, and the rule for resolving conflicts between annotators.
- Annotator instructions: how to interpret borderline examples, what counts as a defect, how to mark uncertain annotation — discrepancies between people drop severalfold.
- Data filtering rules: deduplication by normalized text, length and language thresholds, removal of template and machine-generated fragments, personal data processing.
- Data quality report: distribution by classes and segments, share of discrepancies between annotators, coverage of the set by checks, list of problematic slices.
- Log of excluded examples: every discarded fragment with a reason and a link to the rule — the decision can be challenged and reviewed, rather than taken on faith.
- Cleanup configurations and scripts: a repeated run on a new slice produces the same result and the same reports.
- Re-validation procedure: by which metrics and how often to recheck the set after fine-tuning.
We hand over the set with a walkthrough: we show where the rules diverge from real data and what to change first. After that, the client's team runs the validation cycle independently.
Case study: fine-tuning on noisy data before and after
Before: what the audit showed
The project is a first-line support assistant that answers from an internal knowledge base. The client brought an archive of requests and wanted to fine-tune the model on it.
Before training, we ran an audit: out of 240 thousand pairs, almost 19% turned out to be near-duplicates with different annotations, 8% contained contradictory answers, and another 6% had lost context during export — the answer was there, but the question it referred to was no longer in the pair.
Training on such a set produced instability: two runs with the same configuration differed by several percentage points in the share of correct answers on the control set.
After: what was changed and what was achieved
Deduplication was calculated by normalized phrasings with a similarity threshold, not by exact match, so paraphrases of the same request did not enter the set.
Conflicting pairs were sent to arbitration, annotation rules were recorded in writing, and annotation was done in Label Studio — annotation versions became reproducible. Truncated examples were restored from the original dialogues or discarded.
The share of duplicates fell from 19% to 2%, contradictory examples became less than 1%, and the dataset volume shrank by about a quarter.
The share of correct answers on the control set grew from 61% to 79%, and the variance between repeated runs was less than one percent, meaning the result no longer depended on the data sequence.
The control set was fixed before work began and was not used in training. This gives the client the ability to compare model versions with each other and understand that the quality gain is a consequence of the data, not of fitting to the validation set.
What to do if the training set has already been collected and annotated?
Full re-annotation is a last resort, not the first step. A second pass doubles the labor costs and introduces a new layer of noise: two annotators interpret the instruction differently, and the previous annotation stops being a reference for comparing runs. So we first measure — how many truly broken examples are in the set and in which slices they lie.
We build a partial dataset audit in a stratified way — by sources, task types, and answer length.
We look at the discrepancy rate between annotators on overlapping examples, the share of truncated and empty answers, duplicates, and the overlap of the test part with the training part.
Defects are usually concentrated in two or three slices, and re-annotating the rest is an expense without a gain in metrics.
Next comes additional annotation of disputed segments: where annotators disagreed or the answer is formally correct but contradicts the instruction, we clarify the rule and re-annotate only that slice.
We leave the rest of the dataset untouched in order to preserve comparability of runs and see the contribution of the corrected data specifically.
Reusing instructions covers most of the volume: already annotated answers remain in the set if they do not contradict the updated rules.
An automatic compliance check pulls only borderline cases for manual additional annotation — in our practice, that is single-digit percentages of the dataset.
The set becomes suitable for fine-tuning without full re-annotation, and decisions for each slice are recorded in the rules so that the next data collection does not repeat the same mistakes.
Limitations and common doubts about data quality
Cleanup relies on statistics: rules are written for mass cases — duplicates, annotation fragments, template answers.
On rare classes, the same filters produce false positives and cut out exactly what the model is fine-tuned for: narrow terminology, atypical phrasings, isolated refusal scenarios.
Deletion without quarantine is the most expensive type of error: the defect in the dataset is invisible, and the needed signal can no longer be recovered.
We do not measure the effect of cleanup on a noisy metric with a single number. We calculate on a stratified holdout set, look at rare classes in separate slices, repeat the run on the same set, and compare the variance. Alongside this, we keep the share of preserved rare examples and the volume of rejected data.
Will we not lose rare examples under strict rules?
We delete only obvious garbage; disputed records go into quarantine with the reason for rejection and are reviewed manually. Before the run, we fix the share of rare classes by slice; after cleanup, we compare — a drop above the agreed threshold stops the process.
How can we tell that cleanup had an effect if the metric is noisy?
We fix the holdout set, the order of examples, and the run conditions. We look not at the overall score but at the change by slice: where the rule worked in a targeted way, the shift is visible on small numbers. A difference within the variance of repeated runs is not considered an effect.
Let's discuss the task: where the work begins
Before taking on the entire dataset, we need to understand how it is structured. For an estimate, a description of the dataset, the current annotation instructions, and a set of disputed examples are enough — this is sufficient to see the error classes and give timelines.
- Volume and composition: languages, topics, how the data was collected, and what share was obtained by generation versus from real requests.
- Annotation instructions and a guide for annotators, including places where practice diverged from the text.
- 30–50 fragments where annotators did not agree with each other or the answer looks questionable: they show where consistency breaks.
- Product requirements: which errors are unacceptable, what the model must not answer, which scenarios are considered target.
- How the dataset is split into training and control — to exclude overlaps and leakage between parts.
- Access: an anonymized export or work on your side, who confirms disputed cases.
- Acceptance criteria and deadlines that are critical for you.
Based on this set, we assemble a cleanup work plan: deduplication and normalization, toxicity and personal data filters, re-annotation of disputed cases, and reassembly of the training/control split.
The first step is a pilot stage on a small slice — it shows whether a rule works on your data, rather than in theory, and where cleanup rules conflict with each other. After that, the timeline estimate is based on measurements, not average numbers. Let's discuss the task — we will come back with an estimate and a plan.
