Why LLM data cleaning breaks at scale
On a demo sample of a couple thousand examples, the corpus looks clean, but at full volume fine-tuning behaves unpredictably: the validation metric rises, while in production the model repeats formulations from instructions and loses response format.
The cause is almost never architecture, but how data cleaning was performed—defects in the training corpus are not visible during spot checks by eye.
Duplicates in the sample are the first and most expensive defect. Exact copies are caught by normalization and hashing, but near-duplicates remain: the same instruction with different intros, reprints from the same source.
The model sees the same example dozens of times, overfits to it, and rare but important cases drown in repetitions. Alongside it is label noise: fragments of service tags, unmatched quotes, fields left over from automated collection, answers to the wrong question.
A separate category is hidden templates. If nine out of ten answers begin with the same introductory phrase or contain a source marker, the model learns the template instead of solving the task and confidently reproduces it on user requests.
Leakage of reference examples undermines trust in metrics: as long as the validation set overlaps with the training set, measurements show quality that does not exist in production.
The drop in fine-tuning quality surfaces after rollout, when retraining is most expensive—we examine how LLM data cleaning closes this before training starts.
What controlled corpus quality provides for fine-tuning
Data quality control begins not with cleaning but with measurability: until the share of rejected examples is counted for each defect class, it is impossible to distinguish corpus improvement from random fluctuation.
We keep a defect profile as a report for each run—how many duplicates, template insertions, broken encodings, truncated dialogues, and overlaps with evaluation sets were rejected and by which rule.
The defect profile shows which rules work and which cut out too much.
The repetition threshold and language filter are tuned by domain: where rare constructions are valuable for the target scenario, the rule is relaxed rather than disabled—otherwise the rare class simply disappears from the corpus, and the model loses robustness on it.
Training reproducibility rests on versioning: the set gets a manifest with checksums, filter composition, and splits. A run repeated a month later yields the same result, and comparison of two trainings comes down to the difference in rules, not noise in the data.
For deduplication and filtering at large volumes we use NeMo Curator, while metrics are calculated with our own scripts.
Model robustness on rare scenarios is a direct consequence of careful cleaning: excessive filtering removes exactly the long tail, while metrics on the overall set rise.
Therefore we compare the share of rare classes before and after filtering and keep it no lower than the original.
Stream or one-time export: how the approaches differ
The corpus preparation mode affects not only the timeline of the first run but also how the corpus lives on: a one-time corpus export gives a good snapshot as of a date, regular batch reprocessing keeps it up to date, and a continuous stream catches fresh data almost immediately.
The difference shows up later—in the cost of oversight, version reproducibility, and how painful it is to roll back a corrupted run.
| Option | When it fits | Mode limitations |
|---|---|---|
| One-time corpus export | First run, hypothesis check, source with rare changes | Snapshot becomes outdated; reproducibility rests only on the archived version |
| Regular batch reprocessing | Data changes predictably: daily, weekly, monthly | Requires a data update procedure and comparison of only new fragments, otherwise duplicates |
| Continuous stream processing | Live stream of events or user feedback | Rollback is harder, duplicates at window boundaries, higher oversight requirements |
In practice, a combination is more often used: the stream validates records and puts them into a buffer, while batch reprocessing at an agreed interval rebuilds a complete corpus version—with repeated deduplication, normalization, and personal data labeling.
The data update procedure fixes what counts as a new version, the minimum number of clean records needed for a rebuild, and the rule by which failed runs are rolled back.
This way, stream processing does not turn into uncontrolled growth, and each corpus version remains traceable to the source.
How corpus cleaning is built: stages of engineering work
Corpus cleaning is built not around an off-the-shelf library, but around specific data and the model's task.
First we agree on what counts as noise here specifically: broken encodings, insertions from extraneous sources, template blocks, machine translation, identical instructions with different labels.
Then the work proceeds in stages, and each leaves a verifiable artifact—a report, a rule, or a test.
- Corpus audit — stratified sampling by sources and languages, manual labeling of noise classes and their shares. This sets the order of work: first we remove what produces the bulk of the garbage.
- Filtering rules — each rule is written as a verifiable condition: share of service characters, length in tokens, match with templates. We select thresholds based on labeling and version the set according to individual calculationExample deduplication — comparison by proximity signatures rather than exact match: this catches paraphrases and concatenations. We separately process "instruction — answer" pairs: a duplicate question with different answers is more harmful than a complete repetition.
- Pipeline assembly — steps are formatted as a reproducible chain with logging: how many examples each filter removed. We change a threshold and immediately see the consequences, instead of restarting blindly.
- Control set — a run on a held-out set that did not participate in rule selection: share of false removals, residual noise, preservation of rare but valuable examples.
- Pipeline handover — configurations, tests, metric reports, and launch instructions. The client's team runs cleaning itself on a new data slice.
Configurations, schemas, and corpus preparation procedures
Each corpus preparation step is fixed in artifacts that remain in your environment, not in correspondence with the contractor.
From them, a new engineer reproduces the assembly from scratch, an auditor sees the origin of each record, and a repeat run yields the same result on the same input data.
- Data flow schema — the path of a record from source to final corpus: processing nodes, branching points, quarantine of rejected items, checks at each transition.
- Pipeline settings — declarative configuration files: normalization rules, filter thresholds by language and duplicates, deduplication windows, limits on record length and encoding.
- Corpus versioning — dataset snapshots with checksums and a change log: you can see which build went into fine-tuning and which version to roll back to.
- Corpus access policies — roles and sensitivity levels, masking of personal fields, data access log, procedure for granting and revoking access.
- Reprocessing procedure — what happens to rejected records: quarantine, re-labeling, retention period, procedure for deletion at the data owner's request.
- Data documentation — corpus passport: language composition and shares, volume, sources and usage rights, cleanliness metrics report.
- Acceptance checklist — reproducible checks: a control run on a fixed sample and thresholds at which the build is considered valid.
We hand over the package together with a launch on your infrastructure: the client's engineers edit the configuration independently, and the pipeline remains manageable without us.
Case: how the share of defects in the training corpus was reduced
What came in
The client is a product team fine-tuning a support assistant: a corpus of 4.2 million documents was collected from tickets, chats, and an internal knowledge base.
Measurements before the work showed 34% nearly identical records, 11% fragments—excerpts truncated by field limit—and inconsistency in service labeling.
Because of this, fine-tuning was restarted 5–6 times: the model reproduced both working operator answers and garbage drafts with equal confidence.
What was changed
We assembled a corpus preparation pipeline: normalization of text and line breaks, removal of service markup, deduplication by fuzzy shingle matching—distributed computation on Apache Spark.
Then quality filters: removal of short and fragmentary excerpts, language detection, removal of template greetings, masking of personal data. Metrics were calculated on control sets after each step—this shows the contribution of an individual filter, not just the overall result.
Metrics before and after
| Metric | Before | After |
|---|---|---|
| Share of duplicates | 34% | 6% |
| Fragments and broken excerpts | 11% | 1.4% |
| Training sample size | 4.2 million documents | 2.6 million |
| Repeated fine-tuning runs | 5–6 | 1–2 |
The pipeline remained with the team as a regular run: new exports go through the same checks, the corpus quality report is generated automatically, and the share of duplicates stays at 6% and does not grow with volume.
Repeated fine-tuning runs dropped from 5–6 to 1–2—data fixes are now visible before training starts, not after.
What to do if the corpus is already collected and partially labeled?
The first step is an inventory of the existing corpus: we count the share of labeled examples, the current label schema, how many examples were labeled by several people, and where disagreements exceed the agreement threshold.
More is usually reused than the client expects: if the label schema has not changed, the labeled core goes into fine-tuning as is, while disputed examples go to arbitration rather than full re-labeling from scratch.
Partial labeling does not prevent work. The labeled part works as a seed: we calibrate automatic labeling on it, validate it on a held-out set, and manually label only what has low confidence. This way the corpus grows without a proportional increase in manual labor.
Re-labeling of examples is needed where the label schema or the rule for assigning to a class has changed.
We build a mapping table from old labels to new ones, run label migration, and check the result on a control set: the share of examples that lost their label must be visible before fine-tuning starts.
We do not delete the previous labeling—it remains a separate revision that can be returned to during rollback.
Data change history is a mandatory part, not an add-on. We record each example change with author, reason, and version, and keep the corpus in revisions so that fine-tuning on any of them is reproducible.
This makes it possible to compare metrics before and after edits and avoid launching training blindly after mass re-labeling.
Technical doubts about corpus quality
We check corpus suitability not by eye but by a set of measurable features: share of exact duplicates and near-duplicates, spread of document lengths, share of fragments in another language, density of service markup and broken text, coverage of target topic clusters.
We fix thresholds before work starts—then acceptance becomes a comparison with numbers, not a debate about tastes.
If after filtering 60–70% of the original volume remains and it covers all required topics, the corpus is suitable; if less, we rebuild the sources rather than tighten the rules.
Will we lose rare examples during strict cleaning?
We will if filters are applied in a single wave. Therefore we do not delete rare classes but mark them and move them to a separate set: deduplication on them uses a lower threshold, and their weight is increased during training.
Control is by label frequency distribution: no class should disappear after cleaning, even if its share is a fraction of a percent.
How do we measure quality if there is no reference labeling?
We assemble a "golden hundred": a small sample labeled manually according to instructions, and calibrate all automatic rules on it.
We evaluate the rest of the corpus indirectly—by agreement of two independent annotators on control fragments and by model behavior on a held-out set before and after fine-tuning. If answer quality does not improve, the problem is more often in the selection rules than in volume.
How complex is labeling, and how long does preparation take?
Most of the time goes not to filtering—it takes days—but to labeling and resolving disputed cases: it takes weeks and requires a written instruction with examples of edge cases.
Plus we allow for one or two iterations: after the first check on the held-out set, the rules almost always need refinement. We state these timelines before work begins, not after the fact.
Let's discuss the task: what is needed to evaluate the corpus
Estimating the scope of corpus cleaning work depends on what is already available at the input, so we start the task discussion with the inputs, not with general promises. Below are the factors that most strongly affect timeline estimates and the composition of stages.
- Fine-tuning goals: which class of answers the model should produce and on which examples correctness is checked.
- Corpus sources: where the texts came from, in what form they currently exist, approximate volume in documents and dialogues.
- Labeling and references: whether there are manually verified examples, who selected them and by what rules.
- Known noise: repetitions, fragments, language mixing, service insertions, traces of auto-generation.
- Data constraints: personal information, commercial secrets, storage and access requirements.
- Acceptance criteria: by which metrics you will understand that the corpus is ready for training.
- Operating mode: one-time cleaning or a regular stream with updates and re-checks.
Describe the task in your own words—we will come back with a scope estimate, work plan, and role distribution by stage.
If some inputs have not yet been collected, we will start by clarifying them: this is more reliable than re-labeling the corpus after the model has already been trained.
