Fine-tuning LLM: Dataset Preparation — Formats, Deduplication, and Checklist

Why the LLM dataset determines fine-tuning quality

Fine-tuning rarely fails on training settings: the model confidently passes the held-out evaluation and falls apart on live queries. The reason is not the number of steps, but the quality of the training sample. A raw data slice gives a nice number on evaluation and a weak result in production.

The first source of failure is duplicates in the sample. The same example in several formulations makes the model memorize a specific phrase instead of the pattern, and an inflated rare class pulls answers onto itself at the expense of others.

Data preparation for fine-tuning begins with normalization and semantic deduplication, not exact string matching.

The second is leakage of the validation part into training. Paraphrases, template answers, and fragments of the same document remaining in both sets turn validation into fiction: the metric grows, but behavior on new data does not change. Therefore, splitting is done by sources and entities, not by lines.

The third is inconsistent labeling: annotators interpret the instruction in their own way, somewhere a field is renamed, somewhere an answer is truncated, somewhere extra text is added. The model diligently learns this noise.

An LLM dataset is considered ready when the field schema is fixed, disputed examples are removed, and class proportions and answer lengths are checked across slices.

Next, we analyze how this is addressed: deduplication rules, group splitting, a unified labeling schema, and a sample acceptance checklist before launching training.

What a verified sample provides: effect in measurable metrics

The main risk of fine-tuning is not the architecture, but the composition of the sample. Duplicates and near-duplicates make the model consider a repeated formulation the main signal, and conflicting labeling on the same examples teaches it to guess instead of answer.

After removing near-duplicates, bringing labeling to a single agreement, and reviewing disputed examples, the share of correct answers on the held-out validation set grows by 8–15 points.

And the gap between training and validation parts shrinks — meaning the gain comes from signal in the data, not memorization.

The number of repeated fine-tuning runs drops from six–eight to two–three. Deduplication removes examples that the model assigns either to one class or another, and training stops fluctuating between epochs in search of a compromise.

On live traffic, this reads as stability: the share of refusals and evasions decreases, the output format holds even on formulations that were not in the sample.

Separately, we calculate the share of truncated answers — this is an early sign that there is still little data, not that the model “can't handle it.”

We curate in Argilla: dataset versions, labeling by multiple annotators, and edit history in one place — after each iteration, you can see exactly which change produced the gain.

We calculate metrics on a frozen control set that does not enter training, so the result is reproducible on the next run.

LLM dataset formats and fine-tuning modes: comparison

Labeled data formats and fine-tuning modes are chosen by task, not by habit. The same corpus can be laid out as query-answer pairs, instructions, or role-based dialogues — this determines which scenarios the model will see during training at all.

Below is a comparison of labeling options and fine-tuning modes by suitability and limitations; a unified template, for example ChatML with role service tokens, allows a single sample to cover both instructions and dialogues.

You can mix formats in one sample, but the template must be single: different role markers in the dataset cause divergence in model behavior on similar queries.

A working technique is to bring everything to one template, fix it in the training configuration, and store the labeling version next to the data.

Then a repeated run reproduces the result, and comparison of fine-tuning modes is done on identical data, not on different dataset versions.

Data preparation stages: from source audit to acceptance

The order of stages here is not a formality: if labeling starts before the source audit, part of the corpus has to be redone — the data does not cover the needed scenarios or contains fragments that cannot be used.

Therefore, we first fix what we are assembling the sample from, and only then spend resources on labeling.

1. Source audit. Inventory: request logs, knowledge base, support correspondence, internal regulations. For each source — owner, right to use, volume, domain coverage, and share of corrupted records.
2. Record schema. We agree on fields, a unified storage format, and rules for describing the reference assistant's answer, so annotators do not diverge in interpretation.
3. Cleaning and deduplication. We remove markup of markup (HTML, service headers), normalize text, catch exact and near-duplicates, cut out personal data. Close repetitions skew training more than noise.
4. Labeling and cross-checking. Each record is checked by a second specialist; disputed cases are reviewed separately and added to the rules.
5. Splitting into training and validation parts. We split by sources and semantic clusters, not randomly — otherwise similar formulations will end up in both parts and validation quality will be inflated.
6. Control measurement. We run the fine-tuned model on the validation part and compare with the baseline on scenarios important to the customer.
7. Dataset acceptance. Checklist by composition, statistics, sample version, and description of limitations — what the model will not be able to master on this data.

Acceptance closes the stage when the customer sees not a “finished dataset” but a clear artifact: how many records, of what quality, with what exceptions. This way, the next fine-tuning cycle starts from a known point, not from re-analyzing sources.

Field schemas, labeling rules, and reassembly regulations

We hand over the documentation package together with the trained model — otherwise the sample cannot be reproduced or extended without us.

Below is listed what remains on the customer's side and works after delivery: it is enough to hand it to a new team of annotators or an engineer who will continue the fine-tuning cycle.

What is included in the delivered set:

• Labeling field schema: required and optional fields, value types, allowed ranges, rules for linking a record to the source document and its version.
• Labeling rules and annotator instruction: edge cases for each class, what to do in case of ambiguity, which interpretations to avoid, escalation procedure for disputed examples.
• Reference examples: several labeled records per class that serve as a cross-check point when hiring new annotators.
• Data quality check scripts: duplicate search by normalized text, control of empty and truncated fields, checking class balance and record lengths, comparing the schema with the actual file content.
• Labeling control report: share of matches between annotators on the overlapped part of the sample, list of discrepancies with analysis of causes.
• Sample update regulations: reassembly frequency, who initiates, by what criterion it is considered that the model has degraded and data is insufficient.
• Dataset version log: what changed and when, which version each release was trained on, how this relates to measurement results.

Such documents remove the main risk of retraining — the discrepancy between how data was labeled the first time and how it will be labeled six months later. When the schema, rules, and checks are fixed, updating the sample turns into a regulatory procedure, not a new project. Describe the task — we will come back with an estimate and work plan.

Case study: how sample reassembly removed a stream of junk answers

The client is an internal assistant for the regulatory framework: employees ask about regulations, the model answers with a reference to a clause. The first version was fine-tuned on a dump from first-line tickets — about fifty thousand question-answer pairs over a year.

In demonstrations, everything looked smooth, but on live queries the model referenced revoked orders and glued together clauses from different documents.

What was found in the sample

A third of the pairs turned out to be duplicates down to punctuation — the model memorized frequent formulations instead of the rule. In 12% of cases, one question had two different answers from different operators: conflicting labeling, and the model averaged the contradictions.

In another 7%, the answer was glued from an adjacent row of the dump — an export failure, not human error. Labeling was done in Label Studio, but without a unified instruction: each operator understood “answer briefly” in their own way.

What was changed

We assembled an instruction from four criteria: reference to a specific clause, answer only according to the current revision, fixed length, refusal instead of a guess.

Deduplication was by normalized text, and conflicting pairs went to a lawyer for arbitration, rather than being resolved by voting. The sample shrank from 48 to 29 thousand examples, but stopped teaching contradictions.

On live queries, the share of answers referencing a non-existent clause fell from 22% to 3%, and the share of honest refusals “I don't find it in the regulation” grew from 4% to 15% — the model stopped guessing.

Escalations to a live lawyer became noticeably fewer: the assistant stopped being a source of false references.

What to do if the data is already labeled in a different format?

Full relabeling is rarely needed: in a typical corpus, 80–90% of records are transferred automatically, because the old and new schemas differ in details, not in essence.

Labeling transfer is done as a separate layer between the source data and the training representation, not by editing files in place. The original labeling remains unchanged — this is a point you can always return to.

The work starts with a formal description of the target schema: required fields, allowed label list, rules for labeling entity boundaries, what is considered a negative example.

Then a converter is written that lays out the old labeling into these fields and brings data from all sources to a unified schema — one label dictionary, one boundary type, one dialogue representation.

On a control sample of several thousand records, they verify that the transfer loses nothing and does not duplicate labels before running it on the full volume.

Some records do not fit into the conversion: multiple labels on one fragment, nested turns, labels that simply do not exist in the new schema.

Such cases are not “completed” automatically, but put into a queue for partial relabeling — usually this is 5–15% of the volume, and they provide the main quality gain. Manually relabeling the rest is redoing work already done.

Preserving edit history here is not a formality: each edit is a separate record with author, date, and reason, and the converter itself lives as code under version control.

If the schema changes again, the conversion is rerun on the entire corpus in minutes, not rebuilt from scratch. This also allows training the model on the old and new labeling and comparing the result on the same validation sample before switching to the new one.

Deduplication, licenses, and privacy: engineers' questions

During task analysis, the same set of doubts almost always comes up: how to filter out nearly identical examples that do not match word-for-word but repeat the same answer, and what to do with rights to sources.

We answer them before the start, because reassembling the sample after labeling is the most expensive rework in the project.

Below are four questions that come up more often than others, with brief answers from engineers. If your case is not among them, we will review it separately.

What do you check in licenses for training data?

We maintain a source registry: license type, whether commercial use is allowed, whether attribution is required, whether there are restrictions on derivative datasets. Sources with unclear terms are either excluded or replaced.

The output is a sample passport: where the data came from, how it was obtained, what is forbidden to do with it.

How does personal data get into the sample and how to remove it?

We label potentially sensitive fragments before training — names, contacts, addresses, document numbers — using, among other things, Presidio. Then masking and manual review of disputed places on the sample. For small corpora, it is cheaper to clean the source than to rely on a filter.

How to make sure the model has not memorized the data?

Leakage check is a separate stage: we feed prefix queries from the corpus and measure in what share of cases the model restores the continuation word-for-word. If the share is noticeable, we strengthen deduplication and masking and repeat the measurement. We record the result in a report — with it, you can see that the leakage risk is manageable.

Let's discuss the task: what data slice is needed for an estimate

To come back with an estimate of the scope of work and a sample preparation plan, we do not need the entire data corpus — a representative slice is enough.

A few hundred real examples show the structure, the share of duplicates and junk, the spread of answer lengths, and how much manual work labeling will require. From such a slice, labor intensity is calculated based on facts, not on a task description.

• List of source materials: where the data comes from — support requests, correspondence, internal knowledge base, documents — and in what form they are stored now.
• Data slice for evaluation: examples of “question — expected answer” exactly in the form in which they occur in work, including unsuccessful and incomplete ones.
• Reference labeling: 20–30 examples labeled manually by your specialist — we use them to calibrate quality checks and cleaning rules.
• Glossary: your accepted product names, terms, abbreviations, and spelling variants, so the model does not distort the subject area.
• Restrictions: topics and formulations that must not be in answers, and rules of behavior for disputed or incomplete queries.
• Answer requirements and acceptance criteria: what you consider a correct result, and what is an error requiring sample rework.

Send what you already have — even an incomplete list and an unlabeled dump will provide enough material for analysis.

Let's discuss the task: we will show where the slice is heterogeneous, which fields will need to be brought to a unified form, and how many examples need to be collected to reach the target quality.

After that, we will come back with an estimate of the scope of work and a stage-by-stage sample preparation plan.