DPO fine-tuning of LLMs: alignment without RLHF

Why DPO fine-tuning of LLMs stalls on the first run

DPO fine-tuning of LLMs usually fails not at launch, but before it.

The preference dataset is assembled formally: "chosen/rejected response" pairs are labeled by general impression, without criteria, and no one fixes which specific skill should grow — format adherence, refusal of fabricated facts, accuracy on long requests.

The second source of failure is an unfixed reference model. In fine-tuning without RLHF, improvement is measured relative to the reference policy, and if its version changes from run to run, the weights pull in different directions.

Two runs are no longer comparable: it is unclear what produced the effect — the data, the order of pairs, or the updated reference. Ready-made implementations exist in Hugging Face, and the issue is not the launch code but what is fed into it.

Next, degradation after fine-tuning appears. The model wins on labeled scenarios and degrades where there were no similar examples: it holds context worse, agrees with the user instead of objecting, and loses format on compound requests.

The holdout set shows this, but the runs are already spent, and the cause looks like "not enough data."

The root is that model alignment is run as a training experiment rather than as data and reference design. Below we cover how this is addressed: labeling criteria, fixing the reference, and control measurements before and after fine-tuning.

What alignment delivers: effect in measurable quantities

The effect of alignment is measured not by the feeling that "it got better," but by the share of undesirable responses on a fixed set of requests.

The same set is run before and after fine-tuning: the share of responses violating the given policy drops severalfold — from tens of percent to single digits, and this is visible for each request class separately.

The quality of the model's responses should not degrade. Alignment often over-dries the model — it starts refusing where an answer is appropriate.

Therefore, two metrics go into acceptance at once: reduction in violations and preservation of completeness of useful answers on the control set. If completeness drops, we change the composition of preferred pairs, not the refusal threshold.

Robustness on edge requests is checked with its own set: provocations, ambiguous wording, attempts to extract the system instruction, out-of-domain questions.

What matters here is not the average score but the share of correct reactions — how many times the model refused appropriately and did not fall into a template refusal to a harmless question.

Fine-tuning and rerun time is also part of the effect. DPO does not require a separately trained reward model, so the cycle is shorter: we verify a dataset edit with a new run rather than planning a week ahead.

The output is a fixed model version and a protocol with acceptance metrics: share of policy violations, completeness of appropriate answers, robustness on edge requests, and reproducibility of the result between runs.

Work formats: full cycle, audit, or LLM maintenance

Alignment work formats differ in depth of intervention: from analyzing already collected data to supporting the model in production.

The choice is determined not by model size but by the state of the preference pair corpus and the availability of a reproducible response comparison protocol.

If the pairs are heterogeneous, the full fine-tuning cycle will begin with relabeling; if the dataset is valid but evaluation cannot be repeated on the same set, the result can neither be confirmed nor rolled back.

Comparison of modes

The boundaries between modes are fluid: a dataset audit often turns into a full cycle, and maintenance includes periodic retraining on fresh pairs.

The common limitation is one — without a reference set and a fixed comparison order, alignment cannot be measured, so we first check whether evaluation is reproducible, and only then start training.

Implementation stages: from data audit to model acceptance

The order of stages matters more than speed here. If you start with tuning training parameters rather than reviewing the data, the model will learn the noise in the "preferred / rejected" pairs and begin confidently reproducing it in live requests.

Therefore, we build DPO fine-tuning from data to acceptance: each step relies on the artifact from the previous one.

1. Data and metrics audit. We look at what already exists — request logs, operator edit history, ready-made preference pairs. We measure the current quality of the base model on 8–12 scenarios so there is something to compare against.
2. Alignment scheme design. We agree on which pairs we consider preferred and why, how we label disputed cases, and which refusal profile enters training.
3. Dataset assembly. We remove duplicates and mutual contradictions, and split the array into training, validation, and holdout parts before the first run.
4. Fine-tuning runs. We tune the learning rate and number of epochs, and monitor that the model does not lose general utility while quality grows in target scenarios.
5. Holdout validation — on data the model has not seen. We compare with the base version on the same scenarios and look not only at the average result but also at variance across them.
6. Model acceptance. We agree on criteria on real tasks and decide for each one: release, continue training, or rebuild the data.
7. Solution handover. We deliver weights, run configurations, labeling, and retraining regulations, train the team, and set up quality control in production.

Alignment artifacts: dataset, configurations, regulations

We hand over the kit not as "model plus a couple of files." If the client is left with only the final weights, it will no longer be possible to repeat the run, explain the system's behavior, or update it without us — each next cycle starts from scratch.

Therefore, we deliver a set of artifacts where each answers its own question: what it was trained on, what it was launched with, how it was checked, and what to do on the next update.

• Labeled preference dataset — "better response / worse response" pairs with comparison criteria, indication of labelers, and a log of disputed cases. Without it, the next fine-tuning starts over.
• Model card — what data it was aligned on, which tasks it is suitable for, where the limits of applicability are, and known weak points.
• Working configurations — training and evaluation parameters, sample splits, stopping conditions. The run is reproduced one-to-one, not "approximately the same."
• Metrics and evaluation report — what was before alignment and what became after, which sets were used for checking, and how hints were excluded from control examples.
• Update regulations — which events trigger a new cycle: quality drop, influx of fresh data, change in behavior requirements; who approves the release.
• Alignment documentation — accepted trade-offs and prohibitions: why this response profile was chosen and what restrictions are imposed on generation.

Such a kit lives without us: preferences can be supplemented, evaluation can be repeated, and a disputed case can be reviewed using the labeling records.

Case study: aligning a support model without an increase in refusals

Before alignment

The support model on the open Qwen branch answered requests about tariffs and connection.

Based on measurements on a sample of 1,200 dialogues, 11% of responses contained conditions that were not in the tariff grid: the model extrapolated them out of inertia, and the operator learned about the error from the client.

Another 8% of dialogues it closed with a template "clarify your question" instead of escalation, although the topic was within the second line's responsibility.

What was changed

We collected 14 thousand preference pairs from chat history and operator labeling: chosen response versus rejected, including pairs on the topic "escalation needed." We fine-tuned with DPO — without a separate reward model, with a constraint on deviation from the base model so as not to degrade on rare topics.

Negative examples were weighted higher: one confident fabricated answer is more damaging than three extra clarifying questions.

Metrics before and after

Share of responses with unreliable conditions: 11% → 0.4%. Share of correct escalations to the second line: 62% → 94%. First response time did not increase, inference remained on one model — no second network was needed for evaluation.

The refusal class "confident answer outside the competence area" did not reproduce even once on the repeat sample, and measurements a month later on new dialogues gave the same shares: alignment holds, the model did not slide back to its original behavior.

How to distinguish a weak dataset from an under-trained model?

The first thing that separates a weak dataset from an under-trained model is the behavior of the metric on held-out preference pairs the model has not seen.

For an under-trained model, loss on training and validation falls synchronously: add runs — both confidence in selection and quality on new requests grow.

If the margin between the chosen and rejected response grows, but quality on the holdout set stays flat, run volume is irrelevant. Then we look at the labeling: how the model behaves on pairs with an obvious preference and where annotators disagreed with each other.

Noisy labeling reveals itself through an unstable margin: on simple pairs it is narrow and fluctuates from run to run.

Direct dataset diagnostics — send part of the pairs for relabeling and calculate the share of contradictions: with low agreement, the model fixes random decisions, and extra runs only amplify the noise.

Signs of overfitting are visible in the form of responses: the model copies length, openings, and turns from training pairs, loses diversity, grows on training, and degrades on the holdout set.

Often this is not overfitting but leakage — the same topics and formulations appear in both training and validation.

Hypothesis testing is built as a controlled experiment. We fix the alignment scheme and change only the data: remove contradictory pairs, balance the topic distribution.

If the holdout metric comes alive, the issue was labeling; if there is no shift with another scheme or with more runs, the data set the ceiling.

Technical concerns: overfitting, degradation, rollbacks

Before starting alignment, engineers ask the same questions: will the model "chew up" its general vocabulary for narrow behavior, how to roll back if quality drops, and what happens with access to sensitive data. Below are substantive answers, without generalities.

Is it actually possible to roll back a version after alignment?

Yes. The base model remains unchanged, and changes live in a separate layer — rolling back a version takes minutes, without rebuilding the entire pipeline. Each run gets its own artifact and metrics, so the before/after comparison is preserved.

What about the security of changes and data?

Work takes place in an isolated environment: fine-tuning data does not leave your infrastructure, access is granted by roles and logged. Alignment risks — loss of some skills and tone shift — are mitigated by a regression task set that is run before each rollout.

Let's discuss the task: what to send and how we will return with an estimate

For the fine-tuning estimate to rely on data rather than assumptions, we need project inputs. The earlier an example preference set and current quality metrics arrive, the more accurate the work plan and the fewer approval iterations.

• Base model and how it is obtained: open weights or access via an external service, license restrictions on commercial use, whether multilingual support is needed.
• Preference data: "chosen / rejected" response pairs with a brief explanation of who labeled them and by what criterion. If you only have instructions and responses without comparisons, that is a different scenario — we will say so right away.
• Target behavior: what we consider a good answer — adherence to format, refusal of undesirable topics, references to sources, length, and tone.
• A control question set on which you currently see a difference, and the acceptable regression share on it.
• Constraints: legal and domain prohibitions, mandatory warnings, data storage and environment isolation requirements.
• Compute resources for training and launch, and who is responsible for accepting the result on your side.

After that, we return with a fine-tuning estimate, a stage-by-stage work plan, and a collaboration scheme: what we do, what remains on your side, where the approval points are, and how rollback to the previous model is organized. Describe the task in free form — we will discuss the task and ask clarifying questions ourselves.