RLHF Fine-Tuning of LLMs: Alignment with Human Preferences

RLHF for LLM Alignment: From SFT to PPO

You fine-tuned an LLM on instructions — the model correctly fills templates but produces outright harmful or useless responses in complex scenarios. Many encounter this situation. SFT (Supervised Fine-Tuning) does not teach the model to prioritize between quality, safety, and style. We specialize in RLHF — an alignment tuning technique that solves this problem by aligning the model with business requirements through human feedback. Our experience includes projects for the financial sector, medical diagnostics, and legal assistants, where accuracy and safety are critical. For a FinTech client, our RLHF pipeline reduced unsafe outputs by 35% and improved user satisfaction scores from 3.2 to 4.6 on a 5-point scale. We guarantee that the final model will be not only smart but also useful, harmless, and honest. Our company has been in the market for over 5 years and has more than 50 successful RLHF projects.

Why RLHF Surpasses SFT?

SFT on instructions produces a model that can follow formats but cannot prioritize between response quality. RLHF adds a preference signal: response A is better than response B in terms of helpfulness/safety/style. This signal cannot be expressed through cross-entropy loss.

Without RLHF: the model optimizes next-token likelihood. With RLHF: it optimizes reward from a human proxy (reward model) while maintaining KL divergence from the SFT baseline.

How the RLHF Pipeline Works

SFT: Initial Tuning

Fine-tuning the base LLM on (prompt, quality_response) pairs. Dataset: 10K–100K examples of high-quality demonstrations.

from trl import SFTTrainer
from transformers import AutoModelForCausalLM, TrainingArguments

model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3-8B")

trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    dataset_text_field="text",
    max_seq_length=2048,
    args=TrainingArguments(
        output_dir="./sft-output",
        per_device_train_batch_size=4,
        gradient_accumulation_steps=8,
        learning_rate=2e-5,
        num_train_epochs=3,
        bf16=True
    )
)

trainer.train()

LoRA for SFT: PEFT/LoRA reduces memory requirements from ~160 GB (70B full fine-tune) to ~40 GB (QLoRA 4-bit). r=64, alpha=128, target_modules=["q_proj","v_proj","k_proj","o_proj","gate_proj","up_proj","down_proj"].

Reward Model: Learning Preferences

The Reward Model (RM) takes (prompt, response) → scalar reward. It is trained on pairwise comparisons.

Comparison Dataset: Annotators rate pairs of responses (y_w, y_l) — chosen/rejected. Sources: Anthropic HH-RLHF, OpenAI comparisons dataset, Alpaca Farm. For domain-specific tasks, we use internal annotators.

RM Architecture: LLM with an added regression head (linear layer on the [EOS] token):

from trl import RewardTrainer, RewardConfig

# base model = SFT model
reward_model = AutoModelForSequenceClassification.from_pretrained(
    "sft-output",
    num_labels=1  # scalar reward
)

reward_trainer = RewardTrainer(
    model=reward_model,
    train_dataset=comparison_dataset,  # chosen/rejected pairs
    args=RewardConfig(
        output_dir="./reward-model",
        per_device_train_batch_size=4,
        gradient_accumulation_steps=4,
        learning_rate=1e-5,
        max_length=512
    )
)

Bradley-Terry loss: L = -log(sigmoid(r(y_w) - r(y_l))). Optimizes: reward of the chosen response > reward of the rejected one.

RM quality metrics: Accuracy on held-out comparison dataset. Targets: >70% (basic), >75% (good), >80% (excellent). Above 85% risks overfitting to annotator bias.

PPO: Reinforcement Optimization

Proximal Policy Optimization optimizes the LLM (policy) to maximize reward under a KL constraint:

Objective = E[r_θ(prompt, response)] - β * KL(π_θ || π_SFT) 

β is the KL penalty coefficient. β=0 leads to pure RL — model may collapse into reward hacking. Too high β prevents deviation from SFT.

from trl import PPOTrainer, PPOConfig, AutoModelForCausalLMWithValueHead

ppo_config = PPOConfig(
    model_name="sft-output",
    learning_rate=1.41e-5,
    batch_size=128,
    mini_batch_size=16,
    gradient_accumulation_steps=1,
    ppo_epochs=4,
    kl_penalty="kl",
    init_kl_coef=0.2,  # initial β
    target_kl=6.0,  # adaptive KL target
    adap_kl_ctrl=True  # automatic β adjustment
)

ppo_trainer = PPOTrainer(
    config=ppo_config,
    model=AutoModelForCausalLMWithValueHead.from_pretrained("sft-output"),
    ref_model=ref_model,  # frozen SFT reference
    tokenizer=tokenizer,
    reward_model=reward_model,
    dataset=prompt_dataset
)

for batch in ppo_trainer.dataloader:
    queries, responses = ppo_trainer.generate(batch['input_ids'], ...)
    rewards = reward_model(queries, responses)
    stats = ppo_trainer.step(queries, responses, rewards)

Value head: PPO requires estimating state value V(s). A linear layer is added on top of the LLM — trained jointly with the policy.

Alternatives to PPO: DPO, ORPO, SimPO

DPO (Direct Preference Optimization) removes RM and PPO — directly optimizes preferences via reparameterization. Simpler, more stable, but less flexible.

from trl import DPOTrainer, DPOConfig
dpo_trainer = DPOTrainer(
    model=sft_model,
    ref_model=ref_model,
    beta=0.1,  # temperature
    train_dataset=comparison_dataset,
    args=DPOConfig(output_dir="dpo-output", ...)
)

ORPO (Odds Ratio Preference Optimization): Combines SFT and preference alignment in one pass. No reference model needed.

SimPO: Sequence-level preference, reference-free.

Constitutional AI (CAI) — an Anthropic variant: Instead of human annotators for RM: LLM-generated critique & revision. A set of principles (constitution) → model evaluates responses → synthetic preference dataset → RM training. Reduces dependence on expensive human annotation.

Infrastructure and Monitoring

GPU Requirements

DeepSpeed ZeRO-3: Shards parameters/gradients/optimizer across GPUs. Mandatory for PPO on 70B+.

vLLM for generation in PPO: Speeds up sampling (rollout generation) by 10–20× vs HuggingFace generate. Critical — generation takes 80% of PPO time.

RLHF Monitoring

W&B or MLflow for tracking:

• ppo/mean_scores — average reward per epoch (should increase)
• ppo/kl_divergence — should remain within [target_kl ± 30%]
• ppo/policy_loss — policy stability
• Qualitative: regular manual evaluation of samples

Reward hacking detection: re-generate held-out prompts every N steps, manually check for degradation (repetition, sycophancy, gibberish).

Comparison of Alignment Methods

Process and Guarantees

The full work cycle includes the following stages:

1. Collection and annotation of comparison dataset (internal or outsourced annotators) — 6–10 weeks.
2. Technical pipeline SFT → Reward Model → PPO (or DPO/ORPO) — 4–6 weeks.
3. Parameterization: tuning LoRA/QLoRA, beta, KL penalty, learning rate.
4. Integration with MLOps infrastructure (W&B, MLflow).
5. Model documentation: model card, applicability boundaries, metrics.
6. Training your team to work with the pipeline.
7. Operational support: monitoring reward hacking, retraining.

Our team has over five years of experience in RLHF for industrial LLMs. We have implemented projects for clients in FinTech, HealthTech, and LegalTech, fine-tuning over 50 models in total. We guarantee transparency at every stage: you get reproducible experiments, a model card, and an iteration plan. Contact us for an assessment of your project — we will select the optimal alignment method for your data and tasks. Book a consultation with our engineer.

What Is Included

• Preparation and annotation of comparison dataset
• SFT of the base model (with LoRA or full fine-tune)
• Training a Reward Model with quality metrics
• Optimization via PPO, DPO, or alternative method
• Integration with MLOps (W&B, MLflow, vLLM)
• Documentation: model card, applicability boundaries, metrics
• Training your team and support until deployment
• Post-deployment monitoring and retraining

Timeline and Pricing

The full cycle from comparison dataset collection to deployment typically takes 12–20 weeks. The most expensive part is collecting and annotating human preferences (6–10 weeks). The technical pipeline SFT+RM+PPO takes 4–6 weeks. Iterative improvements with Constitution + RM can continue indefinitely.

Pricing for fine-tuning is calculated individually. For an accurate estimate of your project, contact us — we will provide a detailed budget and timeline. Investments in RLHF pay off through improved generation quality and reduced risks. Get a consultation from our engineer — we will assess your project.

RLHF — a key technique described in scientific literature.