How RLHF and DPO Techniques Align Language Models: Implementation Guide

RLHF uses a three-stage pipeline with a reward model and PPO optimization, while DPO collapses alignment into a single supervised loss by optimizing the log-ratio between the policy and a frozen reference model.

The rohitg00/ai-engineering-from-scratch repository provides a comprehensive curriculum for understanding how modern RLHF and DPO techniques transform raw language models into helpful, harmless assistants. This guide examines the theoretical foundations and practical implementations found in the curriculum's Phase 10 and Phase 19 lessons, complete with executable code examples.

What is RLHF (Reinforcement Learning from Human Feedback)?

Reinforcement Learning from Human Feedback (RLHF) is the classic three-stage alignment pipeline that dominates production language model training. According to the curriculum's Phase 10–Lesson 07, RLHF transforms a base model through supervised fine-tuning, reward modeling, and policy optimization.

The Three-Stage RLHF Pipeline

The pipeline implemented in phases/10-llms-from-scratch/07-rlhf/code/train_ppo.py follows this strict sequence:

1. Supervised Fine-Tuning (SFT) – Train the base model on instruction-following examples to establish basic prompt obedience.
2. Reward Model (RM) Training – Present pairs of model outputs to human annotators; train a binary classifier to predict which response is preferred based on the Bradley-Terry preference model.
3. Proximal Policy Optimization (PPO) – Fine-tune the language model to maximize the reward model's score while maintaining proximity to the SFT reference via a KL-penalty.

The visualization of this pipeline appears in site/figures-llms2.js (line 169-171), while the metadata entry RLHF — Reward Model + PPO in site/data.js catalogs the lesson resources.

What is DPO (Direct Preference Optimization)?

Direct Preference Optimization (DPO) eliminates the reward model and PPO stages entirely, collapsing alignment into a single supervised loss function. The curriculum's Phase 19–Lesson 40 (40-dpo-from-scratch) derives this approach from first principles.

The Mathematical Foundation

DPO relies on the implicit reward identity that emerges from the RLHF objective:

r(x,y) = β·(log πθ(y|x) – log πref(y|x))

This equation, explained in phases/19-capstone-projects/40-dpo-from-scratch/docs/en.md (line 40-50), treats the log-ratio between the current policy (πθ) and the frozen SFT reference (πref) as the reward signal.

The DPO loss function optimizes the probability gap between the winning response (y_w) and losing response (y_l):

L_DPO(θ) = –E[(x, y_w, y_l)][log σ(β·(log πθ(y_w|x) – log πθ(y_l|x)))]

Here, β acts as the KL-budget scaling factor. Unlike RLHF, the KL regularization is enforced implicitly because the loss grows rapidly when the policy diverges from the reference.

Length Normalization and Practical Engineering

The reference implementation in phases/19-capstone-projects/40-dpo-from-scratch/code/main.py includes length normalization to prevent the length-bias failure mode, where shorter generations receive artificially high scores. The implementation divides the sum of log-probs by the completion length to normalize the reward signal.

Key Differences Between RLHF and DPO

Aspect	RLHF	DPO
Architecture	Separate reward model + PPO trainer	Single policy model
Training stages	Three distinct phases (SFT → RM → PPO)	Single stage after SFT
Memory requirements	Holds policy, reference, and reward models	Holds policy and reference only
KL control	Explicit penalty term in PPO	Implicit via log-ratio in loss
Failure modes	Reward model drift, reward hacking	Length bias (mitigated via normalization)

Implementation Examples from the Curriculum

RLHF Training with PPO

The following snippet from phases/10-llms-from-scratch/07-rlhf/code/train_ppo.py demonstrates the standard PPO implementation using the Hugging Face TRL library:

from trl import PPOTrainer
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("meta-llama/Meta-Llama-3.1-8B")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3.1-8B")

# Load the separately trained reward model

reward_model = AutoModelForCausalLM.from_pretrained("reward-model-checkpoint")

ppo_trainer = PPOTrainer(
    model=model,
    ref_model=model,              # Frozen SFT reference

    reward_model=reward_model,
    kl_coef=0.1,                  # KL penalty coefficient β

    tokenizer=tokenizer,
    ppo_epochs=4,
)
ppo_trainer.train()

DPO Loss Implementation

The implementation in phases/19-capstone-projects/40-dpo-from-scratch/code/main.py shows the simplified training loop:

import torch
from torch.nn import functional as F

beta = 0.08                     # KL-budget scaling factor

ref_model = torch.load("sft_ref.pt")   # Frozen SFT reference (π_ref)

policy = torch.load("policy.pt")       # Trainable policy (π_θ)

def dpo_loss(winner_logits, loser_logits):
    # Log-prob differences per token

    logp_w = winner_logits.log_softmax(-1).gather(-1, winner_ids).squeeze(-1)
    logp_l = loser_logits.log_softmax(-1).gather(-1, loser_ids).squeeze(-1)
    # Length-normalized gap

    gap = (logp_w - logp_l).mean(dim=-1) * beta
    return -F.logsigmoid(gap).mean()

optimizer = torch.optim.Adam(policy.parameters(), lr=5e-5)

for batch in pref_loader:       # Each batch contains (x, y_w, y_l)

    win_logits = policy(batch.x, labels=batch.y_w).logits
    los_logits = policy(batch.x, labels=batch.y_l).logits
    loss = dpo_loss(win_logits, los_logits)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

Running the DPO Demo

Execute the toy implementation on a single GPU:

cd phases/19-capstone-projects/40-dpo-from-scratch
python -m pip install -r requirements.txt
python code/main.py

When to Use RLHF vs DPO

The curriculum outlines specific scenarios for selecting between RLHF and DPO techniques:

Situation	Preferred Method	Reason
Abundant high-quality human preference data and generous compute budget	RLHF (PPO)	Robust, well-studied signal; inspectable reward model
Limited preference data and memory-constrained environment	DPO	No reward model training; single pass over preference pairs
Need explicit safety monitoring	RLHF	Separate reward model allows monitoring and intervention
Concerned about length bias or chosen-response degradation	DPO with length normalization or variants like SimPO/BPO	Structural mitigation of bias via normalized objectives

Summary

• RLHF implements a three-stage pipeline (SFT → Reward Model → PPO) using explicit reward maximization with KL regularization, implemented in phases/10-llms-from-scratch/07-rlhf/.
• DPO collapses alignment into a single supervised loss using the implicit reward identity r(x,y) = β·(log πθ(y|x) – log πref(y|x)), eliminating the need for separate reward model training.
• The curriculum provides runnable implementations in phases/19-capstone-projects/40-dpo-from-scratch/code/main.py and phases/10-llms-from-scratch/07-rlhf/code/train_ppo.py.
• DPO includes length normalization to prevent the short-generation bias that plagues naive preference optimization.
• Both techniques maintain a frozen reference model (πref) to prevent policy drift, but DPO enforces this structurally while RLHF requires explicit KL penalties.

Frequently Asked Questions

What is the main advantage of DPO over RLHF?

DPO requires significantly less compute and memory because it eliminates the separate reward model training and PPO optimization stages. According to the curriculum's Phase 19, DPO achieves comparable alignment performance with only a single supervised loss over preference pairs, making it ideal for rapid iteration and resource-constrained environments.

How does DPO avoid training a separate reward model?

DPO leverages the reward-difference identity derived from the RLHF objective with KL regularization. Under the Bradley-Terry preference model, the optimal reward can be expressed purely as the log-ratio between the policy and reference model. By substituting this identity back into the objective, the reward model cancels out, leaving a loss function that depends only on the policy parameters.

What is length bias in preference optimization?

Length bias occurs when shorter model outputs receive artificially high preference scores because their log-probabilities sum over fewer tokens. The curriculum's DPO implementation in phases/19-capstone-projects/40-dpo-from-scratch/code/main.py fixes this by length-normalizing the log-probability gap, dividing by the completion length before computing the sigmoid loss.

Where can I find the implementation code for RLHF and DPO?

The complete implementations are located in the rohitg00/ai-engineering-from-scratch repository. RLHF code resides in phases/10-llms-from-scratch/07-rlhf/code/train_ppo.py, while the DPO implementation and derivation appear in phases/19-capstone-projects/40-dpo-from-scratch/code/main.py and the accompanying documentation in phases/19-capstone-projects/40-dpo-from-scratch/docs/en.md.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}