# How RLHF and DPO Techniques Align Language Models: Implementation Guide

> Learn how RLHF and DPO align language models. Explore RLHF's reward model and PPO optimization, and DPO's single supervised loss. Implement these techniques with our guide.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: implementation-guide
- Published: 2026-06-16

---

**RLHF uses a three-stage pipeline with a reward model and PPO optimization, while DPO collapses alignment into a single supervised loss by optimizing the log-ratio between the policy and a frozen reference model.**

The `rohitg00/ai-engineering-from-scratch` repository provides a comprehensive curriculum for understanding how modern **RLHF and DPO techniques** transform raw language models into helpful, harmless assistants. This guide examines the theoretical foundations and practical implementations found in the curriculum's Phase 10 and Phase 19 lessons, complete with executable code examples.

## What is RLHF (Reinforcement Learning from Human Feedback)?

**Reinforcement Learning from Human Feedback (RLHF)** is the classic three-stage alignment pipeline that dominates production language model training. According to the curriculum's Phase 10–Lesson 07, RLHF transforms a base model through supervised fine-tuning, reward modeling, and policy optimization.

### The Three-Stage RLHF Pipeline

The pipeline implemented in [`phases/10-llms-from-scratch/07-rlhf/code/train_ppo.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/10-llms-from-scratch/07-rlhf/code/train_ppo.py) follows this strict sequence:

1. **Supervised Fine-Tuning (SFT)** – Train the base model on instruction-following examples to establish basic prompt obedience.
2. **Reward Model (RM) Training** – Present pairs of model outputs to human annotators; train a binary classifier to predict which response is preferred based on the Bradley-Terry preference model.
3. **Proximal Policy Optimization (PPO)** – Fine-tune the language model to maximize the reward model's score while maintaining proximity to the SFT reference via a KL-penalty.

The visualization of this pipeline appears in [`site/figures-llms2.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/figures-llms2.js) (line 169-171), while the metadata entry `RLHF — Reward Model + PPO` in [`site/data.js`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/site/data.js) catalogs the lesson resources.

## What is DPO (Direct Preference Optimization)?

**Direct Preference Optimization (DPO)** eliminates the reward model and PPO stages entirely, collapsing alignment into a single supervised loss function. The curriculum's Phase 19–Lesson 40 (`40-dpo-from-scratch`) derives this approach from first principles.

### The Mathematical Foundation

DPO relies on the **implicit reward** identity that emerges from the RLHF objective:

```

r(x,y) = β·(log πθ(y|x) – log πref(y|x))

```

This equation, explained in [`phases/19-capstone-projects/40-dpo-from-scratch/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/40-dpo-from-scratch/docs/en.md) (line 40-50), treats the log-ratio between the current policy (`πθ`) and the frozen SFT reference (`πref`) as the reward signal.

The **DPO loss function** optimizes the probability gap between the winning response (`y_w`) and losing response (`y_l`):

```python
L_DPO(θ) = –E[(x, y_w, y_l)][log σ(β·(log πθ(y_w|x) – log πθ(y_l|x)))]

```

Here, `β` acts as the KL-budget scaling factor. Unlike RLHF, the KL regularization is enforced implicitly because the loss grows rapidly when the policy diverges from the reference.

### Length Normalization and Practical Engineering

The reference implementation in [`phases/19-capstone-projects/40-dpo-from-scratch/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/40-dpo-from-scratch/code/main.py) includes **length normalization** to prevent the *length-bias* failure mode, where shorter generations receive artificially high scores. The implementation divides the sum of log-probs by the completion length to normalize the reward signal.

## Key Differences Between RLHF and DPO

| Aspect | RLHF | DPO |
|--------|------|-----|
| **Architecture** | Separate reward model + PPO trainer | Single policy model |
| **Training stages** | Three distinct phases (SFT → RM → PPO) | Single stage after SFT |
| **Memory requirements** | Holds policy, reference, and reward models | Holds policy and reference only |
| **KL control** | Explicit penalty term in PPO | Implicit via log-ratio in loss |
| **Failure modes** | Reward model drift, reward hacking | Length bias (mitigated via normalization) |

## Implementation Examples from the Curriculum

### RLHF Training with PPO

The following snippet from [`phases/10-llms-from-scratch/07-rlhf/code/train_ppo.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/10-llms-from-scratch/07-rlhf/code/train_ppo.py) demonstrates the standard PPO implementation using the Hugging Face TRL library:

```python
from trl import PPOTrainer
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("meta-llama/Meta-Llama-3.1-8B")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3.1-8B")

# Load the separately trained reward model

reward_model = AutoModelForCausalLM.from_pretrained("reward-model-checkpoint")

ppo_trainer = PPOTrainer(
    model=model,
    ref_model=model,              # Frozen SFT reference

    reward_model=reward_model,
    kl_coef=0.1,                  # KL penalty coefficient β

    tokenizer=tokenizer,
    ppo_epochs=4,
)
ppo_trainer.train()

```

### DPO Loss Implementation

The implementation in [`phases/19-capstone-projects/40-dpo-from-scratch/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/40-dpo-from-scratch/code/main.py) shows the simplified training loop:

```python
import torch
from torch.nn import functional as F

beta = 0.08                     # KL-budget scaling factor

ref_model = torch.load("sft_ref.pt")   # Frozen SFT reference (π_ref)

policy = torch.load("policy.pt")       # Trainable policy (π_θ)

def dpo_loss(winner_logits, loser_logits):
    # Log-prob differences per token

    logp_w = winner_logits.log_softmax(-1).gather(-1, winner_ids).squeeze(-1)
    logp_l = loser_logits.log_softmax(-1).gather(-1, loser_ids).squeeze(-1)
    # Length-normalized gap

    gap = (logp_w - logp_l).mean(dim=-1) * beta
    return -F.logsigmoid(gap).mean()

optimizer = torch.optim.Adam(policy.parameters(), lr=5e-5)

for batch in pref_loader:       # Each batch contains (x, y_w, y_l)

    win_logits = policy(batch.x, labels=batch.y_w).logits
    los_logits = policy(batch.x, labels=batch.y_l).logits
    loss = dpo_loss(win_logits, los_logits)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

```

### Running the DPO Demo

Execute the toy implementation on a single GPU:

```bash
cd phases/19-capstone-projects/40-dpo-from-scratch
python -m pip install -r requirements.txt
python code/main.py

```

## When to Use RLHF vs DPO

The curriculum outlines specific scenarios for selecting between **RLHF and DPO techniques**:

| Situation | Preferred Method | Reason |
|-----------|------------------|--------|
| Abundant high-quality human preference data and generous compute budget | **RLHF (PPO)** | Robust, well-studied signal; inspectable reward model |
| Limited preference data and memory-constrained environment | **DPO** | No reward model training; single pass over preference pairs |
| Need explicit safety monitoring | **RLHF** | Separate reward model allows monitoring and intervention |
| Concerned about length bias or chosen-response degradation | **DPO with length normalization** or variants like SimPO/BPO | Structural mitigation of bias via normalized objectives |

## Summary

- **RLHF** implements a three-stage pipeline (SFT → Reward Model → PPO) using explicit reward maximization with KL regularization, implemented in `phases/10-llms-from-scratch/07-rlhf/`.
- **DPO** collapses alignment into a single supervised loss using the implicit reward identity `r(x,y) = β·(log πθ(y|x) – log πref(y|x))`, eliminating the need for separate reward model training.
- The curriculum provides runnable implementations in [`phases/19-capstone-projects/40-dpo-from-scratch/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/40-dpo-from-scratch/code/main.py) and [`phases/10-llms-from-scratch/07-rlhf/code/train_ppo.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/10-llms-from-scratch/07-rlhf/code/train_ppo.py).
- DPO includes **length normalization** to prevent the short-generation bias that plagues naive preference optimization.
- Both techniques maintain a frozen reference model (`πref`) to prevent policy drift, but DPO enforces this structurally while RLHF requires explicit KL penalties.

## Frequently Asked Questions

### What is the main advantage of DPO over RLHF?

DPO requires significantly less compute and memory because it eliminates the separate reward model training and PPO optimization stages. According to the curriculum's Phase 19, DPO achieves comparable alignment performance with only a single supervised loss over preference pairs, making it ideal for rapid iteration and resource-constrained environments.

### How does DPO avoid training a separate reward model?

DPO leverages the **reward-difference identity** derived from the RLHF objective with KL regularization. Under the Bradley-Terry preference model, the optimal reward can be expressed purely as the log-ratio between the policy and reference model. By substituting this identity back into the objective, the reward model cancels out, leaving a loss function that depends only on the policy parameters.

### What is length bias in preference optimization?

Length bias occurs when shorter model outputs receive artificially high preference scores because their log-probabilities sum over fewer tokens. The curriculum's DPO implementation in [`phases/19-capstone-projects/40-dpo-from-scratch/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/40-dpo-from-scratch/code/main.py) fixes this by **length-normalizing** the log-probability gap, dividing by the completion length before computing the sigmoid loss.

### Where can I find the implementation code for RLHF and DPO?

The complete implementations are located in the `rohitg00/ai-engineering-from-scratch` repository. RLHF code resides in [`phases/10-llms-from-scratch/07-rlhf/code/train_ppo.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/10-llms-from-scratch/07-rlhf/code/train_ppo.py), while the DPO implementation and derivation appear in [`phases/19-capstone-projects/40-dpo-from-scratch/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/40-dpo-from-scratch/code/main.py) and the accompanying documentation in [`phases/19-capstone-projects/40-dpo-from-scratch/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/40-dpo-from-scratch/docs/en.md).