RLHF vs DPO: Comparing Reward Model + PPO and Direct Preference Optimization in AI Engineering

Direct Preference Optimization (DPO) eliminates the explicit reward model and PPO training loop by re-parameterizing the RLHF objective into a single supervised loss, achieving mathematical equivalence to three-stage RLHF with significantly lower compute costs and memory requirements.

The AI Engineering from Scratch curriculum maintained by rohitg00 implements both alignment approaches side-by-side. This guide examines how RLHF (Reward Model + PPO) and DPO (Direct Preference Optimization) differ in pipeline architecture, mathematical foundations, and practical training costs based on the source implementations.

Core Architectural Differences

The curriculum highlights a fundamental divergence in how each method approaches preference alignment.

The Three-Stage RLHF Pipeline

Classic RLHF requires three sequential training phases. First, Supervised Fine-Tuning (SFT) adapts the base model to instruction-following behavior. Second, a dedicated reward model is trained on human-rated preference pairs to learn a latent scoring function r(x,y). Third, Proximal Policy Optimization (PPO) updates the policy to maximize predicted rewards while enforcing a KL-divergence penalty to remain close to the SFT reference. This architecture requires maintaining three separate models in memory during the PPO phase.

The Two-Model DPO Approach

DPO collapses this architecture into two components: a frozen reference model (initialized from the SFT checkpoint) and a trainable policy model. As implemented in phases/19-capstone-projects/40-dpo-from-scratch/code/main.py, DPO skips explicit reward modeling entirely. Instead, it trains the policy directly on preference pairs using a single supervised loss that computes log-probability ratios between the policy and reference. The KL constraint is baked into the loss formulation rather than enforced as a separate penalty term.

Mathematical Foundations and Equivalence

Both methods derive from the Bradley-Terry preference model, but DPO eliminates the intermediate reward representation through algebraic re-parameterization.

The RLHF Objective

In the RLHF lesson (phases/10-llms-from-scratch/07-rlhf/), the PPO objective maximizes:

max_π E[r(x,y)] - β KL(π||π_ref)

The reward model r is learned first, then plugged into this optimization. The curriculum emphasizes that this requires careful tuning of the KL penalty coefficient β to prevent reward hacking.

The DPO Loss Derivation

According to the DPO lesson documentation (phases/19-capstone-projects/40-dpo-from-scratch/docs/en.md), starting from the optimal policy form:

π*(y|x) ∝ π_ref(y|x) e^(r(x,y)/β)

One can derive the DPO loss that eliminates r:

def dpo_loss(log_pi_w, log_pi_ref_w, log_pi_l, log_pi_ref_l, beta=1.0):
    """
    Compute DPO loss for (chosen, rejected) pairs.
    From phases/19-capstone-projects/40-dpo-from-scratch/docs/en.md lines 62-66.
    """
    diff = beta * (log_pi_w - log_pi_ref_w - log_pi_l + log_pi_ref_l)
    loss = -F.logsigmoid(diff).mean()
    return loss

This implements the mathematical equivalence:

L_DPO(θ) = -E[log σ(β(log π_θ(y_w|x) - log π_ref(y_w|x) - log π_θ(y_l|x) + log π_ref(y_l|x)))]

The gradient analysis in the lesson (lines 88-94) confirms that increasing the chosen completion's log-probability decreases loss, while increasing the rejected completion's log-probability increases it.

Implementation Complexity and Compute Costs

The source code reveals stark operational differences between the approaches.

RLHF (Reward Model + PPO) requires:

• Three separate model instances (SFT reference, reward model, policy)
• Memory-heavy PPO loops with two models in memory simultaneously
• Reward model forward/backward passes plus PPO updates
• Explicit KL-penalty tuning to avoid reward hacking

DPO requires:

• Only two models (reference and policy)
• Single forward-backward pass per preference pair with no inner optimization loop
• Implicit KL constraint through the log-ratio difference
• Lower memory footprint and faster training cycles

As noted in site/data.js (lines 1858-1861), DPO is catalogued specifically for "limited compute budgets" and "fast prototyping," while RLHF is recommended for scenarios requiring rich reward signal modeling.

Practical Code Comparison

The curriculum provides minimal implementations highlighting the pipeline differences.

DPO Training Loop

From phases/19-capstone-projects/40-dpo-from-scratch/code/main.py:

# Initialize from SFT checkpoint

reference = TinyGPT()
reference.load_state_dict(sft_weights)
reference.eval()  # frozen

policy = TinyGPT()
policy.load_state_dict(sft_weights)  # start from same point

optimizer = torch.optim.AdamW(policy.parameters(), lr=1e-5)

for epoch in range(30):
    for batch in pref_data:  # (prompt, chosen, rejected) triples

        # Compute log-probs under both models

        log_pi_w = policy.log_prob(batch.prompt, batch.chosen)
        log_pi_l = policy.log_prob(batch.prompt, batch.rejected)
        
        with torch.no_grad():
            log_pi_ref_w = reference.log_prob(batch.prompt, batch.chosen)
            log_pi_ref_l = reference.log_prob(batch.prompt, batch.rejected)
        
        # Single loss computation and update

        loss = dpo_loss(log_pi_w, log_pi_ref_w, log_pi_l, log_pi_ref_l, beta=0.1)
        loss.backward()
        optimizer.step()

RLHF PPO Training Loop

From the RLHF lesson (phases/10-llms-from-scratch/07-rlhf/):

# Requires pre-trained reward model

reward_model = RewardModel()
reward_model.load_state_dict(rm_weights)

policy = TinyGPT()
policy.load_state_dict(sft_weights)

for epoch in range(ppo_epochs):
    # Sample completions

    completions = policy.generate(prompts)
    
    # Evaluate rewards (separate model)

    rewards = reward_model(prompts, completions)
    
    # Compute KL to reference

    logp_policy = policy.log_prob(prompts, completions)
    logp_ref = reference.log_prob(prompts, completions)
    kl_term = (logp_policy - logp_ref).mean()
    
    # PPO clipped surrogate objective

    surrogate = (rewards - beta * kl_term).mean()
    loss = -surrogate  # maximize

    
    loss.backward()
    optimizer.step()

The DPO loop executes one optimizer step per batch, while the RLHF implementation requires reward model inference plus multiple PPO epochs, illustrating the computational savings.

When to Use Each Approach

The curriculum identifies specific scenarios favoring each method.

Choose DPO when:

• Operating with limited GPU memory or compute budgets
• Working with small-to-medium preference datasets where a learned reward model would be noisy
• Prioritizing fast research iteration and prototyping cycles
• Implementing alignment for smaller models (e.g., TinyGPT experiments)

Choose RLHF (Reward Model + PPO) when:

• Scaling to very large preference corpora where a dedicated reward model captures richer signal
• Requiring multi-objective alignment where the reward function needs shaping independent of the policy
• The implicit reward representation (log-ratio) proves insufficient for complex preference structures

Summary

• RLHF uses a three-stage pipeline (SFT → Reward Model → PPO) with explicit KL penalties, requiring three models and higher compute.
• DPO achieves mathematical equivalence through a single supervised loss on preference pairs, requiring only two models and lower memory.
• The DPO loss computed in phases/19-capstone-projects/40-dpo-from-scratch/code/main.py eliminates the reward model by re-parameterizing the Bradley-Terry model via log-probability ratios.
• DPO is optimal for compute-constrained environments and rapid prototyping, while RLHF excels with large-scale data and complex reward shaping requirements.

Frequently Asked Questions

Is DPO mathematically equivalent to RLHF?

Yes, according to the derivation in phases/19-capstone-projects/40-dpo-from-scratch/docs/en.md, DPO is mathematically equivalent to RLHF under the Bradley-Terry preference model. The key insight is that the optimal policy under RLHF has a closed-form solution that can be re-parameterized to eliminate the explicit reward function, resulting in the DPO loss that operates directly on preference pairs.

Why does DPO require less memory than RLHF?

DPO requires only two models in memory (the frozen reference and the trainable policy), whereas PPO-based RLHF typically requires three models simultaneously: the SFT reference, the reward model, and the current policy being optimized. Additionally, RLHF's PPO loop involves multiple forward passes through both the policy and reward model per update, while DPO performs a single forward-backward pass per batch.

Can I use DPO for large-scale production models?

Yes, though the curriculum notes trade-offs. DPO scales efficiently to large models because it avoids the memory overhead of maintaining a separate reward model and the computational cost of PPO's inner optimization loop. However, for very large preference corpora with complex, multi-dimensional quality criteria, a dedicated reward model (RLHF) may capture richer supervisory signals than the implicit reward representation used by DPO.

How does the beta hyperparameter function in DPO compared to RLHF?

In both methods, beta controls the strength of the KL divergence constraint between the optimized policy and the reference model. In RLHF, beta scales an explicit KL penalty term in the PPO objective. In DPO, beta scales the log-probability ratios inside the sigmoid loss function. According to the lesson documentation, higher beta values enforce stronger adherence to the reference policy, while lower values allow greater deviation to satisfy preferences.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}