RLHF vs DPO: Understanding the Difference Between RLHF and DPO Optimization Techniques

RLHF trains a separate reward model and optimizes via a multi-step PPO pipeline, while DPO directly optimizes the policy on preference data without an explicit reward model, reducing training complexity from three loops to one.

The ai-engineering-from-scratch repository covers both alignment methods in Phase 10 (LLMs from Scratch), specifically in lessons 07 and 08. While both techniques aim to align language models with human preferences, they differ fundamentally in their approach to optimization and computational requirements. Understanding these differences is crucial for implementing efficient preference tuning pipelines.

Core Architectural Differences

RLHF: The Three-Stage Reward Model Approach

RLHF treats alignment as a classic reinforcement learning problem with a learned reward function. According to the source code in phases/10-llms-from-scratch/07-rlhf/code/main.py, the pipeline first trains a reward model on human-annotated preference data, then optimizes the policy using a PPO-style reinforcement learning loop that maximizes the learned reward while keeping the policy close to the supervised-fine-tuned (SFT) model via a KL-penalty.

This approach requires 3-4 models simultaneously in memory:

• The SFT reference model
• The trained reward model
• The current policy being updated
• A reference copy for KL-regularization

DPO: Direct Policy Optimization

In contrast, DPO eliminates the separate reward model entirely. As implemented in phases/10-llms-from-scratch/08-dpo/code/main.py, DPO directly optimizes the policy to prefer chosen completions over rejected ones using a simple pairwise preference loss. The curriculum notes this reduces memory requirements to just 2 models (the current policy and a reference copy) and compresses the training pipeline into a single loop without the multi-stage RL overhead.

Training Pipeline Complexity

The practical implementation differs significantly in the number of training stages:

RLHF Pipeline:

1. Supervised Fine-Tuning (SFT)
2. Reward Model (RM) training
3. PPO-based reinforcement learning loop

DPO Pipeline:

1. Supervised Fine-Tuning (SFT)
2. Direct preference optimization (single-step update)

As noted in the repository documentation, RLHF requires "3-4 separate scripts" while DPO completes alignment in "1 training loop (vs 3 for RLHF)."

Computational Cost and Hyperparameters

RLHF incurs higher computational costs due to reward-model inference during training plus PPO gradient updates per minibatch. Key hyperparameters include the KL-penalty coefficient β (controlling distance to SFT), PPO clipping parameters, learning-rate schedules, and entropy bonuses. The curriculum emphasizes that "small-beta KL keeps us near SFT," but setting β=0 degenerates to pure RL without regularization.

DPO computes preference loss in a single forward pass without additional reward model inference, making it considerably faster and cheaper to train. The beta parameter in DPO similarly controls the KL-divergence penalty, but can be set to 0 for pure DPO without requiring a separate reward model value.

Code Implementation Comparison

The repository provides reference implementations showing the structural differences:

# RLHF approach - see phases/10-llms-from-scratch/07-rlhf/code/main.py

sft_model = load_model('sft')
reward_model = train_reward_model(preferences)          # learns a scalar reward

ppo = PPOAgent(policy=sft_model, reward_fn=reward_model, kl_beta=0.1)
ppo.train(dataset)                                      # several epochs of PPO updates

# DPO approach - see phases/10-llms-from-scratch/08-dpo/code/main.py

policy = load_model('sft')
dpo = DPOAgent(policy=policy, kl_beta=0.0)             # can also set a small beta

dpo.train(preference_pairs)                            # single-step preference loss

Both scripts conclude with policy.save('aligned_model'), but DPO completes alignment significantly faster due to the absence of the reward model training phase.

When to Use Each Technique

According to the curriculum documentation in phases/10-llms-from-scratch/07-rlhf/docs/en.md and phases/10-llms-from-scratch/08-dpo/docs/en.md:

Choose RLHF when a high-quality reward model already exists or when you need robust policy regularization through the multi-stage pipeline. RLHF provides a three-stage pipeline that mirrors many production systems and offers fine-grained control over the KL-divergence from the base model.

Choose DPO when the reward model is expensive or unreliable to train, or when you need quick alignment fine-tuning after RLHF plateaus. The curriculum explicitly recommends a hybrid approach: "Many production systems use both: RLHF first, DPO to refine."

Summary

• RLHF uses a separate reward model and PPO-based optimization across three training stages, requiring 3-4 models in memory and higher computational resources.
• DPO directly optimizes the policy on preference pairs in a single training loop with only 2 models, making it simpler, faster, and cheaper to implement.
• The primary trade-off involves RLHF's greater flexibility through explicit reward modeling versus DPO's efficiency and reduced memory footprint.
• Both methods use a KL-penalty coefficient (beta) to control divergence from the reference SFT model, but DPO achieves this without maintaining a separate reward model.
• Production implementations often combine both approaches, using RLHF for initial alignment and DPO for refinement.

Frequently Asked Questions

Can RLHF and DPO be used together in the same pipeline?

Yes, according to the ai-engineering-from-scratch curriculum, many production systems use RLHF first to establish a strong base policy, then apply DPO as a refinement step to improve alignment without the overhead of maintaining a reward model. This hybrid approach leverages the robustness of RLHF's three-stage pipeline while benefiting from DPO's computational efficiency for final tuning.

Why does RLHF require significantly more GPU memory than DPO?

RLHF requires maintaining 3-4 models simultaneously in memory: the SFT reference model, the trained reward model, the current policy being updated by PPO, and a reference copy for KL-regularization. In contrast, DPO only needs 2 models (the current policy and a reference copy), as implemented in the repository's training scripts at phases/10-llms-from-scratch/08-dpo/code/main.py.

Is DPO always faster to train than RLHF?

Yes, DPO typically reduces wall-clock training time by eliminating the separate reward model training stage and PPO optimization loops. While RLHF requires three distinct training phases (SFT → RM → PPO), DPO compresses alignment into a single optimization step that computes preference loss in one forward pass per preference pair, making it considerably faster in practice.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}