# How Reinforcement Learning (PPO, DPO, RLHF) Aligns LLM Outputs: A Complete Technical Guide

> Master reinforcement learning for LLM alignment with PPO, DPO, and RLHF. Learn how these techniques optimize models against human preferences for superior text generation. A technical guide.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: deep-dive
- Published: 2026-06-24

---

**Reinforcement learning aligns large language models by optimizing them against human preference data—using PPO to maximize reward model scores with KL penalties, or DPO to skip the reward model entirely and optimize directly from log-probability ratios to a reference policy.**

Large language models require alignment to transform raw next-token prediction into helpful, truthful, and safe behavior. The `rohitg00/ai-engineering-from-scratch` repository implements three distinct reinforcement learning pipelines—RLHF with PPO, Direct Preference Optimization (DPO), and the underlying PPO algorithm—to demonstrate how human feedback shapes model outputs. These techniques move beyond supervised fine-tuning by incorporating preference data that encodes complex human values directly into the training objective.

## The RLHF Pipeline: Reward Models and PPO Optimization

The RLHF (Reinforcement Learning from Human Feedback) pipeline implemented in [`phases/10-llms-from-scratch/07-rlhf/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/10-llms-from-scratch/07-rlhf/docs/en.md) consists of three distinct stages that gradually transform a base model into an aligned policy.

### Stage 1: Supervised Fine-Tuning (SFT)

Before reinforcement learning begins, the base model undergoes **Supervised Fine-Tuning** on instruction-response pairs. This stage teaches the model the general format of following prompts but does not incorporate human preferences about quality or safety. The SFT checkpoint serves as the initialization for subsequent RL stages and as the frozen reference model that prevents the policy from drifting too far during optimization.

### Stage 2: Training the Reward Model

Human annotators provide pairwise comparisons between responses to the same prompt. The **reward model** is trained to predict these preferences using the Bradley-Terry pairwise loss:

$$
-\log \sigma(r_{+} - r_{-})
$$

Where $r_{+}$ and $r_{-}$ represent the reward scores for the preferred and rejected responses, respectively. The reward model architecture mirrors the base transformer but replaces the language modeling head with a single scalar output head:

```python

# Simplified reward-model forward pass from phases/10-llms-from-scratch/07-rlhf/docs/en.md

def forward(self, token_ids):
    x = self.embedding.forward(token_ids)
    for block in self.blocks:
        x = block.forward(x, mask)
    last_hidden = x[:, -1, :]
    reward = last_hidden @ self.reward_head
    return reward

```

This model takes a `(prompt, response)` pair and outputs a scalar score representing the human-judged quality of the response.

### Stage 3: PPO Training with KL Divergence

The final stage treats the language model as a **reinforcement learning policy** $\pi_{\theta}$ and optimizes it using **Proximal Policy Optimization (PPO)**. The objective balances reward maximization against a KL divergence penalty that keeps the policy close to the SFT reference model:

```python

# Core PPO step with KL computation from the RLHF implementation

def compute_kl_divergence(policy_logits, reference_logits):
    policy_probs = np.exp(policy_logits - policy_logits.max(axis=-1, keepdims=True))
    policy_probs /= policy_probs.sum(axis=-1, keepdims=True)
    ref_probs = np.exp(reference_logits - reference_logits.max(axis=-1, keepdims=True))
    ref_probs /= ref_probs.sum(axis=-1, keepdims=True)
    kl = np.sum(policy_probs * np.log(policy_probs / ref_probs), axis=-1)
    return kl.mean()

```

The PPO surrogate loss clips the policy update to prevent destructive large steps:

$$
L_{\text{PPO}} = -\min\big(\text{ratio} \cdot \text{adv},\; \text{clip}(\text{ratio}) \cdot \text{adv}\big)
$$

Where $\text{ratio} = \pi_{\text{new}}/\pi_{\text{old}}$ and $\text{adv} = r - b$ (reward minus baseline). The KL penalty term $\beta \cdot \text{KL}(\pi_{\theta} \| \pi_{\text{ref}})$ prevents **reward hacking**, where the model exploits loopholes in the reward model to achieve high scores without actually producing helpful content.

## Direct Preference Optimization (DPO): Reward-Free Alignment

**Direct Preference Optimization (DPO)** eliminates the separate reward model and PPO loop entirely. Implemented in [`phases/10-llms-from-scratch/08-dpo/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/10-llms-from-scratch/08-dpo/docs/en.md), DPO derives a closed-form optimal policy that depends only on the ratio between the current policy and the frozen reference SFT model.

### The Mathematical Foundation of DPO

The DPO objective starts from the optimal policy form:

$$
\pi^{*}(y|x) = \pi_{\text{ref}}(y|x) \frac{\exp(R(x,y)/\beta)}{Z(x)}
$$

Rearranging yields an **implicit reward** function that can be computed directly from log-probabilities:

$$
R(x,y) = \beta \left(\log \pi(y|x) - \log \pi_{\text{ref}}(y|x)\right) + \beta \log Z(x)
$$

Since the normalizing constant $Z(x)$ cancels when comparing preferred ($w$) and rejected ($l$) responses, the loss becomes:

$$
L_{\text{DPO}} = -\log \sigma\left(\beta \left(\log\frac{\pi(w|x)}{\pi_{\text{ref}}(w|x)} - \log\frac{\pi(l|x)}{\pi_{\text{ref}}(l|x)}\right)\right)
$$

### DPO Training Loop Implementation

The DPO training loop requires only four forward passes per preference pair—computing log-probabilities for both preferred and rejected responses under both the policy and reference models:

```python
def dpo_loss(policy_logprob_preferred, policy_logprob_rejected,
             ref_logprob_preferred, ref_logprob_rejected, beta=0.1):
    preferred_ratio = policy_logprob_preferred - ref_logprob_preferred
    rejected_ratio = policy_logprob_rejected - ref_logprob_rejected
    logit = beta * (preferred_ratio - rejected_ratio)
    loss = -np.log(sigmoid(logit) + 1e-8)
    return loss, {"logit": float(logit)}

```

Because DPO removes the reward model and PPO machinery, it requires only **two models in memory** (policy and reference) and executes a **single supervised training loop**, reducing compute requirements to approximately one-third of the full RLHF pipeline.

## PPO vs. DPO: Architectural and Computational Trade-offs

The choice between RLHF with PPO and DPO involves distinct engineering trade-offs:

- **Model Requirements**: RLHF requires maintaining three to four models simultaneously (policy, reference, reward model, and optional value model), while DPO requires only two models (policy and reference).
- **Training Complexity**: RLHF involves three distinct training loops (SFT → reward model → PPO), whereas DPO compresses this into two loops (SFT → DPO).
- **Stability**: PPO is sensitive to hyperparameters including the KL coefficient $\beta$, clip-ratio, and learning rate. DPO eliminates clipping and advantage estimation, resulting in significantly more stable training dynamics.
- **Compute Cost**: RLHF requires approximately three times the computational resources of DPO due to multiple forward and backward passes through the reward model and value network.

## The Critical Role of the KL Penalty (β)

Both PPO and DPO rely on a scalar **β** (beta) to control the deviation from the reference SFT model:

- **Small β values** allow the policy to move rapidly toward high-reward regions but increase the risk of **reward hacking**—exploiting weaknesses in the reward signal to produce degenerate outputs.
- **Large β values** constrain the policy to stay close to the SFT distribution, ensuring stability at the cost of slower alignment progress.

In PPO, β multiplies the explicit KL divergence term. In DPO, β scales the log-probability ratio, serving the same function as a "leash" on the policy. The repository's DPO lesson includes a "Beta Sensitivity Analysis" demonstrating this classic stability-vs-performance trade-off.

## Summary

- **RLHF with PPO** trains a separate reward model on human preferences, then uses PPO to maximize expected reward while penalizing KL divergence from the SFT checkpoint.
- **DPO** eliminates the reward model by deriving the optimal policy directly from log-probability ratios to a reference model, executing alignment in a single supervised loop.
- Both methods require a frozen **reference model** (the SFT checkpoint) and a **KL coefficient β** to prevent the policy from drifting into degenerate behaviors.
- Implementation details for both pipelines are available in [`phases/10-llms-from-scratch/07-rlhf/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/10-llms-from-scratch/07-rlhf/docs/en.md) (RLHF) and [`phases/10-llms-from-scratch/08-dpo/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/10-llms-from-scratch/08-dpo/docs/en.md) (DPO), using the MiniGPT core from [`phases/10-llms-from-scratch/04-pre-training-mini-gpt/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/10-llms-from-scratch/04-pre-training-mini-gpt/code/main.py).

## Frequently Asked Questions

### What is the difference between PPO and DPO in LLM alignment?

**PPO** is a generic reinforcement learning algorithm used within RLHF to maximize scores from a separately trained reward model, requiring clipping and advantage estimation. **DPO** (Direct Preference Optimization) is a specialized alignment method that eliminates the reward model entirely by optimizing directly against preference pairs using only log-probability ratios between the policy and a reference model, resulting in a simpler supervised learning objective.

### Why is a reward model necessary for RLHF but not for DPO?

RLHF requires an explicit reward model because PPO needs a scalar reward signal to optimize. The reward model is trained to predict human preferences using the Bradley-Terry loss. DPO shows that the optimal policy under the RLHF objective has a closed form where the reward can be expressed implicitly as the log-ratio between the policy and reference model, eliminating the need for a separate neural network to estimate rewards.

### How does the KL penalty prevent reward hacking in PPO?

The KL penalty $\beta \cdot \text{KL}(\pi_{\theta} \| \pi_{\text{ref}})$ penalizes the policy for deviating too far from the SFT reference distribution. Without this constraint, the PPO agent could exploit loopholes in the reward model—such as generating high-scoring but nonsensical repetitive patterns—to maximize reward without producing genuinely helpful outputs. The KL term forces the policy to remain close to the well-behaved SFT distribution while still improving on the reward metric.

### When should I choose RLHF over DPO for model alignment?

Choose **RLHF** when working with large-scale multi-objective alignment where a separate reward model captures nuanced criteria (helpfulness, truthfulness, safety) that are difficult to encode directly. Choose **DPO** for small-to-medium datasets, limited compute budgets, or when rapidly fine-tuning an already-aligned model, as it requires approximately one-third the compute and offers greater training stability without complex hyperparameter tuning of PPO-specific parameters like clip-ratio and advantage estimation.