How the Curriculum Teaches Reinforcement Learning from MDPs to PPO and RLHF

The AI-Engineering-From-Scratch curriculum constructs a complete reinforcement learning stack by progressively layering algorithms from Markov Decision Process (MDP) foundations through Proximal Policy Optimization (PPO) to Reinforcement Learning from Human Feedback (RLHF), re-using core mathematical formalisms at every stage.

The rohitg00/ai-engineering-from-scratch repository delivers a comprehensive reinforcement learning from MDPs to PPO and RLHF curriculum that treats the subject as a single architectural narrative rather than isolated techniques. Each lesson builds upon the exact same objects—states, actions, rewards, and advantages—allowing learners to trace any advanced algorithm back to the original MDP definition.

Foundations in Markov Decision Processes

The journey begins in phases/09-reinforcement-learning/01-mdps-states-actions-rewards/docs/en.md with the formal MDP expressed as the tuple (S, A, P, R, γ). This lesson establishes that every reinforcement learning problem can be defined by states, actions, transition probabilities, a reward function, and a discount factor.

Lessons here derive the Bellman equations that underpin all later methods. Students implement a minimal GridWorld to see the recursion in action:

GRID = 4
TERMINAL = (3, 3)
ACTIONS = {"up": (-1, 0), "down": (1, 0), "left": (0, -1), "right": (0, 1)}

def step(state, action):
    if state == TERMINAL:
        return state, 0.0, True
    dr, dc = ACTIONS[action]
    r, c = state
    nr = min(max(r + dr, 0), GRID - 1)
    nc = min(max(c + dc, 0), GRID - 1)
    return (nr, nc), -1.0, (nr, nc) == TERMINAL

The full implementation is available in the lesson’s code/main.py, demonstrating how the Bellman expectation backup computes value functions for small, discrete state spaces.

From Value Iteration to Deep Q-Networks

Once the MDP formalism is solid, the curriculum introduces three progressively more sophisticated methods for estimating value functions without assuming knowledge of the environment model.

Dynamic Programming and Monte Carlo Methods

phases/09-reinforcement-learning/02-dynamic-programming/docs/en.md shows how to apply the Bellman equations exactly when the model is known (policy iteration and value iteration). Following this, phases/09-reinforcement-learning/03-monte-carlo-methods/docs/en.md teaches sampling trajectories to estimate returns, introducing the concept of returns that re-appears in policy-gradient methods later.

Temporal-Difference Learning (Q-Learning)

phases/09-reinforcement-learning/04-q-learning-sarsa/docs/en.md introduces bootstrapping, which enables learning from incomplete trajectories. The Q-learning update directly implements the Bellman optimality operator:

1. Sample a transition (s, a, r, s').
2. Compute the temporal-difference target r + γ·maxₐ' Q(s', a').
3. Update Q(s, a) toward that target.

This bridges the gap between Monte Carlo sampling and dynamic programming, preparing students for function approximation.

Scaling with Deep Q-Networks (DQN)

phases/09-reinforcement-learning/05-dqn/docs/en.md extends tabular Q-learning with a neural network to handle high-dimensional state spaces such as Atari pixels. The lesson demonstrates how the same Bellman update is applied to a function approximator, using a replay buffer and target networks to stabilize training.

Policy Optimization and Proximal Policy Optimization

The curriculum then shifts from value-based learning to direct policy optimization, culminating in the "default" algorithm for modern reinforcement learning.

Policy Gradients with REINFORCE

phases/09-reinforcement-learning/06-policy-gradients-reinforce/docs/en.md introduces the policy-gradient theorem. The REINFORCE estimator uses the same advantage-based return formula introduced for Monte Carlo methods, but now differentiates through the policy parameters to perform gradient ascent on expected return.

Actor-Critic Architecture

phases/09-reinforcement-learning/07-actor-critic-a2c-a3c/docs/en.md combines the policy gradient with a learned value function. This Actor-Critic structure re-uses the Bellman-derived advantage estimator (A = R - V(s)) to reduce variance while maintaining the bias characteristics of bootstrapping.

Proximal Policy Optimization (PPO) Implementation

phases/09-reinforcement-learning/08-ppo/docs/en.md presents Proximal Policy Optimization, the industry-standard policy-gradient algorithm. PPO retains the actor-critic structure but adds a clipped importance ratio that permits multiple epochs over the same rollout while keeping the update approximately on-policy.

The core PPO surrogate objective shown in the lesson code looks like this:

ratio = exp(logp_new - logp_old)                # importance ratio

surrogate = min(ratio * adv,
                clip(ratio, 1 - EPS, 1 + EPS) * adv)
loss = -surrogate + c_v * (V_pred - V_target)**2 - c_e * entropy
grad = torch.autograd.grad(loss, policy_params)

This clipping mechanism solves the instability problem described in the Actor-Critic lesson by preventing the policy from changing too drastically in a single update. The full training loop, including KL divergence diagnostics and clip-fraction logging, is implemented in phases/09-reinforcement-learning/08-ppo/code/main.py.

Alignment Through Reinforcement Learning from Human Feedback

The final stage closes the loop by embedding a learned reward signal into theRL pipeline.

Training the Reward Model

phases/09-reinforcement-learning/09-reward-modeling-rlhf/docs/en.md explains how a reward model is trained on pairwise human preferences using the Bradley-Terry model. The curriculum provides a minimal linear implementation:

# Train a Bradley-Terry reward model

def rm_update(w, x, y_pos, y_neg, lr):
    r_pos = dot(w, bag(y_pos))
    r_neg = dot(w, bag(y_neg))
    p = sigmoid(r_pos - r_neg)
    w += lr * (1 - p) * (bag(y_pos) - bag(y_neg))

This reward model effectively becomes the environment for the final PPO stage.

The SFT-RM-PPO Pipeline

The RLHF lesson presents the complete three-stage pipeline:

• Supervised Fine-Tuning (SFT) on high-quality demonstrations.
• Reward Model (RM) training on preference data.
• PPO against the learned reward model with a KL penalty to prevent deviation from the reference policy.

Crucially, this stage explicitly re-uses the exact PPO loss, the KL-penalty variant, and the advantage computation defined in earlier lessons, demonstrating how the entire curriculum converges on practical alignment. The complete toy implementation resides in phases/09-reinforcement-learning/09-reward-modeling-rlhf/code/main.py.

Summary

• The curriculum expresses every RL problem through the MDP formalism (S, A, P, R, γ) introduced in the first lesson, ensuring mathematical consistency throughout.
• Value-based methods (Dynamic Programming → Monte Carlo → TD → DQN) teach progressively more scalable ways to approximate the Bellman equations.
• Policy-based methods (REINFORCE → Actor-Critic → PPO) shift to direct optimization while re-using the advantage estimators and return calculations from earlier modules.
• PPO serves as the capstone policy-gradient algorithm, with its clipped objective implemented in phases/09-reinforcement-learning/08-ppo/code/main.py.
• RLHF completes the stack by substituting a learned reward model for the environment reward, applying the exact same PPO machinery to align large language models.

Frequently Asked Questions

What is the Bellman equation's role in this curriculum?

The Bellman equation acts as the connective tissue across all nine lessons. It first appears in the MDP definition lesson (phases/09-reinforcement-learning/01-mdps-states-actions-rewards/docs/en.md) as the recursive definition of value, then re-emerges in Dynamic Programming, Q-Learning, and Actor-Critic methods as the foundation for bootstrapping and advantage estimation.

How does PPO differ from standard Actor-Critic methods?

While standard Actor-Critic methods perform a single gradient step per batch of experience, PPO permits multiple epochs over the same rollout by clipping the probability ratio between the new and old policies. This clipping, shown in phases/09-reinforcement-learning/08-ppo/docs/en.md, prevents destabilizingly large updates that would push the policy off the data manifold.

Why does the curriculum teach RLHF as the final RL stage?

RLHF is positioned last because it requires all preceding components: a policy architecture (from REINFORCE/Actor-Critic), stable optimization (PPO), and a value-based understanding of reward signals. The lesson in phases/09-reinforcement-learning/09-reward-modeling-rlhf/docs/en.md demonstrates how the reward model replaces the environment R function, completing the conceptual arc from MDP foundations to modern alignment.

Where are the runnable code examples located?

Minimal implementations are distributed throughout the module. The GridWorld MDP solver lives in phases/09-reinforcement-learning/01-mdps-states-actions-rewards/code/main.py, the PPO training loop is in phases/09-reinforcement-learning/08-ppo/code/main.py, and the toy RLHF pipeline with the Bradley-Terry reward model is in phases/09-reinforcement-learning/09-reward-modeling-rlhf/code/main.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}