Deep Reinforcement Learning with Policy Gradients and DQN: A Beginner's Guide

Deep reinforcement learning with Policy Gradients and DQN enables agents to learn optimal behaviors through trial and error, with DQN approximating action-value functions for discrete control tasks and Policy Gradients directly optimizing stochastic policies for continuous action spaces.

The microsoft/AI-For-Beginners curriculum introduces these foundational algorithms in lesson lessons/6-Other/22-DeepRL/, providing implementations in both TensorFlow 2.x and PyTorch. This guide walks through the architectural differences between value-based and policy-based methods, referencing the actual source code and Jupyter notebooks used in the repository.

Overview of Deep RL Algorithms

The repository covers two complementary approaches to deep reinforcement learning. Deep Q-Network (DQN) approximates the Q-value function Q(s, a; θ) using a neural network, storing transitions in an experience replay buffer and sampling mini-batches to break temporal correlations. Policy Gradient (PG) methods, specifically REINFORCE, parameterize a stochastic policy πθ(a|s) and perform gradient ascent on expected returns without learning a value function.

Both implementations rely on OpenAI Gym environments and share common scaffolding for environment interaction, logging, and checkpointing. The dependencies are specified in environment.yml, which includes tensorflow, torch, gym, numpy, and matplotlib.

Deep Q-Network (DQN) Implementation

Architecture and Q-Value Approximation

In lessons/6-Other/22-DeepRL/DQN.ipynb, the DQN uses a fully-connected neural network to predict expected returns for each action. The architecture consists of an input layer matching the state dimension, a hidden layer with 128 units and ReLU activation, and an output layer with units equal to the number of discrete actions.

Experience Replay and Target Networks

The implementation employs two critical stabilization techniques. Experience replay stores transition tuples (s, a, r, s', done) in a deque buffer (collections.deque(maxlen=10_000)) and samples uniform mini-batches to decorrelate training data. A target network—a separate network Q_target with parameters θ⁻—provides stable temporal difference targets and periodically syncs with the online policy network.

Training Loop and Loss Computation

The loss function implements the Bellman equation:

L(θ) = E[(r + γ(1-d) max_a' Q_target(s',a';θ⁻) - Q(s,a;θ))²]

Here is the minimal PyTorch implementation from the repository:

import gym, torch, random, numpy as np
from collections import deque
import torch.nn as nn
import torch.optim as optim

env = gym.make('CartPole-v1')
n_states = env.observation_space.shape[0]
n_actions = env.action_space.n

class DQN(nn.Module):
    def __init__(self):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(n_states, 128),
            nn.ReLU(),
            nn.Linear(128, n_actions)
        )
    def forward(self, x):
        return self.net(x)

policy = DQN()
target = DQN()
target.load_state_dict(policy.state_dict())
optimizer = optim.Adam(policy.parameters(), lr=1e-3)
criterion = nn.MSELoss()
replay = deque(maxlen=10_000)

def select_action(state, ε=0.1):
    if random.random() < ε:
        return env.action_space.sample()
    with torch.no_grad():
        q_vals = policy(torch.FloatTensor(state))
        return q_vals.argmax().item()

def train_step(batch_size=64, γ=0.99):
    batch = random.sample(replay, batch_size)
    states, actions, rewards, next_states, dones = map(np.stack, zip(*batch))
    states = torch.FloatTensor(states)
    next_states = torch.FloatTensor(next_states)
    rewards = torch.FloatTensor(rewards)
    dones = torch.FloatTensor(dones)

    q_vals = policy(states).gather(1, torch.LongTensor(actions).unsqueeze(1)).squeeze()
    with torch.no_grad():
        q_next = target(next_states).max(1)[0]
    target_vals = rewards + γ * q_next * (1 - dones)

    loss = criterion(q_vals, target_vals)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

Key implementation details include ε-greedy action selection (select_action), the gather operation for selecting Q-values corresponding to taken actions, and the target value calculation that masks terminal states using (1 - dones).

Policy Gradient (REINFORCE) Implementation

Stochastic Policy Optimization

The Policy Gradient notebook (lessons/6-Other/22-DeepRL/PolicyGradient.ipynb) implements REINFORCE, which parameterizes the policy as a probability distribution πθ(a|s) using a softmax output layer. Unlike DQN, this method directly optimizes the policy by following the gradient:

∇_θ J(θ) ≈ (1/N) Σ_i Σ_t ∇_θ log πθ(a_t^i|s_t^i) G_t^i

Monte Carlo Return Estimation

The algorithm collects full trajectories (s₀,a₀,r₀,…,s_T) and computes discounted returns G_t = Σ_{k=t}^T γ^{k-t} r_k using a backward pass through the episode rewards. The loss function maximizes the log probability of actions weighted by their returns.

Here is the TensorFlow 2 implementation:

import gym, tensorflow as tf, numpy as np

env = gym.make('CartPole-v1')
n_states = env.observation_space.shape[0]
n_actions = env.action_space.n

model = tf.keras.Sequential([
    tf.keras.layers.Dense(128, activation='relu', input_shape=(n_states,)),
    tf.keras.layers.Dense(n_actions, activation='softmax')
])
optimizer = tf.keras.optimizers.Adam(1e-3)

def run_episode():
    states, actions, rewards = [], [], []
    state = env.reset()
    done = False
    while not done:
        probs = model(np.expand_dims(state, 0)).numpy()[0]
        action = np.random.choice(n_actions, p=probs)
        next_state, reward, done, _ = env.step(action)
        states.append(state)
        actions.append(action)
        rewards.append(reward)
        state = next_state
    return np.array(states), np.array(actions), np.array(rewards)

def compute_returns(rewards, γ=0.99):
    returns = np.zeros_like(rewards, dtype=np.float32)
    running_add = 0
    for t in reversed(range(len(rewards))):
        running_add = rewards[t] + γ * running_add
        returns[t] = running_add
    return returns

@tf.function
def train_step(states, actions, returns):
    with tf.GradientTape() as tape:
        logits = model(states)
        action_masks = tf.one_hot(actions, n_actions)
        log_probs = tf.reduce_sum(action_masks * tf.math.log(logits + 1e-8), axis=1)
        loss = -tf.reduce_mean(log_probs * returns)
    grads = tape.gradient(loss, model.trainable_variables)
    optimizer.apply_gradients(zip(grads, model.trainable_variables))

Notice the use of tf.one_hot to mask the log probabilities of selected actions and the negative sign in the loss to perform gradient ascent on the expected return.

Repository Structure and Key Files

The deep reinforcement learning materials are organized under lessons/6-Other/22-DeepRL/:

• README.md — Contains lesson objectives, theoretical background, and links to OpenAI Gym documentation.
• DQN.ipynb — Complete TensorFlow 2 or PyTorch walkthrough of the DQN algorithm with experience replay visualization.
• PolicyGradient.ipynb — Step-by-step REINFORCE implementation including live plotting of episode returns.
• environment.yml — Conda specification listing tensorflow, torch, gym, and visualization dependencies required to run the notebooks.

The examples/ directory contains auxiliary scripts like 04-text-sentiment.py that demonstrate model loading patterns useful for comparing supervised learning with reinforcement learning approaches.

Summary

• DQN uses a neural network to approximate Q-values with experience replay and target networks to stabilize learning in discrete action spaces like CartPole.
• Policy Gradients (REINFORCE) directly optimize a stochastic policy by following gradients of log-probabilities weighted by Monte Carlo returns, suitable for continuous or high-dimensional action spaces.
• Both implementations in the microsoft/AI-For-Beginners repository use OpenAI Gym environments and are available as interactive Jupyter notebooks with complete training loops.
• The environment.yml file provides the necessary dependencies for TensorFlow 2.x, PyTorch, and Gym to reproduce the results locally.

Frequently Asked Questions

What is the main difference between DQN and Policy Gradient methods?

DQN is a value-based method that learns to estimate the expected return for each state-action pair and derives the policy implicitly by selecting actions with maximum Q-values. Policy Gradient methods are policy-based and directly optimize the parameters of a stochastic policy, making them better suited for continuous action spaces or environments where the value function is difficult to approximate.

Why does DQN require a separate target network?

The target network provides stable learning targets for the temporal difference update. Without it, the DQN would be chasing a moving target—updating its parameters to match Q-values that are themselves changing with every gradient step. By freezing the target network parameters θ⁻ and syncing them periodically (e.g., every 1,000 steps), the algorithm reduces oscillations and divergence during training.

When should I use Policy Gradients instead of DQN?

Use Policy Gradients when dealing with continuous action spaces (e.g., robotic control with joint torques) or high-dimensional discrete actions where enumerating all Q-values is impractical. DQN excels in discrete, low-dimensional action spaces like Atari games or CartPole, where computing the maximum Q-value over all actions is computationally feasible.

What environment setup is required to run the Deep RL notebooks?

You need Python 3.x with OpenAI Gym (gym), a deep learning framework (TensorFlow 2.x or PyTorch), NumPy, and Matplotlib. The repository provides an environment.yml file for Conda users that installs all necessary dependencies, or you can install via pip using the provided requirements.txt.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}