# Deep Reinforcement Learning with Policy Gradients and DQN: A Beginner's Guide

> Master deep reinforcement learning with Policy Gradients and DQN. Learn how agents optimize behaviors through trial and error for discrete and continuous control tasks. Start your AI journey today.

- Repository: [Microsoft/AI-For-Beginners](https://github.com/microsoft/AI-For-Beginners)
- Tags: tutorial
- Published: 2026-08-02

---

**Deep reinforcement learning with Policy Gradients and DQN enables agents to learn optimal behaviors through trial and error, with DQN approximating action-value functions for discrete control tasks and Policy Gradients directly optimizing stochastic policies for continuous action spaces.**

The microsoft/AI-For-Beginners curriculum introduces these foundational algorithms in lesson `lessons/6-Other/22-DeepRL/`, providing implementations in both TensorFlow 2.x and PyTorch. This guide walks through the architectural differences between value-based and policy-based methods, referencing the actual source code and Jupyter notebooks used in the repository.

## Overview of Deep RL Algorithms

The repository covers two complementary approaches to deep reinforcement learning. **Deep Q-Network (DQN)** approximates the Q-value function `Q(s, a; θ)` using a neural network, storing transitions in an experience replay buffer and sampling mini-batches to break temporal correlations. **Policy Gradient (PG)** methods, specifically REINFORCE, parameterize a stochastic policy `πθ(a|s)` and perform gradient ascent on expected returns without learning a value function.

Both implementations rely on **OpenAI Gym** environments and share common scaffolding for environment interaction, logging, and checkpointing. The dependencies are specified in [`environment.yml`](https://github.com/microsoft/AI-For-Beginners/blob/main/environment.yml), which includes `tensorflow`, `torch`, `gym`, `numpy`, and `matplotlib`.

## Deep Q-Network (DQN) Implementation

### Architecture and Q-Value Approximation

In `lessons/6-Other/22-DeepRL/DQN.ipynb`, the DQN uses a fully-connected neural network to predict expected returns for each action. The architecture consists of an input layer matching the state dimension, a hidden layer with 128 units and ReLU activation, and an output layer with units equal to the number of discrete actions.

### Experience Replay and Target Networks

The implementation employs two critical stabilization techniques. **Experience replay** stores transition tuples `(s, a, r, s', done)` in a deque buffer (`collections.deque(maxlen=10_000)`) and samples uniform mini-batches to decorrelate training data. A **target network**—a separate network `Q_target` with parameters `θ⁻`—provides stable temporal difference targets and periodically syncs with the online policy network.

### Training Loop and Loss Computation

The loss function implements the Bellman equation:

```

L(θ) = E[(r + γ(1-d) max_a' Q_target(s',a';θ⁻) - Q(s,a;θ))²]

```

Here is the minimal PyTorch implementation from the repository:

```python
import gym, torch, random, numpy as np
from collections import deque
import torch.nn as nn
import torch.optim as optim

env = gym.make('CartPole-v1')
n_states = env.observation_space.shape[0]
n_actions = env.action_space.n

class DQN(nn.Module):
    def __init__(self):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(n_states, 128),
            nn.ReLU(),
            nn.Linear(128, n_actions)
        )
    def forward(self, x):
        return self.net(x)

policy = DQN()
target = DQN()
target.load_state_dict(policy.state_dict())
optimizer = optim.Adam(policy.parameters(), lr=1e-3)
criterion = nn.MSELoss()
replay = deque(maxlen=10_000)

def select_action(state, ε=0.1):
    if random.random() < ε:
        return env.action_space.sample()
    with torch.no_grad():
        q_vals = policy(torch.FloatTensor(state))
        return q_vals.argmax().item()

def train_step(batch_size=64, γ=0.99):
    batch = random.sample(replay, batch_size)
    states, actions, rewards, next_states, dones = map(np.stack, zip(*batch))
    states = torch.FloatTensor(states)
    next_states = torch.FloatTensor(next_states)
    rewards = torch.FloatTensor(rewards)
    dones = torch.FloatTensor(dones)

    q_vals = policy(states).gather(1, torch.LongTensor(actions).unsqueeze(1)).squeeze()
    with torch.no_grad():
        q_next = target(next_states).max(1)[0]
    target_vals = rewards + γ * q_next * (1 - dones)

    loss = criterion(q_vals, target_vals)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

```

Key implementation details include ε-greedy action selection (`select_action`), the `gather` operation for selecting Q-values corresponding to taken actions, and the target value calculation that masks terminal states using `(1 - dones)`.

## Policy Gradient (REINFORCE) Implementation

### Stochastic Policy Optimization

The Policy Gradient notebook (`lessons/6-Other/22-DeepRL/PolicyGradient.ipynb`) implements REINFORCE, which parameterizes the policy as a probability distribution `πθ(a|s)` using a softmax output layer. Unlike DQN, this method directly optimizes the policy by following the gradient:

```

∇_θ J(θ) ≈ (1/N) Σ_i Σ_t ∇_θ log πθ(a_t^i|s_t^i) G_t^i

```

### Monte Carlo Return Estimation

The algorithm collects full trajectories `(s₀,a₀,r₀,…,s_T)` and computes discounted returns `G_t = Σ_{k=t}^T γ^{k-t} r_k` using a backward pass through the episode rewards. The loss function maximizes the log probability of actions weighted by their returns.

Here is the TensorFlow 2 implementation:

```python
import gym, tensorflow as tf, numpy as np

env = gym.make('CartPole-v1')
n_states = env.observation_space.shape[0]
n_actions = env.action_space.n

model = tf.keras.Sequential([
    tf.keras.layers.Dense(128, activation='relu', input_shape=(n_states,)),
    tf.keras.layers.Dense(n_actions, activation='softmax')
])
optimizer = tf.keras.optimizers.Adam(1e-3)

def run_episode():
    states, actions, rewards = [], [], []
    state = env.reset()
    done = False
    while not done:
        probs = model(np.expand_dims(state, 0)).numpy()[0]
        action = np.random.choice(n_actions, p=probs)
        next_state, reward, done, _ = env.step(action)
        states.append(state)
        actions.append(action)
        rewards.append(reward)
        state = next_state
    return np.array(states), np.array(actions), np.array(rewards)

def compute_returns(rewards, γ=0.99):
    returns = np.zeros_like(rewards, dtype=np.float32)
    running_add = 0
    for t in reversed(range(len(rewards))):
        running_add = rewards[t] + γ * running_add
        returns[t] = running_add
    return returns

@tf.function
def train_step(states, actions, returns):
    with tf.GradientTape() as tape:
        logits = model(states)
        action_masks = tf.one_hot(actions, n_actions)
        log_probs = tf.reduce_sum(action_masks * tf.math.log(logits + 1e-8), axis=1)
        loss = -tf.reduce_mean(log_probs * returns)
    grads = tape.gradient(loss, model.trainable_variables)
    optimizer.apply_gradients(zip(grads, model.trainable_variables))

```

Notice the use of `tf.one_hot` to mask the log probabilities of selected actions and the negative sign in the loss to perform gradient ascent on the expected return.

## Repository Structure and Key Files

The deep reinforcement learning materials are organized under `lessons/6-Other/22-DeepRL/`:

- **[`README.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/README.md)** — Contains lesson objectives, theoretical background, and links to OpenAI Gym documentation.
- **`DQN.ipynb`** — Complete TensorFlow 2 or PyTorch walkthrough of the DQN algorithm with experience replay visualization.
- **`PolicyGradient.ipynb`** — Step-by-step REINFORCE implementation including live plotting of episode returns.
- **[`environment.yml`](https://github.com/microsoft/AI-For-Beginners/blob/main/environment.yml)** — Conda specification listing `tensorflow`, `torch`, `gym`, and visualization dependencies required to run the notebooks.

The `examples/` directory contains auxiliary scripts like [`04-text-sentiment.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/04-text-sentiment.py) that demonstrate model loading patterns useful for comparing supervised learning with reinforcement learning approaches.

## Summary

- **DQN** uses a neural network to approximate Q-values with **experience replay** and **target networks** to stabilize learning in discrete action spaces like CartPole.
- **Policy Gradients** (REINFORCE) directly optimize a stochastic policy by following gradients of log-probabilities weighted by **Monte Carlo returns**, suitable for continuous or high-dimensional action spaces.
- Both implementations in the microsoft/AI-For-Beginners repository use **OpenAI Gym** environments and are available as interactive Jupyter notebooks with complete training loops.
- The [`environment.yml`](https://github.com/microsoft/AI-For-Beginners/blob/main/environment.yml) file provides the necessary dependencies for TensorFlow 2.x, PyTorch, and Gym to reproduce the results locally.

## Frequently Asked Questions

### What is the main difference between DQN and Policy Gradient methods?

DQN is a **value-based** method that learns to estimate the expected return for each state-action pair and derives the policy implicitly by selecting actions with maximum Q-values. Policy Gradient methods are **policy-based** and directly optimize the parameters of a stochastic policy, making them better suited for continuous action spaces or environments where the value function is difficult to approximate.

### Why does DQN require a separate target network?

The target network provides **stable learning targets** for the temporal difference update. Without it, the DQN would be chasing a moving target—updating its parameters to match Q-values that are themselves changing with every gradient step. By freezing the target network parameters `θ⁻` and syncing them periodically (e.g., every 1,000 steps), the algorithm reduces oscillations and divergence during training.

### When should I use Policy Gradients instead of DQN?

Use **Policy Gradients** when dealing with **continuous action spaces** (e.g., robotic control with joint torques) or high-dimensional discrete actions where enumerating all Q-values is impractical. DQN excels in discrete, low-dimensional action spaces like Atari games or CartPole, where computing the maximum Q-value over all actions is computationally feasible.

### What environment setup is required to run the Deep RL notebooks?

You need Python 3.x with OpenAI Gym (`gym`), a deep learning framework (TensorFlow 2.x or PyTorch), NumPy, and Matplotlib. The repository provides an [`environment.yml`](https://github.com/microsoft/AI-For-Beginners/blob/main/environment.yml) file for Conda users that installs all necessary dependencies, or you can install via pip using the provided [`requirements.txt`](https://github.com/microsoft/AI-For-Beginners/blob/main/requirements.txt).