# How to Implement Deep Reinforcement Learning with Policy Gradients and Q-Learning

> Learn to implement deep reinforcement learning using policy gradients with TensorFlow or Q-learning with PyTorch. Explore runnable notebooks in the Microsoft AI-For-Beginners curriculum.

- Repository: [Microsoft/AI-For-Beginners](https://github.com/microsoft/AI-For-Beginners)
- Tags: how-to-guide
- Published: 2026-08-03

---

**You can implement deep reinforcement learning using policy gradients (actor-critic) with TensorFlow or Q-learning with PyTorch by training neural networks on the CartPole environment—both methods are fully documented in the Microsoft AI-For-Beginners curriculum with runnable notebooks.**

Deep reinforcement learning combines neural networks with RL algorithms to solve complex control problems. The **AI-For-Beginners** repository provides a complete educational module at `lessons/6-Other/22-DeepRL/` that demonstrates how to implement deep reinforcement learning with policy gradients and Q-learning using the classic CartPole-v1 environment. This guide walks through both architectures, their training loops, and how to run the official notebooks.

## Policy Gradients: Actor-Critic Implementation in TensorFlow

The **policy-gradient** approach directly learns a stochastic policy π(s) that outputs action probabilities. The TensorFlow notebook `CartPole-RL-TF.ipynb` implements this using the **actor-critic** architecture, where two networks cooperate to stabilize training.

### Network Architecture

- **Actor network**: Takes the 4-dimensional state vector *(cart position, cart velocity, pole angle, pole velocity)* and outputs a softmax distribution over two actions (left/right).
- **Critic network**: Shares the same hidden layers but emits a single scalar value estimating the expected return from the current state.

```python
import gym, numpy as np, tensorflow as tf

env = gym.make("CartPole-v1")
num_states = env.observation_space.shape[0]
num_actions = env.action_space.n

# Actor + Critic model

inputs = tf.keras.layers.Input(shape=(num_states,))
hidden = tf.keras.layers.Dense(24, activation='relu')(inputs)
policy = tf.keras.layers.Dense(num_actions, activation='softmax')(hidden)
value  = tf.keras.layers.Dense(1, activation='linear')(hidden)
model = tf.keras.Model(inputs, [policy, value])
optimizer = tf.keras.optimizers.Adam(learning_rate=0.01)

```

### Discounted Rewards Calculation

The policy gradient requires **discounted returns** G_t to weight actions by their long-term consequences. The `discounted_rewards` function in the notebook computes:

G_t = Σ_{k=0}^{∞} γ^k r_{t+k} where γ ≈ 0.99

```python
def discounted_rewards(rews, gamma=0.99):
    ret = np.zeros_like(rews, dtype=np.float32)
    running_add = 0
    for t in reversed(range(len(rews))):
        running_add = rews[t] + gamma * running_add
        ret[t] = running_add
    return (ret - ret.mean()) / (ret.std() + 1e-8)

```

### Training Loop with Policy Gradient Loss

The core training loop collects episode trajectories and computes the combined actor-critic loss:

```python

# Training loop

for ep in range(300):
    state = env.reset()
    states, actions, probs, rewards = [], [], [], []
    done = False
    while not done:
        prob, _ = model(np.expand_dims(state, 0))
        prob = prob.numpy()[0]
        action = np.random.choice(num_actions, p=prob)
        next_state, reward, done, _ = env.step(action)

        states.append(state)
        actions.append(action)
        probs.append(prob)
        rewards.append(reward)

        state = next_state

    G = discounted_rewards(rewards)
    with tf.GradientTape() as tape:
        pi, V = model(np.array(states))
        # Actor loss (policy gradient): -log π(a|s) * G

        action_one_hot = tf.one_hot(actions, num_actions)
        log_pi = tf.math.log(tf.reduce_sum(pi * action_one_hot, axis=1) + 1e-8)
        actor_loss = -tf.reduce_mean(log_pi * G)
        # Critic loss (value regression): MSE between predicted and actual returns

        critic_loss = tf.reduce_mean(tf.square(V[:,0] - G))
        loss = actor_loss + critic_loss
    grads = tape.gradient(loss, model.trainable_variables)
    optimizer.apply_gradients(zip(grads, model.trainable_variables))

```

The **actor loss** implements the policy gradient theorem: ∇_θ J ≈ Σ_t ∇_θ log π_θ(a_t|s_t) * G_t. The **critic loss** trains the value estimator to predict returns accurately, reducing variance in the policy gradient.

## Q-Learning: Deep Q-Network Implementation in PyTorch

The **Q-learning** approach learns an action-value function Q(s,a) rather than a policy directly. The optimal policy selects actions with maximum Q-value: a* = argmax_a Q(s,a). The PyTorch notebook `CartPole-RL-PyTorch.ipynb` implements this using the Bellman equation for target updates.

### Q-Network Architecture

```python
import gym, torch, torch.nn as nn, torch.optim as optim
import random, numpy as np

env = gym.make('CartPole-v1')
state_dim = env.observation_space.shape[0]
n_actions = env.action_space.n

class QNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc = nn.Sequential(
            nn.Linear(state_dim, 64), nn.ReLU(),
            nn.Linear(64, n_actions)
        )
    def forward(self, x):
        return self.fc(x)

qnet = QNet()
optimizer = optim.Adam(qnet.parameters(), lr=0.001)
criterion = nn.MSELoss()
gamma = 0.99

```

### ε-Greedy Exploration and Experience Replay

Q-learning requires **exploration** to discover good actions. The implementation uses ε-greedy action selection with decaying ε, and a simple replay buffer to break correlation between consecutive transitions:

```python
epsilon = 1.0
memory = []          # simple list replay buffer

BATCH = 64
MAX_MEM = 5000

def epsilon_greedy(state):
    if random.random() < epsilon:
        return env.action_space.sample()
    q_vals = qnet(torch.from_numpy(state).float().unsqueeze(0))
    return torch.argmax(q_vals).item()

```

### Training Loop with Bellman Target Updates

The target Q-value follows the Bellman equation: Q(s,a) ← r + γ * max_a' Q(s',a'):

```python
for episode in range(500):
    state = env.reset()
    done = False
    while not done:
        action = epsilon_greedy(state)
        nxt, reward, done, _ = env.step(action)
        memory.append((state, action, reward, nxt, done))
        if len(memory) > MAX_MEM: memory.pop(0)

        # Sample minibatch for experience replay

        if len(memory) >= BATCH:
            batch = random.sample(memory, BATCH)
            s_batch, a_batch, r_batch, ns_batch, d_batch = zip(*batch)
            s_batch = torch.tensor(s_batch, dtype=torch.float32)
            a_batch = torch.tensor(a_batch, dtype=torch.int64).unsqueeze(1)
            r_batch = torch.tensor(r_batch, dtype=torch.float32).unsqueeze(1)
            ns_batch = torch.tensor(ns_batch, dtype=torch.float32)
            d_batch = torch.tensor(d_batch, dtype=torch.float32).unsqueeze(1)

            # Current Q-values for selected actions

            q_vals = qnet(s_batch).gather(1, a_batch)
            # Target Q-values: r + γ * max_a' Q(s',a')

            with torch.no_grad():
                next_q = qnet(ns_batch).max(1)[0].unsqueeze(1)
                target = r_batch + gamma * next_q * (1 - d_batch)
            loss = criterion(q_vals, target)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
        state = nxt
    epsilon = max(0.01, epsilon * 0.995)   # decay exploration

```

## Policy Gradients vs. Q-Learning: When to Use Each

| Aspect | Policy Gradients (Actor-Critic) | Q-Learning (Deep Q-Network) |
|--------|--------------------------------|-----------------------------|
| **What is learned** | Stochastic policy π(a\|s) directly | Action-value function Q(s,a) |
| **Policy type** | Probabilistic, naturally explores | Deterministic (after argmax), requires ε-greedy |
| **Best for** | Continuous action spaces, stochastic domains | Discrete action spaces, sample-efficient learning |
| **Variance** | Higher variance, needs baseline (critic) | Lower variance, more stable with replay |
| **Convergence** | Slower but handles complex policies | Faster on simple tasks like CartPole |

In the AI-For-Beginners curriculum, **policy gradients** are implemented with TensorFlow following the official [TensorFlow actor-critic tutorial](https://www.tensorflow.org/tutorials/reinforcement_learning/actor_critic), while **Q-learning** follows the [PyTorch DQN tutorial](https://pytorch.org/tutorials/intermediate/reinforcement_q_learning.html).

## Key Files and Next Steps

| File | Purpose |
|------|---------|
| [`lessons/6-Other/22-DeepRL/README.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/6-Other/22-DeepRL/README.md) | Environment setup and learning objectives |
| `lessons/6-Other/22-DeepRL/CartPole-RL-TF.ipynb` | Complete TensorFlow actor-critic implementation |
| `lessons/6-Other/22-DeepRL/CartPole-RL-PyTorch.ipynb` | Complete PyTorch Q-learning implementation |
| [`lessons/6-Other/22-DeepRL/lab/README.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/6-Other/22-DeepRL/lab/README.md) | Assignment to apply Q-learning to Mountain Car |

To extend your learning, the lab assignment challenges you to implement deep reinforcement learning with policy gradients and Q-learning on the **Mountain Car** environment—a more difficult control problem requiring sustained exploration.

## Summary

- **Policy gradients** directly optimize action probabilities using the actor-critic architecture, combining policy gradient loss with value regression for stable training.
- **Q-learning** estimates action-values through the Bellman equation and uses ε-greedy exploration with experience replay for sample efficiency.
- Both methods are fully implemented in the Microsoft AI-For-Beginners repository with runnable notebooks for CartPole-v1.
- The TensorFlow notebook (`CartPole-RL-TF.ipynb`) demonstrates actor-critic, while the PyTorch notebook (`CartPole-RL-PyTorch.ipynb`) covers Q-learning.

## Frequently Asked Questions

### What is the difference between actor-critic and Q-learning?

**Actor-critic is a policy-gradient method that learns both a policy (actor) and a value function (critic) simultaneously**, using the critic to reduce variance in policy updates. Q-learning is a value-based method that learns action-values directly and derives the policy implicitly through argmax selection. Actor-critic handles continuous actions naturally; Q-learning works best with discrete actions.

### Why does the policy gradient use discounted rewards?

Discounted rewards **weight immediate and future rewards by their temporal distance**, preventing infinite returns in continuing environments and matching the mathematical objective of maximizing expected cumulative return. The normalization step in `discounted_rewards` further stabilizes training by controlling variance across episodes.

### How does experience replay improve Q-learning?

Experience replay **breaks the correlation between consecutive transitions** by sampling random mini-batches from stored experiences, rather than learning from correlated sequential data. This improves sample efficiency and learning stability, as implemented in the PyTorch notebook's simple list-based replay buffer.