How to Implement Deep Reinforcement Learning with Policy Gradients and Q-Learning

You can implement deep reinforcement learning using policy gradients (actor-critic) with TensorFlow or Q-learning with PyTorch by training neural networks on the CartPole environment—both methods are fully documented in the Microsoft AI-For-Beginners curriculum with runnable notebooks.

Deep reinforcement learning combines neural networks with RL algorithms to solve complex control problems. The AI-For-Beginners repository provides a complete educational module at lessons/6-Other/22-DeepRL/ that demonstrates how to implement deep reinforcement learning with policy gradients and Q-learning using the classic CartPole-v1 environment. This guide walks through both architectures, their training loops, and how to run the official notebooks.

Policy Gradients: Actor-Critic Implementation in TensorFlow

The policy-gradient approach directly learns a stochastic policy π(s) that outputs action probabilities. The TensorFlow notebook CartPole-RL-TF.ipynb implements this using the actor-critic architecture, where two networks cooperate to stabilize training.

Network Architecture

• Actor network: Takes the 4-dimensional state vector (cart position, cart velocity, pole angle, pole velocity) and outputs a softmax distribution over two actions (left/right).
• Critic network: Shares the same hidden layers but emits a single scalar value estimating the expected return from the current state.

import gym, numpy as np, tensorflow as tf

env = gym.make("CartPole-v1")
num_states = env.observation_space.shape[0]
num_actions = env.action_space.n

# Actor + Critic model

inputs = tf.keras.layers.Input(shape=(num_states,))
hidden = tf.keras.layers.Dense(24, activation='relu')(inputs)
policy = tf.keras.layers.Dense(num_actions, activation='softmax')(hidden)
value  = tf.keras.layers.Dense(1, activation='linear')(hidden)
model = tf.keras.Model(inputs, [policy, value])
optimizer = tf.keras.optimizers.Adam(learning_rate=0.01)

Discounted Rewards Calculation

The policy gradient requires discounted returns G_t to weight actions by their long-term consequences. The discounted_rewards function in the notebook computes:

G_t = Σ_{k=0}^{∞} γ^k r_{t+k} where γ ≈ 0.99

def discounted_rewards(rews, gamma=0.99):
    ret = np.zeros_like(rews, dtype=np.float32)
    running_add = 0
    for t in reversed(range(len(rews))):
        running_add = rews[t] + gamma * running_add
        ret[t] = running_add
    return (ret - ret.mean()) / (ret.std() + 1e-8)

Training Loop with Policy Gradient Loss

The core training loop collects episode trajectories and computes the combined actor-critic loss:

# Training loop

for ep in range(300):
    state = env.reset()
    states, actions, probs, rewards = [], [], [], []
    done = False
    while not done:
        prob, _ = model(np.expand_dims(state, 0))
        prob = prob.numpy()[0]
        action = np.random.choice(num_actions, p=prob)
        next_state, reward, done, _ = env.step(action)

        states.append(state)
        actions.append(action)
        probs.append(prob)
        rewards.append(reward)

        state = next_state

    G = discounted_rewards(rewards)
    with tf.GradientTape() as tape:
        pi, V = model(np.array(states))
        # Actor loss (policy gradient): -log π(a|s) * G

        action_one_hot = tf.one_hot(actions, num_actions)
        log_pi = tf.math.log(tf.reduce_sum(pi * action_one_hot, axis=1) + 1e-8)
        actor_loss = -tf.reduce_mean(log_pi * G)
        # Critic loss (value regression): MSE between predicted and actual returns

        critic_loss = tf.reduce_mean(tf.square(V[:,0] - G))
        loss = actor_loss + critic_loss
    grads = tape.gradient(loss, model.trainable_variables)
    optimizer.apply_gradients(zip(grads, model.trainable_variables))

The actor loss implements the policy gradient theorem: ∇_θ J ≈ Σ_t ∇_θ log π_θ(a_t|s_t) * G_t. The critic loss trains the value estimator to predict returns accurately, reducing variance in the policy gradient.

Q-Learning: Deep Q-Network Implementation in PyTorch

The Q-learning approach learns an action-value function Q(s,a) rather than a policy directly. The optimal policy selects actions with maximum Q-value: a* = argmax_a Q(s,a). The PyTorch notebook CartPole-RL-PyTorch.ipynb implements this using the Bellman equation for target updates.

Q-Network Architecture

import gym, torch, torch.nn as nn, torch.optim as optim
import random, numpy as np

env = gym.make('CartPole-v1')
state_dim = env.observation_space.shape[0]
n_actions = env.action_space.n

class QNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc = nn.Sequential(
            nn.Linear(state_dim, 64), nn.ReLU(),
            nn.Linear(64, n_actions)
        )
    def forward(self, x):
        return self.fc(x)

qnet = QNet()
optimizer = optim.Adam(qnet.parameters(), lr=0.001)
criterion = nn.MSELoss()
gamma = 0.99

ε-Greedy Exploration and Experience Replay

Q-learning requires exploration to discover good actions. The implementation uses ε-greedy action selection with decaying ε, and a simple replay buffer to break correlation between consecutive transitions:

epsilon = 1.0
memory = []          # simple list replay buffer

BATCH = 64
MAX_MEM = 5000

def epsilon_greedy(state):
    if random.random() < epsilon:
        return env.action_space.sample()
    q_vals = qnet(torch.from_numpy(state).float().unsqueeze(0))
    return torch.argmax(q_vals).item()

Training Loop with Bellman Target Updates

The target Q-value follows the Bellman equation: Q(s,a) ← r + γ * max_a' Q(s',a'):

for episode in range(500):
    state = env.reset()
    done = False
    while not done:
        action = epsilon_greedy(state)
        nxt, reward, done, _ = env.step(action)
        memory.append((state, action, reward, nxt, done))
        if len(memory) > MAX_MEM: memory.pop(0)

        # Sample minibatch for experience replay

        if len(memory) >= BATCH:
            batch = random.sample(memory, BATCH)
            s_batch, a_batch, r_batch, ns_batch, d_batch = zip(*batch)
            s_batch = torch.tensor(s_batch, dtype=torch.float32)
            a_batch = torch.tensor(a_batch, dtype=torch.int64).unsqueeze(1)
            r_batch = torch.tensor(r_batch, dtype=torch.float32).unsqueeze(1)
            ns_batch = torch.tensor(ns_batch, dtype=torch.float32)
            d_batch = torch.tensor(d_batch, dtype=torch.float32).unsqueeze(1)

            # Current Q-values for selected actions

            q_vals = qnet(s_batch).gather(1, a_batch)
            # Target Q-values: r + γ * max_a' Q(s',a')

            with torch.no_grad():
                next_q = qnet(ns_batch).max(1)[0].unsqueeze(1)
                target = r_batch + gamma * next_q * (1 - d_batch)
            loss = criterion(q_vals, target)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()
        state = nxt
    epsilon = max(0.01, epsilon * 0.995)   # decay exploration

Policy Gradients vs. Q-Learning: When to Use Each

Aspect	Policy Gradients (Actor-Critic)	Q-Learning (Deep Q-Network)
What is learned	Stochastic policy π(a|s) directly	Action-value function Q(s,a)
Policy type	Probabilistic, naturally explores	Deterministic (after argmax), requires ε-greedy
Best for	Continuous action spaces, stochastic domains	Discrete action spaces, sample-efficient learning
Variance	Higher variance, needs baseline (critic)	Lower variance, more stable with replay
Convergence	Slower but handles complex policies	Faster on simple tasks like CartPole

In the AI-For-Beginners curriculum, policy gradients are implemented with TensorFlow following the official TensorFlow actor-critic tutorial, while Q-learning follows the PyTorch DQN tutorial.

Key Files and Next Steps

File	Purpose
lessons/6-Other/22-DeepRL/README.md	Environment setup and learning objectives
lessons/6-Other/22-DeepRL/CartPole-RL-TF.ipynb	Complete TensorFlow actor-critic implementation
lessons/6-Other/22-DeepRL/CartPole-RL-PyTorch.ipynb	Complete PyTorch Q-learning implementation
lessons/6-Other/22-DeepRL/lab/README.md	Assignment to apply Q-learning to Mountain Car

To extend your learning, the lab assignment challenges you to implement deep reinforcement learning with policy gradients and Q-learning on the Mountain Car environment—a more difficult control problem requiring sustained exploration.

Summary

• Policy gradients directly optimize action probabilities using the actor-critic architecture, combining policy gradient loss with value regression for stable training.
• Q-learning estimates action-values through the Bellman equation and uses ε-greedy exploration with experience replay for sample efficiency.
• Both methods are fully implemented in the Microsoft AI-For-Beginners repository with runnable notebooks for CartPole-v1.
• The TensorFlow notebook (CartPole-RL-TF.ipynb) demonstrates actor-critic, while the PyTorch notebook (CartPole-RL-PyTorch.ipynb) covers Q-learning.

Frequently Asked Questions

What is the difference between actor-critic and Q-learning?

Actor-critic is a policy-gradient method that learns both a policy (actor) and a value function (critic) simultaneously, using the critic to reduce variance in policy updates. Q-learning is a value-based method that learns action-values directly and derives the policy implicitly through argmax selection. Actor-critic handles continuous actions naturally; Q-learning works best with discrete actions.

Why does the policy gradient use discounted rewards?

Discounted rewards weight immediate and future rewards by their temporal distance, preventing infinite returns in continuing environments and matching the mathematical objective of maximizing expected cumulative return. The normalization step in discounted_rewards further stabilizes training by controlling variance across episodes.

How does experience replay improve Q-learning?

Experience replay breaks the correlation between consecutive transitions by sampling random mini-batches from stored experiences, rather than learning from correlated sequential data. This improves sample efficiency and learning stability, as implemented in the PyTorch notebook's simple list-based replay buffer.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}