# How to Solve CartPole and MountainCar with DQN Reinforcement Learning: A Complete PyTorch Guide

> Learn to solve CartPole and MountainCar with DQN reinforcement learning using PyTorch. Master experience replay, target networks, and epsilon-greedy exploration in this complete guide.

- Repository: [Microsoft/AI-For-Beginners](https://github.com/microsoft/AI-For-Beginners)
- Tags: tutorial
- Published: 2026-08-20

---

**Train a Deep Q-Network (DQN) agent to solve OpenAI Gym's CartPole-v1 and MountainCar-v0 environments using experience replay, target networks, and ε-greedy exploration.**

The **microsoft/AI-For-Beginners** repository provides production-ready DQN implementations that demonstrate deep reinforcement learning fundamentals through two classic control problems. This guide walks through the complete training pipeline implemented in `lessons/6-Other/22-DeepRL/`, showing how to build, train, and deploy DQN agents that achieve optimal performance on both environments.

## What Is DQN and Why Use It for CartPole and MountainCar?

**Deep Q-Networks** combine Q-learning with deep neural networks to approximate the action-value function **Q(s, a)** in environments with continuous or high-dimensional state spaces. Unlike tabular Q-learning, DQN scales to problems where the state space is too large for explicit enumeration.

The **CartPole-v1** environment presents a 4-dimensional continuous state (cart position, velocity, pole angle, pole velocity). **MountainCar-v0** adds complexity with sparse rewards and requires the agent to learn momentum-based strategies. Both are ideal DQN benchmarks because they:

- Run fast enough for rapid experimentation
- Require function approximation rather than tabular methods
- Demonstrate key DQN components: replay buffers, target networks, and exploration decay

## DQN Architecture for CartPole and MountainCar

The `DQN` class in `CartPole-RL-PyTorch.ipynb` implements a simple yet effective fully-connected architecture:

```python
import torch
import torch.nn as nn
import torch.nn.functional as F

class DQN(nn.Module):
    """Simple fully-connected network for approximating Q(s, a)."""
    def __init__(self, state_dim: int, action_dim: int, hidden_dim: int = 128):
        super().__init__()
        self.fc1 = nn.Linear(state_dim, hidden_dim)
        self.fc2 = nn.Linear(hidden_dim, hidden_dim)
        self.out = nn.Linear(hidden_dim, action_dim)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        return self.out(x)                     # shape: (batch, action_dim)

```

**Key design decisions:**

- **Two hidden layers** (128 units each) with ReLU activations provide sufficient capacity for CartPole's 4D state space
- **Output dimension equals action count**: 2 for CartPole (left/right), 3 for MountainCar (left/none/right)
- **No activation on output layer** — raw Q-values allow the loss function to penalize overestimation appropriately

## The Six Core DQN Components Explained

### 1. Environment Setup with OpenAI Gym

```python
import gym

env = gym.make('CartPole-v1')               # or 'MountainCar-v0' for the lab

state_dim = env.observation_space.shape[0]  # 4 for CartPole, 2 for MountainCar

action_dim = env.action_space.n             # 2 for CartPole, 3 for MountainCar

```

The `gym.make()` interface standardizes environment interaction across both problems, enabling identical agent code with only hyperparameter adjustments.

### 2. Experience Replay Buffer

The **replay buffer** breaks harmful temporal correlations by storing transitions as **(state, action, reward, next_state, done)** tuples and sampling random mini-batches for training:

```python
from collections import deque

replay_buffer = deque(maxlen=10000)         # Circular buffer with 10k capacity

# Store transition after each environment step

replay_buffer.append((state, action, reward, next_state, done))

# Sample random batch for training

batch = random.sample(replay_buffer, batch_size=64)

```

Buffer sizes of **10,000–50,000** work well for these environments; larger buffers improve sample diversity but increase memory consumption.

### 3. ε-Greedy Exploration Policy

**Epsilon-greedy action selection** balances exploration (random actions) and exploitation (greedy Q-value maximization):

```python
def select_action(state, epsilon):
    if random.random() < epsilon:
        return env.action_space.sample()    # Explore: random action

    with torch.no_grad():
        q_vals = policy_net(torch.FloatTensor(state))
        return q_vals.argmax().item()       # Exploit: best known action

```

**Exploration decay schedule:**

```python
epsilon = 1.0           # Start fully random

epsilon_decay = 0.995   # Decay per episode

min_epsilon = 0.01      # Floor to maintain some exploration

epsilon = max(min_epsilon, epsilon * epsilon_decay)

```

This schedule typically reaches minimum epsilon within **500–800 episodes**, by which time the policy has learned viable behaviors.

### 4. Target Network for Stable Learning

The **target network** is a lagged copy of the policy network that stabilizes the TD target calculation:

```python
policy_net = DQN(state_dim, action_dim)
target_net = DQN(state_dim, action_dim)
target_net.load_state_dict(policy_net.state_dict())
target_net.eval()       # No gradient computation for target network

# Periodic synchronization (every N episodes)

if episode % target_update == 0:
    target_net.load_state_dict(policy_net.state_dict())

```

Without target networks, the Q-update chases its own moving target, causing training instability—particularly problematic in MountainCar's sparse-reward setting.

### 5. Huber Loss for Robust TD Errors

The **Huber loss** (Smooth L1 Loss) combines MSE for small errors and MAE for large errors, reducing gradient sensitivity to outliers:

```python
loss = F.smooth_l1_loss(q_curr, q_target)   # More stable than MSE alone

```

### 6. Adam Optimizer with Moderate Learning Rate

```python
optimizer = torch.optim.Adam(policy_net.parameters(), lr=1e-3)

```

Learning rates between **1e-4 and 1e-3** work reliably; higher rates cause divergence in MountainCar, while lower rates slow CartPole convergence unnecessarily.

## Complete DQN Training Loop

The following implementation from `CartPole-RL-PyTorch.ipynb` demonstrates the full training procedure:

```python
import torch, random, numpy as np
from collections import deque
import torch.nn.functional as F

# Hyperparameters

batch_size = 64
gamma = 0.99
target_update = 10          # Episodes between target network sync

# Networks and optimizer

policy_net = DQN(state_dim, action_dim)
target_net = DQN(state_dim, action_dim)
target_net.load_state_dict(policy_net.state_dict())
target_net.eval()
optimizer = torch.optim.Adam(policy_net.parameters(), lr=1e-3)

replay_buffer = deque(maxlen=10000)

# Training loop

for episode in range(1, 1001):
    state = env.reset()
    episode_reward = 0
    done = False
    
    while not done:
        # Select and execute action

        action = select_action(state, epsilon)
        next_state, reward, done, _ = env.step(action)
        replay_buffer.append((state, action, reward, next_state, done))
        episode_reward += reward
        state = next_state

        # Perform network update if buffer has enough samples

        if len(replay_buffer) >= batch_size:
            batch = random.sample(replay_buffer, batch_size)
            s, a, r, s_next, d = zip(*batch)

            # Prepare tensors

            s = torch.FloatTensor(s)
            a = torch.LongTensor(a).unsqueeze(1)
            r = torch.FloatTensor(r).unsqueeze(1)
            s_next = torch.FloatTensor(s_next)
            d = torch.FloatTensor(d).unsqueeze(1)

            # Current Q estimates for taken actions

            q_curr = policy_net(s).gather(1, a)

            # Target Q values (no gradient)

            with torch.no_grad():
                q_next = target_net(s_next).max(1, keepdim=True)[0]
                q_target = r + gamma * q_next * (1 - d)

            # Gradient descent step

            loss = F.smooth_l1_loss(q_curr, q_target)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

    # Decay exploration

    epsilon = max(min_epsilon, epsilon * epsilon_decay)
    
    # Sync target network

    if episode % target_update == 0:
        target_net.load_state_dict(policy_net.state_dict())

    print(f"Episode {episode}, Reward: {episode_reward}, Epsilon: {epsilon:.3f}")

```

## Adapting DQN for MountainCar: Three Critical Modifications

The **MountainCar-lab.ipynb** notebook demonstrates that solving MountainCar-v0 requires environment-specific adjustments:

### 1. Reward Shaping

MountainCar's native reward is **-1 per step** with **0 at goal**, providing no gradient signal until the car reaches the flag. The lab implements modified reward shaping:

- **Negative step penalty** encourages efficiency
- **Height-based bonus** provides intermediate signal: `reward = reward + abs(next_state[1])` (velocity magnitude)

This transformation dramatically accelerates learning compared to sparse original rewards.

### 2. Extended Episode Length

```python
env = gym.make('MountainCar-v0')

# Episode limit: 200 steps (vs 500 for CartPole-v1)

# Buffer must accommodate longer trajectories before learning begins

```

The shorter episode limit (200 vs 500) means fewer transitions per episode, requiring more episodes to fill the replay buffer adequately.

### 3. Increased Network Capacity

```python

# MountainCar often benefits from wider layers

policy_net = DQN(state_dim=2, action_dim=3, hidden_dim=256)  # vs 128 for CartPole

```

The 2D state space (position, velocity) with nonlinear dynamics benefits from additional parameters to approximate the complex value landscape near the hill crest.

## Performance Expectations and Verification

**CartPole-v1**:
- **Convergence**: Typically 300–800 episodes
- **Success criterion**: Average reward ≥ 195 over 100 consecutive episodes
- **Maximum reward**: 200 (environment limit)
- **Final policy**: Near-deterministic, optimal pole balancing

**MountainCar-v0**:
- **Convergence**: Typically 1000–3000 episodes
- **Success criterion**: Consistent goal reaching with minimal steps
- **Final policy**: Builds momentum through back-and-forth swinging

## File Locations and Additional Resources

| Resource | Path | Description |
|----------|------|-------------|
| CartPole DQN (PyTorch) | `lessons/6-Other/22-DeepRL/CartPole-RL-PyTorch.ipynb` | Complete working implementation with visualization |
| MountainCar Lab | `lessons/6-Other/22-DeepRL/lab/MountainCar.ipynb` | Adapted DQN with reward shaping exercises |
| CartPole (TensorFlow) | `lessons/6-Other/22-DeepRL/CartPole-RL-TF.ipynb` | Equivalent implementation in Keras |
| Lesson README | [`lessons/6-Other/22-DeepRL/README.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/6-Other/22-DeepRL/README.md) | Conceptual overview and further reading |

All notebooks are executable in standard Jupyter environments with `gym`, `torch`, and `numpy` installed.

## Summary

- **DQN solves CartPole and MountainCar** by approximating Q-values with neural networks, enabling generalization across continuous state spaces.

- **Six components are essential**: experience replay buffers, ε-greedy exploration, target networks, Huber loss, Adam optimization, and proper network architecture.

- **MountainCar requires adaptation**: reward shaping, extended training, and increased network capacity address its sparse-reward challenge.

- **Code in microsoft/AI-For-Beginners** provides production-ready implementations with clear hyperparameter guidance for both environments.

## Frequently Asked Questions

### What makes DQN different from standard Q-learning for CartPole?

Standard Q-learning uses a table to store Q-values for every state-action pair. CartPole's continuous 4D state space makes this infeasible—**DQN replaces the table with a neural network** that generalizes across similar states. This function approximation enables the agent to learn from states it has never explicitly visited.

### Why does MountainCar need reward shaping when CartPole does not?

**CartPole provides dense rewards** (+1 for every step the pole remains upright), giving continuous learning signal. **MountainCar's native reward is sparse** (-1 per step, 0 at goal), meaning the agent receives identical feedback for most of its experience. Reward shaping adds intermediate signals that guide the policy toward the goal region without changing the optimal solution.

### How often should I update the target network?

The `target_update` parameter in the microsoft/AI-For-Beginners implementation syncs every **10 episodes**. Values between **5–50 episodes** work well for these environments. More frequent updates increase tracking of recent improvements but reduce stability; less frequent updates stabilize training but slow propagation of value discoveries.

### Can I use the same hyperparameters for both environments?

**No—MountainCar requires modification.** The lab notebook specifically increases the hidden layer size to 256 units and modifies reward calculations. CartPole trains reliably with default settings, while MountainCar typically needs 2–3× more episodes and benefits from adjusted exploration decay rates.