How to Solve CartPole and MountainCar with DQN Reinforcement Learning: A Complete PyTorch Guide

Train a Deep Q-Network (DQN) agent to solve OpenAI Gym's CartPole-v1 and MountainCar-v0 environments using experience replay, target networks, and ε-greedy exploration.

The microsoft/AI-For-Beginners repository provides production-ready DQN implementations that demonstrate deep reinforcement learning fundamentals through two classic control problems. This guide walks through the complete training pipeline implemented in lessons/6-Other/22-DeepRL/, showing how to build, train, and deploy DQN agents that achieve optimal performance on both environments.

What Is DQN and Why Use It for CartPole and MountainCar?

Deep Q-Networks combine Q-learning with deep neural networks to approximate the action-value function Q(s, a) in environments with continuous or high-dimensional state spaces. Unlike tabular Q-learning, DQN scales to problems where the state space is too large for explicit enumeration.

The CartPole-v1 environment presents a 4-dimensional continuous state (cart position, velocity, pole angle, pole velocity). MountainCar-v0 adds complexity with sparse rewards and requires the agent to learn momentum-based strategies. Both are ideal DQN benchmarks because they:

• Run fast enough for rapid experimentation
• Require function approximation rather than tabular methods
• Demonstrate key DQN components: replay buffers, target networks, and exploration decay

DQN Architecture for CartPole and MountainCar

The DQN class in CartPole-RL-PyTorch.ipynb implements a simple yet effective fully-connected architecture:

import torch
import torch.nn as nn
import torch.nn.functional as F

class DQN(nn.Module):
    """Simple fully-connected network for approximating Q(s, a)."""
    def __init__(self, state_dim: int, action_dim: int, hidden_dim: int = 128):
        super().__init__()
        self.fc1 = nn.Linear(state_dim, hidden_dim)
        self.fc2 = nn.Linear(hidden_dim, hidden_dim)
        self.out = nn.Linear(hidden_dim, action_dim)

    def forward(self, x):
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        return self.out(x)                     # shape: (batch, action_dim)

Key design decisions:

• Two hidden layers (128 units each) with ReLU activations provide sufficient capacity for CartPole's 4D state space
• Output dimension equals action count: 2 for CartPole (left/right), 3 for MountainCar (left/none/right)
• No activation on output layer — raw Q-values allow the loss function to penalize overestimation appropriately

The Six Core DQN Components Explained

1. Environment Setup with OpenAI Gym

import gym

env = gym.make('CartPole-v1')               # or 'MountainCar-v0' for the lab

state_dim = env.observation_space.shape[0]  # 4 for CartPole, 2 for MountainCar

action_dim = env.action_space.n             # 2 for CartPole, 3 for MountainCar

The gym.make() interface standardizes environment interaction across both problems, enabling identical agent code with only hyperparameter adjustments.

2. Experience Replay Buffer

The replay buffer breaks harmful temporal correlations by storing transitions as (state, action, reward, next_state, done) tuples and sampling random mini-batches for training:

from collections import deque

replay_buffer = deque(maxlen=10000)         # Circular buffer with 10k capacity

# Store transition after each environment step

replay_buffer.append((state, action, reward, next_state, done))

# Sample random batch for training

batch = random.sample(replay_buffer, batch_size=64)

Buffer sizes of 10,000–50,000 work well for these environments; larger buffers improve sample diversity but increase memory consumption.

3. ε-Greedy Exploration Policy

Epsilon-greedy action selection balances exploration (random actions) and exploitation (greedy Q-value maximization):

def select_action(state, epsilon):
    if random.random() < epsilon:
        return env.action_space.sample()    # Explore: random action

    with torch.no_grad():
        q_vals = policy_net(torch.FloatTensor(state))
        return q_vals.argmax().item()       # Exploit: best known action

Exploration decay schedule:

epsilon = 1.0           # Start fully random

epsilon_decay = 0.995   # Decay per episode

min_epsilon = 0.01      # Floor to maintain some exploration

epsilon = max(min_epsilon, epsilon * epsilon_decay)

This schedule typically reaches minimum epsilon within 500–800 episodes, by which time the policy has learned viable behaviors.

4. Target Network for Stable Learning

The target network is a lagged copy of the policy network that stabilizes the TD target calculation:

policy_net = DQN(state_dim, action_dim)
target_net = DQN(state_dim, action_dim)
target_net.load_state_dict(policy_net.state_dict())
target_net.eval()       # No gradient computation for target network

# Periodic synchronization (every N episodes)

if episode % target_update == 0:
    target_net.load_state_dict(policy_net.state_dict())

Without target networks, the Q-update chases its own moving target, causing training instability—particularly problematic in MountainCar's sparse-reward setting.

5. Huber Loss for Robust TD Errors

The Huber loss (Smooth L1 Loss) combines MSE for small errors and MAE for large errors, reducing gradient sensitivity to outliers:

loss = F.smooth_l1_loss(q_curr, q_target)   # More stable than MSE alone

6. Adam Optimizer with Moderate Learning Rate

optimizer = torch.optim.Adam(policy_net.parameters(), lr=1e-3)

Learning rates between 1e-4 and 1e-3 work reliably; higher rates cause divergence in MountainCar, while lower rates slow CartPole convergence unnecessarily.

Complete DQN Training Loop

The following implementation from CartPole-RL-PyTorch.ipynb demonstrates the full training procedure:

import torch, random, numpy as np
from collections import deque
import torch.nn.functional as F

# Hyperparameters

batch_size = 64
gamma = 0.99
target_update = 10          # Episodes between target network sync

# Networks and optimizer

policy_net = DQN(state_dim, action_dim)
target_net = DQN(state_dim, action_dim)
target_net.load_state_dict(policy_net.state_dict())
target_net.eval()
optimizer = torch.optim.Adam(policy_net.parameters(), lr=1e-3)

replay_buffer = deque(maxlen=10000)

# Training loop

for episode in range(1, 1001):
    state = env.reset()
    episode_reward = 0
    done = False
    
    while not done:
        # Select and execute action

        action = select_action(state, epsilon)
        next_state, reward, done, _ = env.step(action)
        replay_buffer.append((state, action, reward, next_state, done))
        episode_reward += reward
        state = next_state

        # Perform network update if buffer has enough samples

        if len(replay_buffer) >= batch_size:
            batch = random.sample(replay_buffer, batch_size)
            s, a, r, s_next, d = zip(*batch)

            # Prepare tensors

            s = torch.FloatTensor(s)
            a = torch.LongTensor(a).unsqueeze(1)
            r = torch.FloatTensor(r).unsqueeze(1)
            s_next = torch.FloatTensor(s_next)
            d = torch.FloatTensor(d).unsqueeze(1)

            # Current Q estimates for taken actions

            q_curr = policy_net(s).gather(1, a)

            # Target Q values (no gradient)

            with torch.no_grad():
                q_next = target_net(s_next).max(1, keepdim=True)[0]
                q_target = r + gamma * q_next * (1 - d)

            # Gradient descent step

            loss = F.smooth_l1_loss(q_curr, q_target)
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

    # Decay exploration

    epsilon = max(min_epsilon, epsilon * epsilon_decay)
    
    # Sync target network

    if episode % target_update == 0:
        target_net.load_state_dict(policy_net.state_dict())

    print(f"Episode {episode}, Reward: {episode_reward}, Epsilon: {epsilon:.3f}")

Adapting DQN for MountainCar: Three Critical Modifications

The MountainCar-lab.ipynb notebook demonstrates that solving MountainCar-v0 requires environment-specific adjustments:

1. Reward Shaping

MountainCar's native reward is -1 per step with 0 at goal, providing no gradient signal until the car reaches the flag. The lab implements modified reward shaping:

• Negative step penalty encourages efficiency
• Height-based bonus provides intermediate signal: reward = reward + abs(next_state[1]) (velocity magnitude)

This transformation dramatically accelerates learning compared to sparse original rewards.

2. Extended Episode Length

env = gym.make('MountainCar-v0')

# Episode limit: 200 steps (vs 500 for CartPole-v1)

# Buffer must accommodate longer trajectories before learning begins

The shorter episode limit (200 vs 500) means fewer transitions per episode, requiring more episodes to fill the replay buffer adequately.

3. Increased Network Capacity

# MountainCar often benefits from wider layers

policy_net = DQN(state_dim=2, action_dim=3, hidden_dim=256)  # vs 128 for CartPole

The 2D state space (position, velocity) with nonlinear dynamics benefits from additional parameters to approximate the complex value landscape near the hill crest.

Performance Expectations and Verification

CartPole-v1:

• Convergence: Typically 300–800 episodes
• Success criterion: Average reward ≥ 195 over 100 consecutive episodes
• Maximum reward: 200 (environment limit)
• Final policy: Near-deterministic, optimal pole balancing

MountainCar-v0:

• Convergence: Typically 1000–3000 episodes
• Success criterion: Consistent goal reaching with minimal steps
• Final policy: Builds momentum through back-and-forth swinging

File Locations and Additional Resources

Resource	Path	Description
CartPole DQN (PyTorch)	lessons/6-Other/22-DeepRL/CartPole-RL-PyTorch.ipynb	Complete working implementation with visualization
MountainCar Lab	lessons/6-Other/22-DeepRL/lab/MountainCar.ipynb	Adapted DQN with reward shaping exercises
CartPole (TensorFlow)	lessons/6-Other/22-DeepRL/CartPole-RL-TF.ipynb	Equivalent implementation in Keras
Lesson README	lessons/6-Other/22-DeepRL/README.md	Conceptual overview and further reading

All notebooks are executable in standard Jupyter environments with gym, torch, and numpy installed.

Summary

• DQN solves CartPole and MountainCar by approximating Q-values with neural networks, enabling generalization across continuous state spaces.

• Six components are essential: experience replay buffers, ε-greedy exploration, target networks, Huber loss, Adam optimization, and proper network architecture.

• MountainCar requires adaptation: reward shaping, extended training, and increased network capacity address its sparse-reward challenge.

• Code in microsoft/AI-For-Beginners provides production-ready implementations with clear hyperparameter guidance for both environments.

Frequently Asked Questions

What makes DQN different from standard Q-learning for CartPole?

Standard Q-learning uses a table to store Q-values for every state-action pair. CartPole's continuous 4D state space makes this infeasible—DQN replaces the table with a neural network that generalizes across similar states. This function approximation enables the agent to learn from states it has never explicitly visited.

Why does MountainCar need reward shaping when CartPole does not?

CartPole provides dense rewards (+1 for every step the pole remains upright), giving continuous learning signal. MountainCar's native reward is sparse (-1 per step, 0 at goal), meaning the agent receives identical feedback for most of its experience. Reward shaping adds intermediate signals that guide the policy toward the goal region without changing the optimal solution.

How often should I update the target network?

The target_update parameter in the microsoft/AI-For-Beginners implementation syncs every 10 episodes. Values between 5–50 episodes work well for these environments. More frequent updates increase tracking of recent improvements but reduce stability; less frequent updates stabilize training but slow propagation of value discoveries.

Can I use the same hyperparameters for both environments?

No—MountainCar requires modification. The lab notebook specifically increases the hidden layer size to 256 units and modifies reward calculations. CartPole trains reliably with default settings, while MountainCar typically needs 2–3× more episodes and benefits from adjusted exploration decay rates.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}