How to Setup Reinforcement Learning Environments Using OpenAI Gym CartPole for AI Beginners

You can setup reinforcement learning environments with OpenAI Gym CartPole by installing the gym package, creating the environment instance with gym.make("CartPole-v1"), and executing an episode loop using env.reset() and env.step() as demonstrated in the Microsoft AI-For-Beginners lesson 22 notebooks.

The AI-For-Beginners repository by Microsoft provides a comprehensive 12-week curriculum that introduces artificial intelligence concepts from symbolic AI to deep learning techniques. Within the "Other AI Techniques" section, lesson 22 covers Deep Reinforcement Learning and includes production-ready notebooks demonstrating how to setup reinforcement learning environments with OpenAI Gym CartPole for AI beginners. This article explains the repository architecture, walks through the exact file locations, and provides runnable code snippets to get your first agent interacting with the CartPole simulation.

Repository Structure for CartPole Examples

The Microsoft AI-For-Beginners curriculum organizes content under the lessons/ directory, with reinforcement learning materials located in lessons/6-Other/22-DeepRL/. This directory contains two primary implementation files that demonstrate the CartPole environment setup:

• CartPole-RL-TF.ipynb – TensorFlow implementation using tf.keras for policy networks
• CartPole-RL-PyTorch.ipynb – PyTorch implementation using torch.nn.Module

Both notebooks import from the shared environment.yml file at the repository root, which defines the Conda environment including Python, TensorFlow, PyTorch, and OpenCV dependencies. For pip-based installations, the repository also provides a requirements.txt file used by Binder and GitHub Codespaces.

Minimal Setup to Run CartPole

You can verify your installation and interact with the CartPole environment using a minimal Python script that requires only the gym package. This example follows the same initialization pattern found in lessons/6-Other/22-DeepRL/CartPole-RL-TF.ipynb but eliminates the deep learning dependencies for a quick sanity check.

# Install Gym (run once)

!pip install gym

import gym

# Create the environment

env = gym.make("CartPole-v1", render_mode="human")

# Reset to start a new episode

observation, info = env.reset()
total_reward = 0

# Simple random-action loop

for step in range(200):                # CartPole solves at ≥ 195 steps

    env.render()                       # Visualizes the cart and pole

    action = env.action_space.sample() # Random action (0 or 1)

    observation, reward, terminated, truncated, info = env.step(action)
    total_reward += reward
    if terminated or truncated:        # Episode finished

        break

env.close()
print(f"Episode finished after {step+1} steps, total reward {total_reward}")

Key methods explained:

• gym.make("CartPole-v1") – Instantiates the classic control environment where a pole attached to a cart must remain upright
• env.reset() – Initializes a fresh episode and returns the starting observation vector
• env.step(action) – Advances the simulation by one timestep, returning the next observation, immediate reward, and termination flags
• render_mode="human" – Opens a display window for visualization; omit this parameter in headless environments

Full Notebook Implementations

For production-ready implementations that extend beyond random actions, the repository provides two fully annotated notebooks in lessons/6-Other/22-DeepRL/.

TensorFlow Version (CartPole-RL-TF.ipynb): Contains installation helpers for gym and pygame, handles deprecation warnings using the new step API, and implements a policy gradient agent using tf.keras layers. The notebook explicitly sets render_mode='human' to suppress compatibility warnings.

PyTorch Version (CartPole-RL-PyTorch.ipynb): Mirrors the TensorFlow structure but utilizes torch tensors and nn.Module for the policy network. Both notebooks include trajectory collection loops that store states, actions, and rewards for batch training updates.

Each notebook references the optional lab exercises located in lessons/6-Other/22-DeepRL/lab/README.md, which provide additional environments like MountainCar for extended practice.

Extending from Random to Learned Policies

The minimal example above uses env.action_space.sample() to select random actions. To implement learning agents as shown in the curriculum, replace the random action selection with a neural network policy:

1. Define a policy network using tf.keras.Sequential (TensorFlow) or nn.Linear layers (PyTorch)
2. Collect trajectories by storing observations and rewards from env.step() calls
3. Compute returns and apply gradient descent to maximize cumulative reward

The CartPole-RL-TF.ipynb and CartPole-RL-PyTorch.ipynb files contain the complete training loops that implement these steps, allowing you to evolve from environment setup to full reinforcement learning training.

Summary

• Install dependencies using the environment.yml Conda file or pip install gym for minimal testing
• Create the simulation with gym.make("CartPole-v1", render_mode="human") located in lessons/6-Other/22-DeepRL/
• Interact with the environment using env.reset() to initialize and env.step() to execute actions
• Reference CartPole-RL-TF.ipynb for TensorFlow implementations or CartPole-RL-PyTorch.ipynb for PyTorch implementations
• Extend the random policy with neural networks using the training loops provided in the lesson 22 notebooks

Frequently Asked Questions

What Python packages are required to setup the CartPole environment?

You need the gym package installed via pip. For the full curriculum experience including deep learning frameworks, use the Conda environment defined in environment.yml at the repository root, which includes TensorFlow, PyTorch, and OpenCV. The minimal setup requires only pip install gym.

Where are the CartPole example notebooks located in the AI-For-Beginners repository?

The primary notebooks reside in lessons/6-Other/22-DeepRL/. Specifically, CartPole-RL-TF.ipynb contains the TensorFlow implementation and CartPole-RL-PyTorch.ipynb contains the PyTorch version. Both files demonstrate environment setup, random policy baselines, and neural network training.

How do I visualize the CartPole simulation during training?

Pass render_mode="human" when calling gym.make(), then invoke env.render() inside your episode loop. This opens a window displaying the cart and pole physics. If running in a headless cloud environment, omit the render_mode parameter to avoid display errors.

Can I run the CartPole example without installing TensorFlow or PyTorch?

Yes. The basic environment interaction requires only the gym package, allowing you to test env.reset() and env.step() with random actions. The deep learning frameworks are only necessary when implementing the policy network training sections of the notebooks.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}