How to Set Up GPU Support for TensorFlow and PyTorch in AI For Beginners

Enable GPU acceleration in the Microsoft AI For Beginners course by installing NVIDIA drivers, CUDA 11.8, and cuDNN 8.x, then configuring TensorFlow with tf.config.experimental.set_memory_growth() and PyTorch with torch.device("cuda") to offload training onto the GPU.

The Microsoft AI For Beginners repository provides hands-on notebooks for deep learning, computer vision, and NLP that execute significantly faster when configured to use GPU resources. To set up GPU support for TensorFlow and PyTorch in AI For Beginners, you must align system-level drivers with the specific library versions pinned in the environment.yml file. This guide walks through the exact configuration steps and code patterns already implemented in the course source code, including references to TransferLearningTF.ipynb and TransferLearningPyTorch.ipynb.

System-Level Prerequisites

Before running any notebooks, verify these components are installed on your host machine. The repository's environment.yml already pins tensorflow==2.17.0 and pulls the GPU-enabled PyTorch build from conda channels, so you only need to configure the underlying drivers.

• NVIDIA Driver (≥450): Provides low-level GPU access. Verify installation by running nvidia-smi in your terminal.
• CUDA Toolkit (11.8): Required by TensorFlow 2.17.0 binaries. Verify with nvcc --version.
• cuDNN (8.x): Deep learning primitives for CUDA 11.8. Check the header file at <CUDA_PATH>/include/cudnn.h.
• Conda Environment: Create the ai4beg environment from environment.yml to install the correct GPU-enabled Python packages without version conflicts.

TensorFlow GPU Configuration

TensorFlow automatically detects visible GPUs but allocates all GPU memory by default, which can cause out-of-memory errors during iterative experimentation in Jupyter notebooks.

Detecting GPUs and Enabling Memory Growth

The course demonstrates GPU setup in lessons/4-ComputerVision/08-TransferLearning/TransferLearningTF.ipynb and lessons/5-NLP/16-RNN/RNNTF.ipynb. Insert this snippet at the start of any TensorFlow notebook to prevent memory allocation issues:

import tensorflow as tf

# List physical GPUs

gpus = tf.config.list_physical_devices('GPU')
print("GPUs detected:", gpus)

# Enable memory growth to prevent allocation of all GPU memory upfront

if gpus:
    try:
        for gpu in gpus:
            tf.config.experimental.set_memory_growth(gpu, True)
        logical_gpus = tf.config.list_logical_devices('GPU')
        print(f"Enabled memory growth on {len(gpus)} physical GPU(s), "
              f"{len(logical_gpus)} logical GPU(s) available.")
    except RuntimeError as e:
        # Memory growth must be set before GPUs are initialized

        print(e)

After execution, subsequent tf.keras operations automatically run on the GPU if available.

PyTorch GPU Configuration

Unlike TensorFlow, PyTorch requires explicit tensor and model movement to the GPU device using the torch.device API.

Device Selection and Tensor Transfer

The pattern appears in lessons/4-ComputerVision/08-TransferLearning/TransferLearningPyTorch.ipynb and lessons/3-NeuralNetworks/05-Frameworks/IntroPyTorch.ipynb. Use this initialization pattern to ensure all computations occur on the accelerator:

import torch

# Detect GPU availability

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(f"Using device: {device}")

# Example: Move model and input tensors to GPU

model = torch.nn.Sequential(
    torch.nn.Conv2d(3, 16, kernel_size=3, padding=1),
    torch.nn.ReLU(),
    torch.nn.Flatten(),
    torch.nn.Linear(16 * 32 * 32, 10)
).to(device)  # Model weights now reside on GPU

sample_input = torch.randn(8, 3, 32, 32, device=device)  # Tensor created directly on GPU

output = model(sample_input)

All tensors, including labels and loss values, must be created on or moved to device using .to(device) or the device= constructor argument to avoid CPU-GPU transfer bottlenecks.

GPU Implementation in Course Notebooks

The AI For Beginners curriculum already embeds these patterns in specific lessons:

• lessons/4-ComputerVision/08-TransferLearning/TransferLearningTF.ipynb: Lists GPUs and configures memory growth for TensorFlow transfer learning workflows.
• lessons/4-ComputerVision/08-TransferLearning/TransferLearningPyTorch.ipynb: Demonstrates full device selection and model transfer for PyTorch implementations.
• lessons/5-NLP/16-RNN/RNNTF.ipynb: Shows memory growth setup specifically for RNN training sequences.
• lessons/4-ComputerVision/09-Autoencoders/AutoencodersTF.ipynb: Checks for GPU availability and warns if hardware acceleration is missing.
• lessons/3-NeuralNetworks/05-Frameworks/IntroPyTorch.ipynb: Provides a complete PyTorch GPU tutorial covering torch.cuda.is_available() and tensor operations.

Common Pitfalls and Solutions

Symptom: tensorflow.python.framework.errors_impl.NotFoundError: The source TensorFlow binary is compiled without GPU support

• Cause: Installed CPU-only TensorFlow instead of the GPU build.
• Fix: Use the conda environment from environment.yml or install tensorflow==2.17.0 with CUDA 11.8 support via conda install tensorflow==2.17.0.

Symptom: torch.cuda.is_available() returns False despite nvidia-smi showing a GPU

• Cause: PyTorch CUDA build mismatches the installed driver version.
• Fix: Recreate the conda environment with conda install pytorch torchvision torchaudio cudatoolkit=11.8 -c pytorch to ensure binary compatibility.

Symptom: Out-of-memory crashes after several training cells

• Cause: TensorFlow allocated all GPU memory or PyTorch tensors accumulated in cache without release.
• Fix: Enable memory growth in TensorFlow using the snippet above; in PyTorch, delete unused tensors with del tensor and call torch.cuda.empty_cache().

Symptom: "Skipping registering GPU devices…" in TensorFlow logs

• Cause: Docker or remote environment without GPU forwarding.
• Fix: Use docker run --gpus all when starting containers, or enable GPU support in VS Code Dev Container settings.

Quick Start Checklist

Add these two cells to the top of any notebook to enable GPU support automatically:

# TensorFlow GPU setup

import tensorflow as tf
if tf.config.list_physical_devices('GPU'):
    for gpu in tf.config.list_physical_devices('GPU'):
        tf.config.experimental.set_memory_growth(gpu, True)
    print("TensorFlow GPU ready with memory growth enabled")

# PyTorch GPU setup

import torch
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(f"PyTorch running on {device}")

Complete Training Examples

TensorFlow GPU-Ready Training Loop

import tensorflow as tf
from tensorflow import keras
import numpy as np

# GPU setup

gpus = tf.config.list_physical_devices('GPU')
if gpus:
    for gpu in gpus:
        tf.config.experimental.set_memory_growth(gpu, True)
    print("GPUs available:", gpus)

# Dummy data

x = np.random.rand(5000, 28, 28, 1).astype('float32')
y = np.random.randint(0, 10, size=(5000,))

# Model definition

model = keras.Sequential([
    keras.layers.Conv2D(32, 3, activation='relu', input_shape=(28,28,1)),
    keras.layers.Flatten(),
    keras.layers.Dense(10, activation='softmax')
])
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])

# Train on GPU automatically

model.fit(x, y, epochs=5, batch_size=128)

PyTorch GPU-Ready Training Loop

import torch
import torch.nn as nn
import torch.optim as optim

# GPU setup

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(f"Using device: {device}")

# Dummy data on GPU

x = torch.randn(5000, 1, 28, 28, device=device)
y = torch.randint(0, 10, (5000,), device=device)

# Model on GPU

model = nn.Sequential(
    nn.Conv2d(1, 32, kernel_size=3, padding=1),
    nn.ReLU(),
    nn.Flatten(),
    nn.Linear(32 * 28 * 28, 10)
).to(device)

criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters())

# Training loop on GPU

for epoch in range(5):
    optimizer.zero_grad()
    out = model(x)
    loss = criterion(out, y)
    loss.backward()
    optimizer.step()
    print(f"Epoch {epoch+1}, loss={loss.item():.4f}")

Summary

• Install NVIDIA drivers ≥450, CUDA 11.8, and cuDNN 8.x before creating the conda environment from environment.yml.
• Use tf.config.experimental.set_memory_growth() in TensorFlow notebooks to prevent memory allocation errors during training.
• Use torch.device("cuda") and .to(device) in PyTorch to explicitly move models and tensors to the GPU.
• Reference TransferLearningTF.ipynb and TransferLearningPyTorch.ipynb for working implementations within the course structure.
• Add the quick setup snippets to any notebook to automatically leverage GPU acceleration for computer vision and NLP labs, reducing training time from hours to minutes.

Frequently Asked Questions

Why does TensorFlow show "GPU devices not found" even though nvidia-smi works?

This occurs when you install the CPU-only TensorFlow build instead of the GPU-enabled version. The AI For Beginners environment.yml pins tensorflow==2.17.0 which includes GPU support when installed via conda. If using pip, ensure you install the specific version compiled against CUDA 11.8, and verify your NVIDIA driver is compatible with CUDA 11.8.

How do I enable GPU support when running the course in a Docker container?

You must explicitly forward the GPU devices to the container. Use the --gpus all flag when running Docker: docker run --gpus all -it -v $(pwd):/workspace microsoft/ai-for-beginners. If using VS Code Dev Containers, ensure the "runArgs": ["--gpus=all"] setting is present in .devcontainer/devcontainer.json to allow the container to access the host GPU.

Why does PyTorch report that CUDA is not available on a fresh install?

PyTorch binaries are compiled against specific CUDA versions. If your system CUDA driver version mismatches the PyTorch build (e.g., CUDA 12.x driver with CUDA 11.8 PyTorch), torch.cuda.is_available() returns False. Recreate the ai4beg conda environment using the provided environment.yml, which installs the PyTorch build matching CUDA 11.8, or manually install with conda install pytorch cudatoolkit=11.8 -c pytorch.

Do I need to modify every notebook to use GPU, or is it automatic?

TensorFlow automatically uses the GPU once the drivers are correct, but you should manually add the memory growth code to prevent out-of-memory errors. PyTorch requires explicit device management—you must define device and move models and tensors using .to(device). The course notebooks TransferLearningTF.ipynb and TransferLearningPyTorch.ipynb already include these patterns for reference, which you can copy into other lessons.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}