Debugging and Profiling Techniques in Phase 0: A Complete Guide to AI Diagnostics

Phase 0 of the AI‑Engineering‑From‑Scratch curriculum teaches nine essential debugging and profiling techniques—from simple print statements to GPU memory tracking—that form a complete diagnostic toolkit for neural‑network development.

The rohitg00/ai-engineering-from-scratch repository structures its curriculum into distinct phases, with Phase 0 (Setup & Tooling) dedicated to foundational workflows. Lesson 12‑debugging‑and‑profiling provides a systematic introduction to identifying performance bottlenecks and silent bugs in PyTorch code before students advance to model building.

The Nine Core Debugging and Profiling Techniques

The lesson phases/00-setup-and-tooling/12-debugging-and-profiling/docs/en.md breaks diagnostics into nine practical categories, each illustrated with runnable Python examples.

Print Debugging

Print debugging remains the fastest way to inspect tensor states during training. The curriculum demonstrates using simple print statements and custom debug_print helpers to validate tensor shapes, dtypes, and detect NaN values mid‑training.

Python Debugger (pdb and breakpoint)

For interactive investigation, the lesson covers the Python Debugger (pdb) and the modern breakpoint() API. Students learn to step through execution line‑by‑line, inspect local variables, and set conditional breakpoints that trigger only when specific tensor conditions are met.

Python Logging

Moving beyond ad‑hoc prints, the curriculum teaches the logging module for production‑grade diagnostics. This includes configuring severity levels (DEBUG, INFO, WARNING), formatting log messages, and capturing output to rotating files for long training runs.

Timing Code Sections

To locate performance bottlenecks, students use time.perf_counter() and custom Timer context managers. These utilities measure elapsed time for critical sections like forward passes or data loading pipelines.

cProfile and line_profiler

For systematic profiling, the lesson introduces cProfile for whole‑program statistical profiling and line_profiler for line‑level granularity. These tools identify hot spots by showing exact execution counts and time spent per line of model code.

Memory Profiling

Memory diagnostics cover three distinct layers:

• CPU memory tracking with tracemalloc for locating reference leaks
• CPU line‑level profiling with memory_profiler
• GPU memory monitoring using PyTorch’s torch.cuda.memory_allocated and torch.cuda.max_memory_allocated

Common AI Bugs and Detection Methods

The curriculum dedicates a section to AI‑specific failure modes:

• Shape mismatches between layers or batch dimensions
• NaN losses from gradient explosions or bad initialization
• Data leakage between train/validation splits
• Wrong device placement (CPU/GPU mismatch)

TensorBoard Basics

For visual debugging, students learn TensorBoard integration: logging scalars for loss curves, histograms for weight distributions, and image grids for batch visualization. This replaces console‑only debugging with time‑series analysis.

VS Code Debugger Integration

Finally, the lesson covers IDE integration by configuring launch.json for VS Code using debugpy. This enables seamless breakpoint debugging inside the editor without terminal pdb commands.

Reference Implementation: debug_tools.py

All nine techniques are implemented in a single utility script located at phases/00-setup-and-tooling/12-debugging-and-profiling/code/debug_tools.py. The file contains runnable demonstrations for immediate use:

# phases/00-setup-and-tooling/12-debugging-and-profiling/code/debug_tools.py

def demo_print_debugging():
    """Simple tensor inspection prints."""
    pass

def demo_timing():
    """Context manager timing examples."""
    pass

def demo_memory_tracking():
    """CPU memory with tracemalloc."""
    pass

def demo_shape_checking():
    """Validate tensor dimensions."""
    pass

def demo_nan_detection():
    """Check for NaN in losses."""
    pass

def demo_device_checking():
    """Verify tensor device placement."""
    pass

def demo_gradient_health():
    """Inspect gradient norms."""
    pass

def demo_gpu_memory():
    """Monitor torch.cuda allocations."""
    pass

def demo_logging():
    """Structured logging setup."""
    pass

def demo_conditional_breakpoint():
    """Smart breakpoint insertion."""
    pass

def main():
    demo_print_debugging()
    demo_timing()
    demo_memory_tracking()
    # Sequential demonstration of all techniques

Running python phases/00-setup-and-tooling/12-debugging-and-profiling/code/debug_tools.py executes a step‑by‑step walkthrough, outputting diagnostic markers like:

▶️  Print debugging – tensor shape: torch.Size([32, 128])
🕒  Timing – forward pass took 0.0123 s
📊  Memory – current CPU allocation: 12.4 MiB, GPU allocation: 45.2 MiB
⚠️  NaN detection – loss contains NaN at step 27

Key Files and Resources

The debugging lesson ships with three primary resources:

• phases/00-setup-and-tooling/12-debugging-and-profiling/docs/en.md — Narrative documentation detailing usage patterns, common pitfalls, and when to apply each technique
• phases/00-setup-and-tooling/12-debugging-and-profiling/code/debug_tools.py — Reference implementations of all nine debugging utilities
• phases/00-setup-and-tooling/12-debugging-and-profiling/outputs/prompt-debug-ai-code.md — A pre‑crafted prompt for generating AI‑assisted debugging suggestions on custom code snippets

Summary

Phase 0 of the AI‑Engineering‑From‑Scratch curriculum provides a layered debugging and profiling toolkit:

• Quick diagnostics via print statements and Python’s built‑in debugger
• Systematic observation through the logging module and TensorBoard visualization
• Performance analysis using cProfile, line_profiler, and custom timing contexts
• Resource monitoring with tracemalloc, memory_profiler, and PyTorch CUDA APIs
• AI‑specific checks for shape mismatches, NaN values, data leakage, and device placement

These techniques, implemented in debug_tools.py, create a portable diagnostic kit students reuse throughout the curriculum.

Frequently Asked Questions

How do I start the interactive debugger in the middle of training?

Insert breakpoint() at any line in your training loop. When Python reaches that line, it drops into the pdb interactive shell where you can inspect variables, step through execution, or continue with c. For conditional stopping, wrap it in an if statement checking for NaN losses or abnormal tensor values.

What is the difference between cProfile and line_profiler?

cProfile provides function‑level statistics showing call counts and total time per function, making it ideal for finding which model components consume the most time. line_profiler requires decoration with @profile and shows time spent on individual lines within functions, perfect for optimizing specific hot loops without guessing which line causes slowdowns.

How do I detect GPU memory leaks in PyTorch training?

Use torch.cuda.memory_allocated() and torch.cuda.max_memory_allocated() at strategic points in your training loop—after forward pass, backward pass, and optimizer step. If allocated memory increases every epoch without resetting, you likely have retained tensors or circular references. Pair this with tracemalloc for CPU leaks to cover both memory spaces.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}