How to Debug a Lesson That Fails to Run or Produces Unexpected Results: A Systematic Framework

Apply the "overfit-one-batch" test first, then use the NetworkDebugger class to monitor per-layer activations and gradients, check loss health for NaN or oscillation patterns, and validate gradients numerically before scaling to full training.

When you encounter silent crashes or puzzling outputs while working through the ai-engineering-from-scratch curriculum, the Debugging Neural Networks lesson (Phase 3, Lesson 13) provides a battle-tested framework to isolate root causes efficiently. This guide walks you through the exact diagnostic workflow implemented in debug_neural_nets.py to help you debug a lesson that fails to run or produces unexpected results without wasting compute on doomed full-scale runs.

Start with the Overfit-One-Batch Test

The fastest way to verify your model's fundamental integrity is to overfit a single tiny batch. This test exposes bugs in the model definition, loss function, or training loop before you commit significant resources.

According to the source code in phases/03-deep-learning-core/13-debugging-neural-networks/code/debug_neural_nets.py, the overfit_one_batch function (line 40) forces the model to memorize 8–32 samples across many optimization steps. If the loss does not approach approximately zero and accuracy fails to reach 100%, your architecture contains a critical flaw requiring immediate repair.

import torch
import torch.nn as nn
from debug_neural_nets import overfit_one_batch

# Minimal reproducible test

model = nn.Sequential(nn.Linear(10, 32), nn.ReLU(), nn.Linear(32, 2))
criterion = nn.CrossEntropyLoss()
x_batch = torch.randn(8, 10)
y_batch = (x_batch[:, 0] > 0).long()

overfit_one_batch(model, x_batch, y_batch, criterion)

Instrument Your Model with NetworkDebugger

Once the basic sanity check passes, layer the NetworkDebugger class over your model to capture per-layer statistics during live training. This hook-based monitor records activation statistics (mean, standard deviation, zero-fraction) and gradient statistics (mean, std, absolute-mean) without modifying your forward pass logic.

After running a few training steps, invoking print_report() generates a concise health summary that identifies dead ReLUs, exploding activations, and vanishing gradients. The class definition resides at line 7 of debug_neural_nets.py.

from debug_neural_nets import NetworkDebugger

debugger = NetworkDebugger(model)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

for step in range(20):
    optimizer.zero_grad()
    out = model(x_batch)
    loss = criterion(out, y_batch)
    debugger.record_loss(loss.item())
    loss.backward()
    optimizer.step()

debugger.print_report()   # prints health, activation & gradient alerts

debugger.remove_hooks()

Diagnose Loss Health Issues

The NetworkDebugger.check_loss_health() method automatically flags three common failure modes that occur when you debug a lesson that fails to run or produces unexpected results:

• NAN_OR_INF: Indicates NaN or Inf values in the loss, typically caused by a learning rate that is too high or log-of-zero operations in the loss computation.
• NOT_DECREASING: Signals loss plateaus, suggesting a learning rate that is too low or severe underfitting requiring architecture adjustments.
• OSCILLATING: Reveals wild loss swings characteristic of an excessively high learning rate causing optimization instability.

Inspect Activation and Gradient Diagnostics

Beyond loss curves, inspect the internal state of your network using the debugger's threshold-based alerts defined in the source:

• Dead Neurons: Flagged when over 50% of activations equal zero (DEAD_NEURONS), indicating dead ReLU syndrome that blocks gradient flow.
• Exploding Activations: Reported when mean magnitude exceeds 10 (EXPLODING_ACTIVATIONS), suggesting initialization or normalization issues.
• Vanishing Gradients: Identified when the absolute-mean gradient falls below 1e-7 (VANISHING_GRADIENT), signaling that earlier layers are not learning.

These diagnostics pinpoint exactly which layer is destabilizing your training without requiring manual tensor inspection across deep architectures.

Optimize Hyperparameters with Learning-Rate Finder

Before launching full training, determine the optimal learning rate using the find_learning_rate function implemented at line 74 of debug_neural_nets.py. This utility exponentially increases the learning rate while tracking loss values to identify the steepest descent point.

The suggested learning rate is typically 10× smaller than the rate where loss begins to rise, providing a data-driven starting point that prevents both divergence and stagnation.

from debug_neural_nets import find_learning_rate

x_full = torch.randn(64, 10)
y_full = (x_full[:, 0] > 0).long()

find_learning_rate(model, x_full, y_full, criterion)

Validate Gradients with Numerical Checking

When backpropagation yields suspicious results, verify your analytical gradients against numerical finite-difference estimates using the gradient_check function at line 35 of debug_neural_nets.py. Large relative differences exceeding 1e-3 indicate bugs in your custom backward pass implementations.

This verification step is critical when extending lessons with custom loss functions or architectural modifications.

from debug_neural_nets import gradient_check

reg_model = nn.Sequential(nn.Linear(3, 4), nn.Tanh(), nn.Linear(4, 1))
x_reg = torch.randn(4, 3)
y_reg = torch.randn(4, 1)

gradient_check(reg_model, x_reg, y_reg, nn.MSELoss())

Follow the 7-Step Pre-Training Checklist

The lesson documentation in phases/03-deep-learning-core/13-debugging-neural-networks/docs/en.md prescribes a rigorous 7-step checklist before scaling to full datasets:

1. Run the overfit-one-batch test to verify basic trainability.
2. Print a complete model summary to verify layer dimensions.
3. Execute a single forward pass to catch shape mismatches.
4. Train for a few epochs while monitoring loss via NetworkDebugger.
5. Inspect activation and gradient statistics for dead neurons or vanishing gradients.
6. Validate the entire data pipeline for corrupted samples or incorrect labels.
7. Launch the full training loop only after all preceding checks pass.

Summary

• Start small: Use overfit_one_batch to verify your model can memorize a single batch before scaling up.
• Instrument everything: Deploy NetworkDebugger hooks to monitor activation and gradient health in real-time.
• Check loss states: Watch for NAN_OR_INF, NOT_DECREASING, or OSCILLATING patterns that indicate specific hyperparameter or architecture faults.
• Validate numerically: Use gradient_check when implementing custom layers to ensure backpropagation correctness.
• Follow the checklist: Adhere to the 7-step pre-training protocol documented in en.md to systematically debug a lesson that fails to run or produces unexpected results.

Frequently Asked Questions

What is the first step when a lesson crashes silently?

Run the overfit_one_batch test from debug_neural_nets.py on a tiny batch of 8–32 samples. If the model cannot achieve near-zero loss and 100% accuracy on this trivial task, the bug lies in your model definition, loss function, or training loop architecture rather than the dataset size or hyperparameters.

How do I identify dead neurons in my network?

Initialize the NetworkDebugger class and train for 10–20 steps, then call print_report(). The debugger automatically flags DEAD_NEURONS when more than 50% of a layer's activations equal zero, typically indicating ReLU saturation that blocks gradient flow through the network.

What learning rate should I start with when debugging?

Use the find_learning_rate function to perform an exponential sweep of learning rates while tracking loss. Select a rate approximately 10× smaller than the point where loss begins to increase, as this steepest-descent point provides a robust initial learning rate that minimizes oscillation risks.

How can I verify my backpropagation code is correct?

Apply the gradient_check function to compare your analytical gradients (from autograd) against numerical finite-difference estimates. Relative differences exceeding 1e-3 indicate implementation bugs in your custom backward pass that you must resolve before proceeding with training.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}