# How to Debug a Lesson That Fails to Run or Produces Unexpected Results: A Systematic Framework

> Effectively debug AI lessons that fail or yield unexpected results. Apply the overfit-one-batch test, use NetworkDebugger, and monitor loss and gradients for faster debugging.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: how-to-guide
- Published: 2026-08-29

---

**Apply the "overfit-one-batch" test first, then use the `NetworkDebugger` class to monitor per-layer activations and gradients, check loss health for NaN or oscillation patterns, and validate gradients numerically before scaling to full training.**

When you encounter silent crashes or puzzling outputs while working through the *ai-engineering-from-scratch* curriculum, the Debugging Neural Networks lesson (Phase 3, Lesson 13) provides a battle-tested framework to isolate root causes efficiently. This guide walks you through the exact diagnostic workflow implemented in [`debug_neural_nets.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/debug_neural_nets.py) to help you debug a lesson that fails to run or produces unexpected results without wasting compute on doomed full-scale runs.

## Start with the Overfit-One-Batch Test

The fastest way to verify your model's fundamental integrity is to **overfit a single tiny batch**. This test exposes bugs in the model definition, loss function, or training loop before you commit significant resources.

According to the source code in [`phases/03-deep-learning-core/13-debugging-neural-networks/code/debug_neural_nets.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/13-debugging-neural-networks/code/debug_neural_nets.py), the `overfit_one_batch` function (line 40) forces the model to memorize 8–32 samples across many optimization steps. If the loss does not approach approximately zero and accuracy fails to reach 100%, your architecture contains a critical flaw requiring immediate repair.

```python
import torch
import torch.nn as nn
from debug_neural_nets import overfit_one_batch

# Minimal reproducible test

model = nn.Sequential(nn.Linear(10, 32), nn.ReLU(), nn.Linear(32, 2))
criterion = nn.CrossEntropyLoss()
x_batch = torch.randn(8, 10)
y_batch = (x_batch[:, 0] > 0).long()

overfit_one_batch(model, x_batch, y_batch, criterion)

```

## Instrument Your Model with NetworkDebugger

Once the basic sanity check passes, layer the **NetworkDebugger** class over your model to capture per-layer statistics during live training. This hook-based monitor records activation statistics (mean, standard deviation, zero-fraction) and gradient statistics (mean, std, absolute-mean) without modifying your forward pass logic.

After running a few training steps, invoking `print_report()` generates a concise health summary that identifies dead ReLUs, exploding activations, and vanishing gradients. The class definition resides at line 7 of [`debug_neural_nets.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/debug_neural_nets.py).

```python
from debug_neural_nets import NetworkDebugger

debugger = NetworkDebugger(model)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

for step in range(20):
    optimizer.zero_grad()
    out = model(x_batch)
    loss = criterion(out, y_batch)
    debugger.record_loss(loss.item())
    loss.backward()
    optimizer.step()

debugger.print_report()   # prints health, activation & gradient alerts

debugger.remove_hooks()

```

## Diagnose Loss Health Issues

The `NetworkDebugger.check_loss_health()` method automatically flags three common failure modes that occur when you debug a lesson that fails to run or produces unexpected results:

- **NAN_OR_INF**: Indicates NaN or Inf values in the loss, typically caused by a learning rate that is too high or log-of-zero operations in the loss computation.
- **NOT_DECREASING**: Signals loss plateaus, suggesting a learning rate that is too low or severe underfitting requiring architecture adjustments.
- **OSCILLATING**: Reveals wild loss swings characteristic of an excessively high learning rate causing optimization instability.

## Inspect Activation and Gradient Diagnostics

Beyond loss curves, inspect the internal state of your network using the debugger's threshold-based alerts defined in the source:

- **Dead Neurons**: Flagged when over 50% of activations equal zero (`DEAD_NEURONS`), indicating dead ReLU syndrome that blocks gradient flow.
- **Exploding Activations**: Reported when mean magnitude exceeds 10 (`EXPLODING_ACTIVATIONS`), suggesting initialization or normalization issues.
- **Vanishing Gradients**: Identified when the absolute-mean gradient falls below 1e-7 (`VANISHING_GRADIENT`), signaling that earlier layers are not learning.

These diagnostics pinpoint exactly which layer is destabilizing your training without requiring manual tensor inspection across deep architectures.

## Optimize Hyperparameters with Learning-Rate Finder

Before launching full training, determine the optimal learning rate using the `find_learning_rate` function implemented at line 74 of [`debug_neural_nets.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/debug_neural_nets.py). This utility exponentially increases the learning rate while tracking loss values to identify the steepest descent point.

The suggested learning rate is typically 10× smaller than the rate where loss begins to rise, providing a data-driven starting point that prevents both divergence and stagnation.

```python
from debug_neural_nets import find_learning_rate

x_full = torch.randn(64, 10)
y_full = (x_full[:, 0] > 0).long()

find_learning_rate(model, x_full, y_full, criterion)

```

## Validate Gradients with Numerical Checking

When backpropagation yields suspicious results, verify your analytical gradients against numerical finite-difference estimates using the `gradient_check` function at line 35 of [`debug_neural_nets.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/debug_neural_nets.py). Large relative differences exceeding 1e-3 indicate bugs in your custom backward pass implementations.

This verification step is critical when extending lessons with custom loss functions or architectural modifications.

```python
from debug_neural_nets import gradient_check

reg_model = nn.Sequential(nn.Linear(3, 4), nn.Tanh(), nn.Linear(4, 1))
x_reg = torch.randn(4, 3)
y_reg = torch.randn(4, 1)

gradient_check(reg_model, x_reg, y_reg, nn.MSELoss())

```

## Follow the 7-Step Pre-Training Checklist

The lesson documentation in [`phases/03-deep-learning-core/13-debugging-neural-networks/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/13-debugging-neural-networks/docs/en.md) prescribes a rigorous 7-step checklist before scaling to full datasets:

1. Run the overfit-one-batch test to verify basic trainability.
2. Print a complete model summary to verify layer dimensions.
3. Execute a single forward pass to catch shape mismatches.
4. Train for a few epochs while monitoring loss via `NetworkDebugger`.
5. Inspect activation and gradient statistics for dead neurons or vanishing gradients.
6. Validate the entire data pipeline for corrupted samples or incorrect labels.
7. Launch the full training loop only after all preceding checks pass.

## Summary

- **Start small**: Use `overfit_one_batch` to verify your model can memorize a single batch before scaling up.
- **Instrument everything**: Deploy `NetworkDebugger` hooks to monitor activation and gradient health in real-time.
- **Check loss states**: Watch for `NAN_OR_INF`, `NOT_DECREASING`, or `OSCILLATING` patterns that indicate specific hyperparameter or architecture faults.
- **Validate numerically**: Use `gradient_check` when implementing custom layers to ensure backpropagation correctness.
- **Follow the checklist**: Adhere to the 7-step pre-training protocol documented in [`en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/en.md) to systematically debug a lesson that fails to run or produces unexpected results.

## Frequently Asked Questions

### What is the first step when a lesson crashes silently?

Run the `overfit_one_batch` test from [`debug_neural_nets.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/debug_neural_nets.py) on a tiny batch of 8–32 samples. If the model cannot achieve near-zero loss and 100% accuracy on this trivial task, the bug lies in your model definition, loss function, or training loop architecture rather than the dataset size or hyperparameters.

### How do I identify dead neurons in my network?

Initialize the `NetworkDebugger` class and train for 10–20 steps, then call `print_report()`. The debugger automatically flags `DEAD_NEURONS` when more than 50% of a layer's activations equal zero, typically indicating ReLU saturation that blocks gradient flow through the network.

### What learning rate should I start with when debugging?

Use the `find_learning_rate` function to perform an exponential sweep of learning rates while tracking loss. Select a rate approximately 10× smaller than the point where loss begins to increase, as this steepest-descent point provides a robust initial learning rate that minimizes oscillation risks.

### How can I verify my backpropagation code is correct?

Apply the `gradient_check` function to compare your analytical gradients (from autograd) against numerical finite-difference estimates. Relative differences exceeding 1e-3 indicate implementation bugs in your custom backward pass that you must resolve before proceeding with training.