Debugging Neural Network Training: How to Detect Overfitting vs Underfitting

The most reliable way to debug neural network training is to plot training versus validation metrics after each epoch: a growing gap between them signals overfitting, while both remaining high indicates underfitting.

Debugging neural network training requires understanding the balance between overfitting (memorizing training data) and underfitting (failing to learn underlying patterns). The microsoft/AI-For-Beginners repository provides hands-on labs and reference implementations that demonstrate exactly how to spot these issues using validation curves, model capacity analysis, and regularization techniques. By examining the codebase—from the simple perceptron in examples/02-simple-neural-network.py to the convolutional labs in lessons/4-ComputerVision/07-ConvNets/—you can master the diagnostics that separate successful models from failed experiments.

Detecting Overfitting and Underfitting Through Validation Curves

The primary diagnostic tool in the AI-For-Beginners curriculum is the train-validation gap. Every major lab notebook emphasizes collecting accuracy or loss values for both datasets at each epoch to visualize model behavior.

Reading the Gap Between Training and Validation Loss

According to the PetFaces lab notebook (lessons/4-ComputerVision/07-ConvNets/lab/PetFaces.ipynb), you should collect accuracy on train and test data at each epoch and plot the results to see whether overfitting occurs. The interpretation follows two distinct patterns:

• Overfitting: Training loss continues decreasing while validation loss begins increasing (or validation accuracy plateaus significantly below training accuracy). This divergence indicates the model is memorizing noise rather than generalizing.
• Underfitting: Both training and validation loss remain high and track closely together, suggesting the model lacks sufficient capacity to capture the underlying data distribution.

Visual Patterns That Indicate Model Health

The IntroKeras notebook (lessons/3-NeuralNetworks/05-Frameworks/IntroKeras.ipynb) warns that a learning rate that is too high can lead to overfitting, while too low means more epochs are needed. This creates a diagnostic triangle:

1. Oscillating validation loss → Learning rate is likely too high, causing the optimizer to bounce out of good minima.
2. Slowly decreasing parallel curves → Model is underfitting; increase capacity or learning rate.
3. Diverging curves after epoch N -> Classic overfitting; apply regularization or early stopping.

Code Implementation: Monitoring Training in Keras and PyTorch

The repository provides framework-specific implementations for tracking these metrics. Below are production-ready patterns derived directly from the codebase.

Plotting Accuracy Curves with TensorFlow/Keras

The LabFrameworks notebook (lessons/3-NeuralNetworks/05-Frameworks/lab/LabFrameworks.ipynb) demonstrates using Keras's built-in validation_split parameter to automate debugging:

import tensorflow as tf
import matplotlib.pyplot as plt

# Load MNIST and normalize

(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()
x_train = x_train[..., None] / 255.0
x_test = x_test[..., None] / 255.0

model = tf.keras.Sequential([
    tf.keras.layers.Conv2D(32, 3, activation='relu', input_shape=(28, 28, 1)),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(10, activation='softmax')
])

model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

# Reserve 20% of training data for validation

history = model.fit(x_train, y_train,
                    epochs=15,
                    batch_size=64,
                    validation_split=0.2,
                    verbose=0)

# Visual debugging: plot the divergence

plt.figure(figsize=(6, 4))
plt.plot(history.history['accuracy'], label='train acc')
plt.plot(history.history['val_accuracy'], label='val acc')
plt.title('Training vs. Validation Accuracy')
plt.xlabel('Epoch')
plt.ylabel('Accuracy')
plt.legend()
plt.show()

Diagnostic rule: If the training line climbs well above the validation line, you are witnessing overfitting. If both lines plateau at low accuracy, the model is underfitting.

Implementing Early Stopping in PyTorch

For PyTorch implementations, the GenerativePyTorch notebook approach (lessons/5-NLP/17-GenerativeNetworks/GenerativePyTorch.ipynb) can be adapted to implement early stopping—halting training when validation performance degrades:

import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import random_split, DataLoader
from torchvision import datasets, transforms
import matplotlib.pyplot as plt

transform = transforms.Compose([transforms.ToTensor()])
full = datasets.FashionMNIST(root='.', download=True, transform=transform)
train_len = int(0.8 * len(full))
val_len = len(full) - train_len
train_set, val_set = random_split(full, [train_len, val_len])

train_loader = DataLoader(train_set, batch_size=64, shuffle=True)
val_loader = DataLoader(val_set, batch_size=64)

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc = nn.Sequential(
            nn.Flatten(),
            nn.Linear(28*28, 128),
            nn.ReLU(),
            nn.Linear(128, 10))
    
    def forward(self, x):
        return self.fc(x)

model = Net()
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(model.parameters(), lr=1e-3)

train_acc, val_acc = [], []
patience, best_val, wait = 3, 0.0, 0

for epoch in range(20):
    # Training phase

    model.train()
    for X, y in train_loader:
        optimizer.zero_grad()
        loss = criterion(model(X), y)
        loss.backward()
        optimizer.step()
    
    # Validation phase

    model.eval()
    correct = total = 0
    with torch.no_grad():
        for X, y in val_loader:
            preds = model(X).argmax(dim=1)
            correct += (preds == y).sum().item()
            total += y.size(0)
    
    val = correct / total
    val_acc.append(val)
    
    # Early stopping logic

    if val > best_val:
        best_val = val
        wait = 0
    else:
        wait += 1
    
    if wait >= patience:
        print(f'Stopping early at epoch {epoch+1} to prevent overfitting')
        break

plt.plot(val_acc, label='validation accuracy')
plt.title('Validation Accuracy Over Epochs')
plt.xlabel('Epoch')
plt.ylabel('Accuracy')
plt.legend()
plt.show()

This pattern automatically guards against overfitting by monitoring validation accuracy trends.

Architectural Factors That Drive Overfitting

Beyond plotting curves, the AI-For-Beginners repository teaches debugging through model architecture analysis—examining parameter counts, layer sizes, and capacity relative to dataset size.

Model Capacity and Parameter Count

The ConvNetsPyTorch notebook (lessons/4-ComputerVision/07-ConvNets/ConvNetsPyTorch.ipynb) explicitly links parameter count to overfitting risk. At lines 329-331, the curriculum notes that keeping trainable parameters around 8.5K helps prevent overfitting even on small datasets.

Key insight: Too many parameters relative to data volume creates memorization capacity; too few parameters restricts the model to underfitting. When debugging, check your parameter count using sum(p.numel() for p in model.parameters() if p.requires_grad) in PyTorch or model.count_params() in Keras.

Learning Rate Sensitivity

As noted in lessons/3-NeuralNetworks/05-Frameworks/IntroKeras.ipynb (lines 390-392), learning rate acts as a hidden overfitting trigger. Rates that are too high can cause the loss landscape to bounce, creating instability that resembles overfitting, while rates that are too low may never escape high-loss regions, mimicking underfitting.

Hidden Layer Size in Recurrent Networks

The GenerativePyTorch notebook (lessons/5-NLP/17-GenerativeNetworks/GenerativePyTorch.ipynb, lines 303-305) provides a specific warning: "A hidden layer that is too large can cause over-fitting; a smaller one may not produce good results." This applies broadly—when debugging RNNs or dense networks, treat hidden size as a primary hyperparameter for controlling the bias-variance tradeoff.

Hands-On Debugging with the Simple Neural Network Example

For understanding underfitting at its most fundamental level, the repository includes examples/02-simple-neural-network.py—a scratch-built perceptron without hidden layers.

Visualizing Underfitting Through Decision Boundaries

A single neuron can only learn linear decision boundaries. When applied to non-linear data distributions (like the binary classification task y > x used in the example), this creates visible underfitting:

import numpy as np
import matplotlib.pyplot as plt
from examples_02_simple_neural_network import SimpleNeuron, generate_training_data

# Initialize single-layer model

neuron = SimpleNeuron(num_inputs=2)
data = generate_training_data(200)

# Train for minimal epochs

for _ in range(200):
    for inputs, target in data:
        neuron.feedforward(inputs)
        neuron.train(inputs, target)

# Generate decision boundary visualization

x_vals = np.linspace(0, 10, 100)
y_vals = np.linspace(0, 10, 100)
X, Y = np.meshgrid(x_vals, y_vals)
Z = np.array([neuron.feedforward([x, y]) for x, y in zip(X.ravel(), Y.ravel())])
Z = Z.reshape(X.shape)

plt.contourf(X, Y, Z, levels=[0, 0.5, 1], alpha=0.3, colors=['red', 'green'])
plt.title('Decision Boundary of a Single Neuron (Underfitting)')
plt.xlabel('x')
plt.ylabel('y')
plt.show()

If the boundary remains a straight line despite the true distribution being non-linear, you have confirmed underfitting due to insufficient model complexity. The solution demonstrated in subsequent notebooks is adding hidden layers to increase representational capacity.

Summary

• Always maintain a validation split (20% is standard) when calling model.fit() or constructing DataLoaders to enable overfitting detection.
• Plot training versus validation curves after every training run; divergence indicates overfitting, parallel high lines indicate underfitting.
• Monitor parameter counts as implemented in ConvNetsPyTorch.ipynb—keep model capacity proportional to dataset size.
• Apply early stopping when validation metrics plateau for more than 2-3 consecutive epochs.
• Adjust hidden layer sizes and learning rates first when debugging, as these are the primary levers for the bias-variance tradeoff demonstrated across the curriculum.

Frequently Asked Questions

What is the clearest sign of overfitting in training logs?

The clearest sign is when training accuracy continues to rise while validation accuracy decreases or plateaus after a certain epoch. As shown in PetFaces.ipynb, this divergence indicates the model is memorizing training noise rather than learning generalizable features.

How do I fix underfitting when both training and validation loss are high?

Increase model capacity by adding layers or neurons, train for more epochs, or raise the learning rate. The 02-simple-neural-network.py example demonstrates that a single neuron underfits non-linear data; adding hidden layers (as shown in the OwnFramework notebooks) resolves this by increasing representational power.

Can a model suffer from both overfitting and underfitting?

Yes, sequentially. A model may initially underfit (both losses high), then overfit (training loss drops while validation rises) if trained too long without regularization. Early stopping implementations, like those in the PyTorch examples, prevent this progression by halting at the optimal intermediate point.

Where does the AI-For-Beginners repo demonstrate these concepts?

The repository teaches debugging across three key locations: lessons/3-NeuralNetworks/05-Frameworks/ for basic Keras monitoring, lessons/4-ComputerVision/07-ConvNets/ for computer vision specific overfitting analysis, and lessons/5-NLP/17-GenerativeNetworks/ for RNN capacity tuning, with the foundational perceptron logic in examples/02-simple-neural-network.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}