How AI-Engineering-From-Scratch Teaches Deep Learning Fundamentals: Backpropagation, Optimizers, and Regularization

The AI-Engineering-From-Scratch curriculum teaches deep learning fundamentals by building a complete training pipeline from scratch, starting with a single-layer perceptron and progressing through manual backpropagation engines, custom optimizers like AdamW, and regularization techniques including dropout and normalization.

The rohitg00/ai-engineering-from-scratch repository provides a self-contained deep learning curriculum that implements every core concept without relying on external ML libraries. By working through Phase 03 (Deep Learning Core), you construct a mini-framework that demonstrates exactly how backpropagation, optimizers, and regularization function under the hood, giving you intuition before switching to production frameworks.

Backpropagation: Building the Gradient Engine from Scratch

The Perceptron and Multi-Layer Networks

The curriculum begins in phases/03-deep-learning-core/01-the-perceptron/docs/en.md by hand-wiring a two-layer network to introduce the limitations of fixed logic and the necessity of learning. You then progress to phases/03-deep-learning-core/02-multi-layer-networks/docs/en.md to stack layers and prepare the computational graph for gradient computation.

Reverse-Mode Autograd Implementation

In phases/03-deep-learning-core/03-backpropagation/docs/en.md, you implement a full reverse-mode automatic differentiation engine using a Value class that tracks operations and gradients. The implementation uses topological sorting to ensure gradients propagate correctly through the computational graph:

class Value:
    def __init__(self, data, children=(), op=''):
        self.data = data
        self.grad = 0.0
        self._backward = lambda: None
        self._children = set(children)
        self._op = op

    def __add__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data + other.data, (self, other), '+')
        def _backward():
            self.grad += out.grad
            other.grad += out.grad
        out._backward = _backward
        return out

    def __mul__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data * other.data, (self, other), '*')
        def _backward():
            self.grad += other.data * out.grad
            other.grad += self.data * out.grad
        out._backward = _backward
        return out

    def sigmoid(self):
        s = 1.0 / (1.0 + math.exp(-self.data))
        out = Value(s, (self,), 'sigmoid')
        def _backward():
            self.grad += (s * (1 - s)) * out.grad
        out._backward = _backward
        return out

    def backward(self):
        topo = []
        visited = set()
        def build_topo(v):
            if v not in visited:
                visited.add(v)
                for child in v._children:
                    build_topo(child)
                topo.append(v)
        build_topo(self)
        self.grad = 1.0
        for v in reversed(topo):
            v._backward()

This engine computes ∂L/∂w for every weight by applying the chain rule through the topologically sorted graph, enabling you to train XOR and circle classification models without external dependencies.

Optimizers: From SGD to AdamW

SGD with Momentum

The curriculum in phases/03-deep-learning-core/06-optimizers/docs/en.md begins with vanilla Stochastic Gradient Descent (SGD), then adds momentum to dampen oscillations and accelerate convergence in relevant directions. You implement the velocity update equations manually to see how exponentially weighted averages smooth the optimization trajectory.

Adam and AdamW Decoupled Weight Decay

The lessons progress to adaptive optimizers, implementing RMSProp, Adam, and finally AdamW with decoupled weight decay. The implementation includes bias correction for the first and second moment estimates:

class AdamW:
    def __init__(self, lr=0.001, beta1=0.9, beta2=0.999,
                 epsilon=1e-8, weight_decay=0.01):
        self.lr = lr
        self.beta1 = beta1
        self.beta2 = beta2
        self.epsilon = epsilon
        self.weight_decay = weight_decay
        self.m = None
        self.v = None
        self.t = 0

    def step(self, params, grads):
        if self.m is None:
            self.m = [0.0] * len(params)
            self.v = [0.0] * len(params)

        self.t += 1
        for i in range(len(params)):
            # Update biased first and second moment estimates

            self.m[i] = self.beta1 * self.m[i] + (1 - self.beta1) * grads[i]
            self.v[i] = self.beta2 * self.v[i] + (1 - self.beta2) * grads[i] ** 2

            # Compute bias-corrected estimates

            m_hat = self.m[i] / (1 - self.beta1 ** self.t)
            v_hat = self.v[i] / (1 - self.beta2 ** self.t)

            # Update parameters with adaptive learning rate

            params[i] -= self.lr * m_hat / (math.sqrt(v_hat) + self.epsilon)
            # Decoupled weight decay (AdamW)

            params[i] -= self.lr * self.weight_decay * params[i]

The curriculum explicitly contrasts Adam (L2 regularization added to gradients) with AdamW (decoupled weight decay applied directly to parameters), explaining why the latter improves generalization.

Regularization Techniques for Generalization

Dropout and L2 Weight Decay

In phases/03-deep-learning-core/07-regularization/code/main.py, you implement Dropout as a stochastic layer that randomly zeroes neurons during training while scaling activations by 1/(1-p) to maintain expected values:

class Dropout:
    def __init__(self, p=0.5):
        self.p = p
        self.training = True
        self.mask = None

    def forward(self, x):
        if not self.training:
            return list(x)
        self.mask = []
        out = []
        for v in x:
            if random.random() < self.p:
                self.mask.append(0)
                out.append(0.0)
            else:
                self.mask.append(1)
                out.append(v / (1 - self.p))
        return out

    def backward(self, grad_out):
        grads = []
        for g, m in zip(grad_out, self.mask):
            grads.append(g / (1 - self.p) if m else 0.0)
        return grads

You also implement L2 weight decay (ridge regularization) directly in the optimizer step, penalizing large weights to reduce model complexity and prevent overfitting.

Normalization Layers: BatchNorm, LayerNorm, and RMSNorm

The regularization module covers Batch Normalization, Layer Normalization, and RMSNorm, comparing their effects on internal covariate shift and train-test gaps. You implement the forward and backward passes for each, learning how normalization acts as a regularizer by adding noise to activations and stabilizing gradients.

Learning Rate Schedules as Implicit Regularizers

The curriculum in phases/03-deep-learning-core/09-learning-rate-schedules/docs/en.md treats learning rate schedules as implicit regularization techniques. You implement warm-up phases and cosine decay, observing how high initial learning rates act as regularizers by allowing the optimizer to explore wider regions of the loss landscape before settling into minima.

The Mini-Framework: Wiring Everything Together

The final lessons in phases/03-deep-learning-core/10-mini-framework/docs/en.md integrate all components into a tiny training loop that mimics PyTorch's API. You train on synthetic circle datasets, visualizing overfitting and underfitting while applying the regularization techniques learned earlier. This demonstrates how backpropagation, optimizers, and regularization interact in a complete deep learning pipeline.

Summary

• Backpropagation: The curriculum implements a reverse-mode autograd engine in phases/03-deep-learning-core/03-backpropagation/docs/en.md using topological sorting to compute gradients through computational graphs.
• Optimizers: You build SGD, momentum, Adam, and AdamW from scratch in phases/03-deep-learning-core/06-optimizers/docs/en.md, including bias correction and decoupled weight decay.
• Regularization: Dropout, L2 penalty, and normalization layers (BatchNorm, LayerNorm, RMSNorm) are implemented in phases/03-deep-learning-core/07-regularization/code/main.py to reduce overfitting and improve generalization.
• Integration: All components combine into a self-contained mini-framework that trains models without external ML libraries, providing foundational understanding of deep learning fundamentals.

Frequently Asked Questions

Does this curriculum use PyTorch or TensorFlow?

No, the AI-Engineering-From-Scratch curriculum implements all deep learning fundamentals using pure Python and NumPy. The repository explicitly avoids external ML frameworks in Phase 03 to ensure you understand the mathematical mechanics of backpropagation, optimizers, and regularization before switching to production tools.

How does the backpropagation implementation compare to PyTorch's autograd?

The curriculum's Value class implements reverse-mode automatic differentiation similarly to PyTorch's autograd, using a topological sort to ensure gradients flow backward through the computational graph in the correct order. However, it is a minimal educational implementation that handles scalar values and basic operations (addition, multiplication, sigmoid) to demonstrate the chain rule mechanics explicitly.

Why does the curriculum implement AdamW instead of standard Adam?

The curriculum implements both Adam and AdamW to demonstrate the difference between coupled and decoupled weight decay. AdamW applies weight decay directly to the parameters rather than to the gradients, which improves generalization and separates the regularization hyperparameter from the learning rate. This implementation appears in phases/03-deep-learning-core/06-optimizers/docs/en.md with full mathematical derivations.

What regularization techniques are implemented from scratch?

The curriculum implements Dropout (random neuron zeroing during training), L2 weight decay (ridge regularization in the optimizer), Batch Normalization, Layer Normalization, and RMSNorm. These are all contained in phases/03-deep-learning-core/07-regularization/code/main.py, allowing you to compare their effects on train-test gaps using synthetic datasets.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}