# How AI-Engineering-From-Scratch Teaches Deep Learning Fundamentals: Backpropagation, Optimizers, and Regularization

> Build deep learning fundamentals from scratch with AI-Engineering-From-Scratch. Learn manual backpropagation, custom optimizers like AdamW, and regularization techniques.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: tutorial
- Published: 2026-06-18

---

**The AI-Engineering-From-Scratch curriculum teaches deep learning fundamentals by building a complete training pipeline from scratch, starting with a single-layer perceptron and progressing through manual backpropagation engines, custom optimizers like AdamW, and regularization techniques including dropout and normalization.**

The **rohitg00/ai-engineering-from-scratch** repository provides a self-contained deep learning curriculum that implements every core concept without relying on external ML libraries. By working through Phase 03 (Deep Learning Core), you construct a mini-framework that demonstrates exactly how backpropagation, optimizers, and regularization function under the hood, giving you intuition before switching to production frameworks.

## Backpropagation: Building the Gradient Engine from Scratch

### The Perceptron and Multi-Layer Networks

The curriculum begins in [`phases/03-deep-learning-core/01-the-perceptron/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/01-the-perceptron/docs/en.md) by hand-wiring a two-layer network to introduce the limitations of fixed logic and the necessity of learning. You then progress to [`phases/03-deep-learning-core/02-multi-layer-networks/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/02-multi-layer-networks/docs/en.md) to stack layers and prepare the computational graph for gradient computation.

### Reverse-Mode Autograd Implementation

In [`phases/03-deep-learning-core/03-backpropagation/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/03-backpropagation/docs/en.md), you implement a full **reverse-mode automatic differentiation** engine using a `Value` class that tracks operations and gradients. The implementation uses topological sorting to ensure gradients propagate correctly through the computational graph:

```python
class Value:
    def __init__(self, data, children=(), op=''):
        self.data = data
        self.grad = 0.0
        self._backward = lambda: None
        self._children = set(children)
        self._op = op

    def __add__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data + other.data, (self, other), '+')
        def _backward():
            self.grad += out.grad
            other.grad += out.grad
        out._backward = _backward
        return out

    def __mul__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data * other.data, (self, other), '*')
        def _backward():
            self.grad += other.data * out.grad
            other.grad += self.data * out.grad
        out._backward = _backward
        return out

    def sigmoid(self):
        s = 1.0 / (1.0 + math.exp(-self.data))
        out = Value(s, (self,), 'sigmoid')
        def _backward():
            self.grad += (s * (1 - s)) * out.grad
        out._backward = _backward
        return out

    def backward(self):
        topo = []
        visited = set()
        def build_topo(v):
            if v not in visited:
                visited.add(v)
                for child in v._children:
                    build_topo(child)
                topo.append(v)
        build_topo(self)
        self.grad = 1.0
        for v in reversed(topo):
            v._backward()

```

This engine computes `∂L/∂w` for every weight by applying the chain rule through the topologically sorted graph, enabling you to train XOR and circle classification models without external dependencies.

## Optimizers: From SGD to AdamW

### SGD with Momentum

The curriculum in [`phases/03-deep-learning-core/06-optimizers/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/06-optimizers/docs/en.md) begins with vanilla Stochastic Gradient Descent (SGD), then adds momentum to dampen oscillations and accelerate convergence in relevant directions. You implement the velocity update equations manually to see how exponentially weighted averages smooth the optimization trajectory.

### Adam and AdamW Decoupled Weight Decay

The lessons progress to **adaptive optimizers**, implementing RMSProp, Adam, and finally **AdamW** with decoupled weight decay. The implementation includes bias correction for the first and second moment estimates:

```python
class AdamW:
    def __init__(self, lr=0.001, beta1=0.9, beta2=0.999,
                 epsilon=1e-8, weight_decay=0.01):
        self.lr = lr
        self.beta1 = beta1
        self.beta2 = beta2
        self.epsilon = epsilon
        self.weight_decay = weight_decay
        self.m = None
        self.v = None
        self.t = 0

    def step(self, params, grads):
        if self.m is None:
            self.m = [0.0] * len(params)
            self.v = [0.0] * len(params)

        self.t += 1
        for i in range(len(params)):
            # Update biased first and second moment estimates

            self.m[i] = self.beta1 * self.m[i] + (1 - self.beta1) * grads[i]
            self.v[i] = self.beta2 * self.v[i] + (1 - self.beta2) * grads[i] ** 2

            # Compute bias-corrected estimates

            m_hat = self.m[i] / (1 - self.beta1 ** self.t)
            v_hat = self.v[i] / (1 - self.beta2 ** self.t)

            # Update parameters with adaptive learning rate

            params[i] -= self.lr * m_hat / (math.sqrt(v_hat) + self.epsilon)
            # Decoupled weight decay (AdamW)

            params[i] -= self.lr * self.weight_decay * params[i]

```

The curriculum explicitly contrasts **Adam** (L2 regularization added to gradients) with **AdamW** (decoupled weight decay applied directly to parameters), explaining why the latter improves generalization.

## Regularization Techniques for Generalization

### Dropout and L2 Weight Decay

In [`phases/03-deep-learning-core/07-regularization/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/07-regularization/code/main.py), you implement **Dropout** as a stochastic layer that randomly zeroes neurons during training while scaling activations by `1/(1-p)` to maintain expected values:

```python
class Dropout:
    def __init__(self, p=0.5):
        self.p = p
        self.training = True
        self.mask = None

    def forward(self, x):
        if not self.training:
            return list(x)
        self.mask = []
        out = []
        for v in x:
            if random.random() < self.p:
                self.mask.append(0)
                out.append(0.0)
            else:
                self.mask.append(1)
                out.append(v / (1 - self.p))
        return out

    def backward(self, grad_out):
        grads = []
        for g, m in zip(grad_out, self.mask):
            grads.append(g / (1 - self.p) if m else 0.0)
        return grads

```

You also implement **L2 weight decay** (ridge regularization) directly in the optimizer step, penalizing large weights to reduce model complexity and prevent overfitting.

### Normalization Layers: BatchNorm, LayerNorm, and RMSNorm

The regularization module covers **Batch Normalization**, **Layer Normalization**, and **RMSNorm**, comparing their effects on internal covariate shift and train-test gaps. You implement the forward and backward passes for each, learning how normalization acts as a regularizer by adding noise to activations and stabilizing gradients.

## Learning Rate Schedules as Implicit Regularizers

The curriculum in [`phases/03-deep-learning-core/09-learning-rate-schedules/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/09-learning-rate-schedules/docs/en.md) treats **learning rate schedules** as implicit regularization techniques. You implement warm-up phases and cosine decay, observing how high initial learning rates act as regularizers by allowing the optimizer to explore wider regions of the loss landscape before settling into minima.

## The Mini-Framework: Wiring Everything Together

The final lessons in [`phases/03-deep-learning-core/10-mini-framework/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/10-mini-framework/docs/en.md) integrate all components into a tiny training loop that mimics PyTorch's API. You train on synthetic circle datasets, visualizing overfitting and underfitting while applying the regularization techniques learned earlier. This demonstrates how backpropagation, optimizers, and regularization interact in a complete deep learning pipeline.

## Summary

- **Backpropagation**: The curriculum implements a reverse-mode autograd engine in [`phases/03-deep-learning-core/03-backpropagation/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/03-backpropagation/docs/en.md) using topological sorting to compute gradients through computational graphs.
- **Optimizers**: You build SGD, momentum, Adam, and AdamW from scratch in [`phases/03-deep-learning-core/06-optimizers/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/06-optimizers/docs/en.md), including bias correction and decoupled weight decay.
- **Regularization**: Dropout, L2 penalty, and normalization layers (BatchNorm, LayerNorm, RMSNorm) are implemented in [`phases/03-deep-learning-core/07-regularization/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/07-regularization/code/main.py) to reduce overfitting and improve generalization.
- **Integration**: All components combine into a self-contained mini-framework that trains models without external ML libraries, providing foundational understanding of deep learning fundamentals.

## Frequently Asked Questions

### Does this curriculum use PyTorch or TensorFlow?

No, the **AI-Engineering-From-Scratch** curriculum implements all deep learning fundamentals using pure Python and NumPy. The repository explicitly avoids external ML frameworks in Phase 03 to ensure you understand the mathematical mechanics of backpropagation, optimizers, and regularization before switching to production tools.

### How does the backpropagation implementation compare to PyTorch's autograd?

The curriculum's `Value` class implements **reverse-mode automatic differentiation** similarly to PyTorch's autograd, using a topological sort to ensure gradients flow backward through the computational graph in the correct order. However, it is a minimal educational implementation that handles scalar values and basic operations (addition, multiplication, sigmoid) to demonstrate the chain rule mechanics explicitly.

### Why does the curriculum implement AdamW instead of standard Adam?

The curriculum implements both Adam and AdamW to demonstrate the difference between **coupled** and **decoupled** weight decay. AdamW applies weight decay directly to the parameters rather than to the gradients, which improves generalization and separates the regularization hyperparameter from the learning rate. This implementation appears in [`phases/03-deep-learning-core/06-optimizers/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/06-optimizers/docs/en.md) with full mathematical derivations.

### What regularization techniques are implemented from scratch?

The curriculum implements **Dropout** (random neuron zeroing during training), **L2 weight decay** (ridge regularization in the optimizer), **Batch Normalization**, **Layer Normalization**, and **RMSNorm**. These are all contained in [`phases/03-deep-learning-core/07-regularization/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/07-regularization/code/main.py), allowing you to compare their effects on train-test gaps using synthetic datasets.