# How the AI Engineering Curriculum Teaches Backpropagation from First Principles Before PyTorch

> Learn backpropagation from scratch using Python. This AI curriculum builds an autograd engine manually before introducing PyTorch, teaching the chain rule and gradient flow.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: deep-dive
- Published: 2026-08-29

---

**The curriculum builds a complete autograd engine from scratch using only Python's standard library, forcing learners to implement the chain rule, computational graphs, and gradient flow manually before revealing how PyTorch automates the identical mathematics.**

The `rohitg00/ai-engineering-from-scratch` repository employs a pedagogical approach that demystifies deep learning by requiring students to write every component of backpropagation by hand. By constructing a scalar-valued autograd engine without external dependencies, learners internalize exactly how gradients propagate through computational graphs before encountering high-level frameworks. This method bridges the gap between abstract calculus and production-ready PyTorch code.

## Building the Autograd Engine from Scratch

The curriculum begins in `phases/03-deep-learning-core/03-backpropagation/` with a `Value` class that serves as the fundamental unit of computation. Unlike PyTorch's `Tensor`, this implementation exposes every internal mechanism of reverse-mode automatic differentiation.

### The Value Node and Gradient Storage

At the core of the lesson is a `Value` class defined in [`phases/03-deep-learning-core/03-backpropagation/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/03-backpropagation/code/main.py) (lines 45-75). Each instance stores not only the scalar data but also the gradient and pointers to its parent nodes in the computational graph:

```python
class Value:
    def __init__(self, data, children=(), op=''):
        self.data = data
        self.grad = 0.0
        self._backward = lambda: None
        self._children = set(children)
        self._op = op

```

The `_backward` attribute holds a closure representing the local gradient computation for the specific operation that created this node. This mirrors the fundamental structure of PyTorch's `torch.autograd.Function` but remains fully transparent to the learner.

### Operator Overloads and Local Gradients

To construct the computational graph automatically, the curriculum implements Python's dunder methods for arithmetic operations. Each overload creates a new `Value` node while registering how to propagate gradients backward through the chain rule:

```python

# addition -------------------------------------------------

def __add__(self, other):
    other = other if isinstance(other, Value) else Value(other)
    out = Value(self.data + other.data, (self, other), '+')
    def _backward():
        self.grad += out.grad
        other.grad += out.grad
    out._backward = _backward
    return out

# multiplication ------------------------------------------

def __mul__(self, other):
    other = other if isinstance(other, Value) else Value(other)
    out = Value(self.data * other.data, (self, other), '*')
    def _backward():
        self.grad += other.data * out.grad
        other.grad += self.data * out.grad
    out._backward = _backward
    return out

```

This implementation teaches that addition distributes gradients equally while multiplication applies the product rule, concepts that remain hidden within PyTorch's C++ backend but are essential for debugging vanishing or exploding gradients.

### Activation Functions and Clamping

The lesson extends the `Value` class with a `sigmoid` method (lines 97-118) that demonstrates numerical stability techniques and the derivative of activation functions:

```python
def sigmoid(self):
    x = max(-500, min(500, self.data))      # clamp for stability

    s = 1.0 / (1.0 + math.exp(-x))
    out = Value(s, (self,), 'sigmoid')
    def _backward():
        self.grad += (s * (1 - s)) * out.grad
    out._backward = _backward
    return out

```

The gradient `s * (1 - s)` explicitly shows why sigmoid units saturate, a concept the curriculum emphasizes by later benchmarking against PyTorch's optimized implementations.

## Implementing the Backward Pass with Topological Sort

Before gradients can flow, the engine must traverse the computational graph in reverse topological order to ensure that a node's gradient is fully accumulated before its parents compute their own derivatives.

### Manual Graph Traversal

The `backward()` method (lines 124-143) implements this traversal using recursive depth-first search, a technique that reveals how PyTorch's autograd engine builds the tape:

```python
def backward(self):
    topo = []
    visited = set()
    def build_topo(v):
        if v not in visited:
            visited.add(v)
            for child in v._children:
                build_topo(child)
            topo.append(v)
    build_topo(self)
    self.grad = 1.0                     # dL/dL = 1

    for v in reversed(topo):
        v._backward()

```

The curriculum emphasizes the **memory-computation trade-off** inherent in this approach: the forward pass must retain intermediate activations so the backward pass can reuse them for gradient calculations, explaining why training requires significantly more RAM than inference.

## Constructing Neural Network Abstractions

After establishing the autograd primitives, the lesson progresses to higher-level constructs that demonstrate how deep networks emerge from simple compositions of `Value` objects.

### Neurons, Layers, and MLPs

The `Network` class stacks `Value`-based neurons into fully-connected layers. This abstraction shows that multi-layer perceptrons are merely collections of the primitive computational blocks already implemented, with no magic introduced by framework code. The parameters remain standard Python lists of `Value` objects, allowing direct manipulation of gradients and weights during training.

## Training on Real Datasets Without PyTorch

To validate the hand-crafted engine, the curriculum implements training loops for canonical non-linear classification problems, proving that the from-scratch gradients converge identically to reference implementations.

### Solving the XOR Problem

The XOR dataset serves as the first litmus test, requiring hidden layers to learn non-linear separability. The training code (lines 209-237) demonstrates manual stochastic gradient descent without `torch.optim`:

```python
random.seed(42)
net = Network([2, 4, 1])               # 2‑in, 4‑hidden, 1‑out

xor_data = [([0,0],0), ([0,1],1), ([1,0],1), ([1,1],0)]
lr = 1.0

for epoch in range(1000):
    total_loss = Value(0.0)
    for inputs, target in xor_data:
        x = [Value(i) for i in inputs]
        pred = net(x)
        loss = mse_loss(pred, target)
        total_loss = total_loss + loss
    net.zero_grad()
    total_loss.backward()
    for p in net.parameters():
        p.data -= lr * p.grad

```

This explicit loop—forward pass, zeroing gradients, backward pass, and parameter updates—mirrors PyTorch's training workflow but forces the learner to handle each step manually.

## Bridging to PyTorch

The curriculum culminates in a side-by-side comparison documented in [`phases/03-deep-learning-core/03-backpropagation/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/03-backpropagation/docs/en.md) (lines 99-124). After mastering the manual implementation, students encounter the equivalent PyTorch code:

```python
import torch, torch.nn as nn

model = nn.Sequential(
    nn.Linear(2, 4),
    nn.Sigmoid(),
    nn.Linear(4, 1),
    nn.Sigmoid(),
)
optimizer = torch.optim.SGD(model.parameters(), lr=1.0)
criterion = nn.MSELoss()

X = torch.tensor([[0,0],[0,1],[1,0],[1,1]], dtype=torch.float32)
y = torch.tensor([[0],[1],[1],[0]], dtype=torch.float32)

for epoch in range(1000):
    pred = model(X)
    loss = criterion(pred, y)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

```

The lesson highlights that **PyTorch replaces only the engine**, not the mathematics. The chain rule, gradient accumulation, and weight updates remain identical; the framework merely automates graph construction and the topological sort previously implemented by hand.

## Summary

- The curriculum requires implementing a **scalar `Value` class** with manual gradient tracking before introducing tensors, ensuring learners understand reverse-mode autodiff at the node level.
- Topological sorting and **manual backward traversal** in [`phases/03-deep-learning-core/03-backpropagation/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/03-backpropagation/code/main.py) demonstrate how gradients flow through computational graphs.
- **Operator overloading** with `_backward` closures teaches the chain rule explicitly, exposing why multiplication gates distribute gradients proportionally to their inputs.
- Training on **XOR and synthetic circle datasets** validates that the hand-coded engine produces identical convergence to established frameworks.
- The final **PyTorch comparison** reveals that modern libraries automate graph building and memory management while preserving the same underlying calculus.

## Frequently Asked Questions

### Why learn backpropagation from scratch instead of using PyTorch immediately?

Writing the autograd engine manually exposes the **chain rule's local application** at every node and the memory-computation trade-offs inherent in storing activations for the backward pass. When learners later encounter vanishing gradients or memory errors in PyTorch, they possess the mental model to debug the underlying mathematics rather than treating the framework as a black box.

### How does the `Value` class handle gradient accumulation?

Each `Value` node initializes `self.grad = 0.0` and uses the `+=` operator in `_backward` closures to accumulate contributions from multiple children. This implements the **multivariate chain rule**, where a node may have multiple outgoing edges in the computational graph and must sum gradients from all paths, exactly as PyTorch's autograd does with `grad_accumulator`.

### What prevents numerical overflow in the sigmoid implementation?

The `sigmoid` method clamps the input to `[-500, 500]` before exponentiation to prevent overflow in the `exp(-x)` calculation. This mirrors PyTorch's internal stability checks and teaches learners that **numerical analysis** considerations are integral to deep learning engineering, not just theoretical concerns.

### How does topological sort ensure correct gradient propagation?

Reverse topological order guarantees that a node's gradient is fully computed from all its children before propagating backward to its parents. This implements the **reverse-mode automatic differentiation** algorithm where partial derivatives flow from outputs to inputs, ensuring that `dL/dw` is available before computing `dL/d(input)` for any weight `w`.