How the AI Engineering Curriculum Teaches Backpropagation from First Principles Before PyTorch

The curriculum builds a complete autograd engine from scratch using only Python's standard library, forcing learners to implement the chain rule, computational graphs, and gradient flow manually before revealing how PyTorch automates the identical mathematics.

The rohitg00/ai-engineering-from-scratch repository employs a pedagogical approach that demystifies deep learning by requiring students to write every component of backpropagation by hand. By constructing a scalar-valued autograd engine without external dependencies, learners internalize exactly how gradients propagate through computational graphs before encountering high-level frameworks. This method bridges the gap between abstract calculus and production-ready PyTorch code.

Building the Autograd Engine from Scratch

The curriculum begins in phases/03-deep-learning-core/03-backpropagation/ with a Value class that serves as the fundamental unit of computation. Unlike PyTorch's Tensor, this implementation exposes every internal mechanism of reverse-mode automatic differentiation.

The Value Node and Gradient Storage

At the core of the lesson is a Value class defined in phases/03-deep-learning-core/03-backpropagation/code/main.py (lines 45-75). Each instance stores not only the scalar data but also the gradient and pointers to its parent nodes in the computational graph:

class Value:
    def __init__(self, data, children=(), op=''):
        self.data = data
        self.grad = 0.0
        self._backward = lambda: None
        self._children = set(children)
        self._op = op

The _backward attribute holds a closure representing the local gradient computation for the specific operation that created this node. This mirrors the fundamental structure of PyTorch's torch.autograd.Function but remains fully transparent to the learner.

Operator Overloads and Local Gradients

To construct the computational graph automatically, the curriculum implements Python's dunder methods for arithmetic operations. Each overload creates a new Value node while registering how to propagate gradients backward through the chain rule:

# addition -------------------------------------------------

def __add__(self, other):
    other = other if isinstance(other, Value) else Value(other)
    out = Value(self.data + other.data, (self, other), '+')
    def _backward():
        self.grad += out.grad
        other.grad += out.grad
    out._backward = _backward
    return out

# multiplication ------------------------------------------

def __mul__(self, other):
    other = other if isinstance(other, Value) else Value(other)
    out = Value(self.data * other.data, (self, other), '*')
    def _backward():
        self.grad += other.data * out.grad
        other.grad += self.data * out.grad
    out._backward = _backward
    return out

This implementation teaches that addition distributes gradients equally while multiplication applies the product rule, concepts that remain hidden within PyTorch's C++ backend but are essential for debugging vanishing or exploding gradients.

Activation Functions and Clamping

The lesson extends the Value class with a sigmoid method (lines 97-118) that demonstrates numerical stability techniques and the derivative of activation functions:

def sigmoid(self):
    x = max(-500, min(500, self.data))      # clamp for stability

    s = 1.0 / (1.0 + math.exp(-x))
    out = Value(s, (self,), 'sigmoid')
    def _backward():
        self.grad += (s * (1 - s)) * out.grad
    out._backward = _backward
    return out

The gradient s * (1 - s) explicitly shows why sigmoid units saturate, a concept the curriculum emphasizes by later benchmarking against PyTorch's optimized implementations.

Implementing the Backward Pass with Topological Sort

Before gradients can flow, the engine must traverse the computational graph in reverse topological order to ensure that a node's gradient is fully accumulated before its parents compute their own derivatives.

Manual Graph Traversal

The backward() method (lines 124-143) implements this traversal using recursive depth-first search, a technique that reveals how PyTorch's autograd engine builds the tape:

def backward(self):
    topo = []
    visited = set()
    def build_topo(v):
        if v not in visited:
            visited.add(v)
            for child in v._children:
                build_topo(child)
            topo.append(v)
    build_topo(self)
    self.grad = 1.0                     # dL/dL = 1

    for v in reversed(topo):
        v._backward()

The curriculum emphasizes the memory-computation trade-off inherent in this approach: the forward pass must retain intermediate activations so the backward pass can reuse them for gradient calculations, explaining why training requires significantly more RAM than inference.

Constructing Neural Network Abstractions

After establishing the autograd primitives, the lesson progresses to higher-level constructs that demonstrate how deep networks emerge from simple compositions of Value objects.

Neurons, Layers, and MLPs

The Network class stacks Value-based neurons into fully-connected layers. This abstraction shows that multi-layer perceptrons are merely collections of the primitive computational blocks already implemented, with no magic introduced by framework code. The parameters remain standard Python lists of Value objects, allowing direct manipulation of gradients and weights during training.

Training on Real Datasets Without PyTorch

To validate the hand-crafted engine, the curriculum implements training loops for canonical non-linear classification problems, proving that the from-scratch gradients converge identically to reference implementations.

Solving the XOR Problem

The XOR dataset serves as the first litmus test, requiring hidden layers to learn non-linear separability. The training code (lines 209-237) demonstrates manual stochastic gradient descent without torch.optim:

random.seed(42)
net = Network([2, 4, 1])               # 2‑in, 4‑hidden, 1‑out

xor_data = [([0,0],0), ([0,1],1), ([1,0],1), ([1,1],0)]
lr = 1.0

for epoch in range(1000):
    total_loss = Value(0.0)
    for inputs, target in xor_data:
        x = [Value(i) for i in inputs]
        pred = net(x)
        loss = mse_loss(pred, target)
        total_loss = total_loss + loss
    net.zero_grad()
    total_loss.backward()
    for p in net.parameters():
        p.data -= lr * p.grad

This explicit loop—forward pass, zeroing gradients, backward pass, and parameter updates—mirrors PyTorch's training workflow but forces the learner to handle each step manually.

Bridging to PyTorch

The curriculum culminates in a side-by-side comparison documented in phases/03-deep-learning-core/03-backpropagation/docs/en.md (lines 99-124). After mastering the manual implementation, students encounter the equivalent PyTorch code:

import torch, torch.nn as nn

model = nn.Sequential(
    nn.Linear(2, 4),
    nn.Sigmoid(),
    nn.Linear(4, 1),
    nn.Sigmoid(),
)
optimizer = torch.optim.SGD(model.parameters(), lr=1.0)
criterion = nn.MSELoss()

X = torch.tensor([[0,0],[0,1],[1,0],[1,1]], dtype=torch.float32)
y = torch.tensor([[0],[1],[1],[0]], dtype=torch.float32)

for epoch in range(1000):
    pred = model(X)
    loss = criterion(pred, y)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

The lesson highlights that PyTorch replaces only the engine, not the mathematics. The chain rule, gradient accumulation, and weight updates remain identical; the framework merely automates graph construction and the topological sort previously implemented by hand.

Summary

• The curriculum requires implementing a scalar Value class with manual gradient tracking before introducing tensors, ensuring learners understand reverse-mode autodiff at the node level.
• Topological sorting and manual backward traversal in phases/03-deep-learning-core/03-backpropagation/code/main.py demonstrate how gradients flow through computational graphs.
• Operator overloading with _backward closures teaches the chain rule explicitly, exposing why multiplication gates distribute gradients proportionally to their inputs.
• Training on XOR and synthetic circle datasets validates that the hand-coded engine produces identical convergence to established frameworks.
• The final PyTorch comparison reveals that modern libraries automate graph building and memory management while preserving the same underlying calculus.

Frequently Asked Questions

Why learn backpropagation from scratch instead of using PyTorch immediately?

Writing the autograd engine manually exposes the chain rule's local application at every node and the memory-computation trade-offs inherent in storing activations for the backward pass. When learners later encounter vanishing gradients or memory errors in PyTorch, they possess the mental model to debug the underlying mathematics rather than treating the framework as a black box.

How does the Value class handle gradient accumulation?

Each Value node initializes self.grad = 0.0 and uses the += operator in _backward closures to accumulate contributions from multiple children. This implements the multivariate chain rule, where a node may have multiple outgoing edges in the computational graph and must sum gradients from all paths, exactly as PyTorch's autograd does with grad_accumulator.

What prevents numerical overflow in the sigmoid implementation?

The sigmoid method clamps the input to [-500, 500] before exponentiation to prevent overflow in the exp(-x) calculation. This mirrors PyTorch's internal stability checks and teaches learners that numerical analysis considerations are integral to deep learning engineering, not just theoretical concerns.

How does topological sort ensure correct gradient propagation?

Reverse topological order guarantees that a node's gradient is fully computed from all its children before propagating backward to its parents. This implements the reverse-mode automatic differentiation algorithm where partial derivatives flow from outputs to inputs, ensuring that dL/dw is available before computing dL/d(input) for any weight w.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}