Backpropagation From Scratch: Teaching Deep Learning Before PyTorch

The curriculum teaches backpropagation by first building a minimal autograd engine in pure Python, then training neural networks on XOR and circle classification tasks, and finally comparing the hand-written implementation line-by-line with PyTorch to reveal that "automatic" differentiation is simply systematic application of the chain rule.

The rohitg00/ai-engineering-from-scratch repository introduces backpropagation through a "no magic" approach in Phase 03, Lesson 03. Students implement a complete scalar-valued autograd engine using only Python standard libraries before ever importing PyTorch. This pedagogical sequence ensures learners understand the underlying calculus and graph traversal mechanics that power modern deep learning frameworks.

Mathematical Foundations: The Chain Rule

The curriculum revisits differential calculus from Phase 01, Lesson 05 before writing any backpropagation code. In phases/01-math-foundations/05-chain-rule-and-autodiff/docs/en.md, the chain rule is expressed formally: for a composite function y = f(g(x)), the derivative expands to dy/dx = f'(g(x))·g'(x).

This mathematical foundation appears in phases/03-deep-learning-core/03-backpropagation/docs/en.md lines 27-30, where the lesson emphasizes that reverse-mode automatic differentiation is merely the chain rule applied recursively through a computational graph. Students learn that gradients flow backward through each operation by multiplying local derivatives with upstream gradients.

Building the Autograd Engine

The core exercise requires implementing a directed-acyclic graph where nodes represent elementary operations including addition, multiplication, and sigmoid activation. During the forward pass, the graph is constructed explicitly; during the backward pass, gradients propagate along the edges according to the chain rule.

The Value Class Implementation

The engine centers on a Value class defined in phases/03-deep-learning-core/03-backpropagation/code/main.py. As documented in lines 45-55 of the lesson docs, each instance stores:

• data: the scalar value
• grad: the gradient accumulator initialized to 0.0
• _backward: a closure defining how to route gradients to parents
• _children: the parent nodes in the computation graph

Every operation creates a new Value node and registers the specific backward logic required to compute partial derivatives.

Operator Overloads with Backward Logic

The Value class implements Python's dunder methods to overload arithmetic operators, each attaching the appropriate gradient computation:

Addition (a + b) forwards the sum and defines _backward that simply adds the upstream gradient to each operand. As shown in lines 65-73 of the docs:

def __add__(self, other):
    other = other if isinstance(other, Value) else Value(other)
    out = Value(self.data + other.data, (self, other), '+')
    def _backward():
        self.grad += out.grad
        other.grad += out.grad
    out._backward = _backward
    return out

Multiplication (a * b) forwards the product and supplies _backward that multiplies the upstream gradient by the other operand’s value (lines 76-84):

def __mul__(self, other):
    other = other if isinstance(other, Value) else Value(other)
    out = Value(self.data * other.data, (self, other), '*')
    def _backward():
        self.grad += other.data * out.grad
        other.grad += self.data * out.grad
    out._backward = _backward
    return out

Sigmoid uses the analytically-derived derivative σ·(1‑σ) and re-uses the already-computed sigmoid output for efficiency (lines 94-103):

def sigmoid(self):
    x = max(-500, min(500, self.data))
    s = 1.0 / (1.0 + math.exp(-x))
    out = Value(s, (self,), 'sigmoid')
    def _backward():
        self.grad += (s * (1 - s)) * out.grad
    out._backward = _backward
    return out

Topological Sort for Correct Gradient Order

A critical implementation detail appears in lines 24-42 of the lesson documentation: gradients must be accumulated only after all downstream gradients have been computed. The backward() method uses depth-first search to collect nodes in reverse-dependency order:

def backward(self):
    topo = []
    visited = set()
    def build_topo(v):
        if v not in visited:
            visited.add(v)
            for child in v._children:
                build_topo(child)
            topo.append(v)
    build_topo(self)
    self.grad = 1.0                # dL/dL = 1

    for v in reversed(topo):
        v._backward()

This topological sort ensures that when v._backward() executes, v.grad already contains the complete upstream gradient from all subsequent operations.

From Engine to Network

With the autograd primitive established, students construct higher-level abstractions in phases/03-deep-learning-core/03-backpropagation/code/main.py. The Neuron class combines weighted Value inputs through addition and multiplication, applies a sigmoid non-linearity, and exposes parameters for optimization.

The Layer class aggregates multiple neurons, and the Network class stacks layers to form arbitrary feedforward architectures. All operations remain transparent—each weight and bias is a Value node with trackable gradients.

Hands-On Training

The curriculum solidifies concepts through two canonical experiments implemented without PyTorch:

XOR Classification – A 2-4-1 network learns the non-linear XOR truth table purely via the hand-written engine (lines 6-38). The training loop manually calculates mean squared error loss, calls total.backward() to populate gradients, and updates parameters with stochastic gradient descent:

net = Network([2, 4, 1])          # 2‑inputs → 4 hidden → 1 output

xor_data = [([0,0],0), ([0,1],1), ([1,0],1), ([1,1],0)]
lr = 1.0

for epoch in range(1000):
    total = Value(0.0)
    for x_vals, target in xor_data:
        x = [Value(v) for v in x_vals]
        pred = net(x)
        loss = (pred + Value(-target)) * (pred + Value(-target))  # MSE

        total = total + loss
    net.zero_grad()
    total.backward()
    for p in net.parameters():
        p.data -= lr * p.grad

Circle Classification – A 2-8-1 network discovers a circular decision boundary (lines 47-90), illustrating SGD updates after each sample and exposing the vanishing-gradient problem when sigmoid derivatives saturate below 0.25.

Bridging to PyTorch

After mastering the manual implementation, the lesson in phases/03-deep-learning-core/03-backpropagation/docs/en.md lines 98-32 presents the exact same training script using PyTorch:

import torch, torch.nn as nn
model = nn.Sequential(nn.Linear(2,4), nn.Sigmoid(),
                      nn.Linear(4,1), nn.Sigmoid())
opt = torch.optim.SGD(model.parameters(), lr=1.0)
criterion = nn.MSELoss()
X = torch.tensor([[0,0],[0,1],[1,0],[1,1]], dtype=torch.float32)
y = torch.tensor([[0],[1],[1],[0]], dtype=torch.float32)

for epoch in range(1000):
    pred = model(X)
    loss = criterion(pred, y)
    opt.zero_grad()
    loss.backward()
    opt.step()

This side-by-side comparison maps each custom step—total_loss.backward(), manual parameter updates, and gradient zeroing—to their PyTorch counterparts. The curriculum demonstrates that PyTorch's "magic" is simply automated chain-rule graph traversal identical to the engine students built.

Summary

• Mathematical grounding first: The curriculum establishes chain-rule calculus in Phase 01 before any implementation.
• Explicit computational graphs: The Value class in phases/03-deep-learning-core/03-backpropagation/code/main.py makes gradient flow visible and modifiable.
• Topological ordering: Depth-first search ensures gradients accumulate correctly before parent nodes execute their _backward methods.
• Runnable toy problems: XOR and circle classification validate that the scratch engine actually learns non-linear patterns.
• Framework bridge: Direct comparison with PyTorch demystifies deep learning libraries as optimized implementations of the same principles.

Frequently Asked Questions

Why implement backpropagation manually before using PyTorch?

Implementing backpropagation from scratch forces learners to confront the mechanics of gradient flow, tensor shapes, and the chain rule explicitly. When students later encounter PyTorch's loss.backward(), they understand it as an automated topological traversal of the computational graph they previously built manually in phases/03-deep-learning-core/03-backpropagation/code/main.py.

What is the chain rule and why does it matter for backpropagation?

The chain rule states that for composite functions, the derivative dy/dx equals the product of intermediate derivatives dy/dg · dg/dx. In neural networks, this allows gradients to flow backward through arbitrary depths of composition—each layer multiplies the upstream gradient by its local Jacobian, enabling efficient calculation of how every parameter affects the loss.

How does topological sort ensure correct gradient calculation?

In a computational graph, a node may feed into multiple downstream consumers. The topological sort (implemented via depth-first search in phases/03-deep-learning-core/03-backpropagation/docs/en.md lines 24-42) visits child nodes before parents. This ensures that when a node's _backward executes, its grad field already contains the sum of gradients from all downstream paths, satisfying the multivariate chain rule.

What are the limitations of this scalar-valued autograd engine?

The educational engine operates on scalar Value objects, making it impractical for high-dimensional tensors or GPU acceleration. It serves as a pedagogical tool to understand reverse-mode autodiff, but production systems like PyTorch use tensor operations and optimized C++ kernels to achieve the same mathematical results with computational efficiency.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}