# Backpropagation From Scratch: Teaching Deep Learning Before PyTorch

> Learn backpropagation from scratch by building an autograd engine in Python before PyTorch. Understand automatic differentiation and the chain rule in this deep learning curriculum.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: tutorial
- Published: 2026-07-31

---

**The curriculum teaches backpropagation by first building a minimal autograd engine in pure Python, then training neural networks on XOR and circle classification tasks, and finally comparing the hand-written implementation line-by-line with PyTorch to reveal that "automatic" differentiation is simply systematic application of the chain rule.**

The rohitg00/ai-engineering-from-scratch repository introduces backpropagation through a "no magic" approach in Phase 03, Lesson 03. Students implement a complete scalar-valued autograd engine using only Python standard libraries before ever importing PyTorch. This pedagogical sequence ensures learners understand the underlying calculus and graph traversal mechanics that power modern deep learning frameworks.

## Mathematical Foundations: The Chain Rule

The curriculum revisits differential calculus from Phase 01, Lesson 05 before writing any backpropagation code. In [`phases/01-math-foundations/05-chain-rule-and-autodiff/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/01-math-foundations/05-chain-rule-and-autodiff/docs/en.md), the chain rule is expressed formally: for a composite function `y = f(g(x))`, the derivative expands to `dy/dx = f'(g(x))·g'(x)`.

This mathematical foundation appears in [`phases/03-deep-learning-core/03-backpropagation/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/03-backpropagation/docs/en.md) lines 27-30, where the lesson emphasizes that reverse-mode automatic differentiation is merely the chain rule applied recursively through a computational graph. Students learn that gradients flow backward through each operation by multiplying local derivatives with upstream gradients.

## Building the Autograd Engine

The core exercise requires implementing a directed-acyclic graph where nodes represent elementary operations including addition, multiplication, and sigmoid activation. During the forward pass, the graph is constructed explicitly; during the backward pass, gradients propagate along the edges according to the chain rule.

### The Value Class Implementation

The engine centers on a `Value` class defined in [`phases/03-deep-learning-core/03-backpropagation/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/03-backpropagation/code/main.py). As documented in lines 45-55 of the lesson docs, each instance stores:

- `data`: the scalar value
- `grad`: the gradient accumulator initialized to 0.0
- `_backward`: a closure defining how to route gradients to parents
- `_children`: the parent nodes in the computation graph

Every operation creates a new `Value` node and registers the specific backward logic required to compute partial derivatives.

### Operator Overloads with Backward Logic

The `Value` class implements Python's dunder methods to overload arithmetic operators, each attaching the appropriate gradient computation:

**Addition** (`a + b`) forwards the sum and defines `_backward` that simply adds the upstream gradient to each operand. As shown in lines 65-73 of the docs:

```python
def __add__(self, other):
    other = other if isinstance(other, Value) else Value(other)
    out = Value(self.data + other.data, (self, other), '+')
    def _backward():
        self.grad += out.grad
        other.grad += out.grad
    out._backward = _backward
    return out

```

**Multiplication** (`a * b`) forwards the product and supplies `_backward` that multiplies the upstream gradient by the *other* operand’s value (lines 76-84):

```python
def __mul__(self, other):
    other = other if isinstance(other, Value) else Value(other)
    out = Value(self.data * other.data, (self, other), '*')
    def _backward():
        self.grad += other.data * out.grad
        other.grad += self.data * out.grad
    out._backward = _backward
    return out

```

**Sigmoid** uses the analytically-derived derivative `σ·(1‑σ)` and re-uses the already-computed sigmoid output for efficiency (lines 94-103):

```python
def sigmoid(self):
    x = max(-500, min(500, self.data))
    s = 1.0 / (1.0 + math.exp(-x))
    out = Value(s, (self,), 'sigmoid')
    def _backward():
        self.grad += (s * (1 - s)) * out.grad
    out._backward = _backward
    return out

```

### Topological Sort for Correct Gradient Order

A critical implementation detail appears in lines 24-42 of the lesson documentation: gradients must be accumulated only after all downstream gradients have been computed. The `backward()` method uses depth-first search to collect nodes in reverse-dependency order:

```python
def backward(self):
    topo = []
    visited = set()
    def build_topo(v):
        if v not in visited:
            visited.add(v)
            for child in v._children:
                build_topo(child)
            topo.append(v)
    build_topo(self)
    self.grad = 1.0                # dL/dL = 1

    for v in reversed(topo):
        v._backward()

```

This topological sort ensures that when `v._backward()` executes, `v.grad` already contains the complete upstream gradient from all subsequent operations.

## From Engine to Network

With the autograd primitive established, students construct higher-level abstractions in [`phases/03-deep-learning-core/03-backpropagation/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/03-backpropagation/code/main.py). The `Neuron` class combines weighted `Value` inputs through addition and multiplication, applies a sigmoid non-linearity, and exposes parameters for optimization.

The `Layer` class aggregates multiple neurons, and the `Network` class stacks layers to form arbitrary feedforward architectures. All operations remain transparent—each weight and bias is a `Value` node with trackable gradients.

## Hands-On Training

The curriculum solidifies concepts through two canonical experiments implemented without PyTorch:

**XOR Classification** – A 2-4-1 network learns the non-linear XOR truth table purely via the hand-written engine (lines 6-38). The training loop manually calculates mean squared error loss, calls `total.backward()` to populate gradients, and updates parameters with stochastic gradient descent:

```python
net = Network([2, 4, 1])          # 2‑inputs → 4 hidden → 1 output

xor_data = [([0,0],0), ([0,1],1), ([1,0],1), ([1,1],0)]
lr = 1.0

for epoch in range(1000):
    total = Value(0.0)
    for x_vals, target in xor_data:
        x = [Value(v) for v in x_vals]
        pred = net(x)
        loss = (pred + Value(-target)) * (pred + Value(-target))  # MSE

        total = total + loss
    net.zero_grad()
    total.backward()
    for p in net.parameters():
        p.data -= lr * p.grad

```

**Circle Classification** – A 2-8-1 network discovers a circular decision boundary (lines 47-90), illustrating SGD updates after each sample and exposing the vanishing-gradient problem when sigmoid derivatives saturate below 0.25.

## Bridging to PyTorch

After mastering the manual implementation, the lesson in [`phases/03-deep-learning-core/03-backpropagation/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/03-backpropagation/docs/en.md) lines 98-32 presents the **exact same** training script using PyTorch:

```python
import torch, torch.nn as nn
model = nn.Sequential(nn.Linear(2,4), nn.Sigmoid(),
                      nn.Linear(4,1), nn.Sigmoid())
opt = torch.optim.SGD(model.parameters(), lr=1.0)
criterion = nn.MSELoss()
X = torch.tensor([[0,0],[0,1],[1,0],[1,1]], dtype=torch.float32)
y = torch.tensor([[0],[1],[1],[0]], dtype=torch.float32)

for epoch in range(1000):
    pred = model(X)
    loss = criterion(pred, y)
    opt.zero_grad()
    loss.backward()
    opt.step()

```

This side-by-side comparison maps each custom step—`total_loss.backward()`, manual parameter updates, and gradient zeroing—to their PyTorch counterparts. The curriculum demonstrates that PyTorch's "magic" is simply automated chain-rule graph traversal identical to the engine students built.

## Summary

- **Mathematical grounding first**: The curriculum establishes chain-rule calculus in Phase 01 before any implementation.
- **Explicit computational graphs**: The `Value` class in [`phases/03-deep-learning-core/03-backpropagation/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/03-backpropagation/code/main.py) makes gradient flow visible and modifiable.
- **Topological ordering**: Depth-first search ensures gradients accumulate correctly before parent nodes execute their `_backward` methods.
- **Runnable toy problems**: XOR and circle classification validate that the scratch engine actually learns non-linear patterns.
- **Framework bridge**: Direct comparison with PyTorch demystifies deep learning libraries as optimized implementations of the same principles.

## Frequently Asked Questions

### Why implement backpropagation manually before using PyTorch?

Implementing backpropagation from scratch forces learners to confront the mechanics of gradient flow, tensor shapes, and the chain rule explicitly. When students later encounter PyTorch's `loss.backward()`, they understand it as an automated topological traversal of the computational graph they previously built manually in [`phases/03-deep-learning-core/03-backpropagation/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/03-backpropagation/code/main.py).

### What is the chain rule and why does it matter for backpropagation?

The chain rule states that for composite functions, the derivative `dy/dx` equals the product of intermediate derivatives `dy/dg · dg/dx`. In neural networks, this allows gradients to flow backward through arbitrary depths of composition—each layer multiplies the upstream gradient by its local Jacobian, enabling efficient calculation of how every parameter affects the loss.

### How does topological sort ensure correct gradient calculation?

In a computational graph, a node may feed into multiple downstream consumers. The topological sort (implemented via depth-first search in [`phases/03-deep-learning-core/03-backpropagation/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/03-backpropagation/docs/en.md) lines 24-42) visits child nodes before parents. This ensures that when a node's `_backward` executes, its `grad` field already contains the sum of gradients from all downstream paths, satisfying the multivariate chain rule.

### What are the limitations of this scalar-valued autograd engine?

The educational engine operates on scalar `Value` objects, making it impractical for high-dimensional tensors or GPU acceleration. It serves as a pedagogical tool to understand reverse-mode autodiff, but production systems like PyTorch use tensor operations and optimized C++ kernels to achieve the same mathematical results with computational efficiency.