# How AI Engineering From Scratch Teaches Backpropagation Before Using PyTorch

> Learn AI engineering from scratch by manually implementing backpropagation in Python before using PyTorch. Build a mini framework and master the chain rule.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: internals
- Published: 2026-06-22

---

**AI Engineering From Scratch teaches backpropagation by having students implement the chain rule and backward pass manually in pure Python or Julia, building a mini-framework from scratch before introducing PyTorch's autograd in Lesson 11.**

The repository follows a **build-first pedagogy** where learners code every component of a neural network by hand. This approach ensures that by the time students encounter **AI Engineering From Scratch backpropagation** concepts using PyTorch, they understand the underlying mathematics of automatic differentiation and gradient flow.

## The Five-Phase Learning Progression

### Phase 1: Mathematical Foundations

Students begin with calculus fundamentals, focusing on the **chain rule** and how gradients propagate through a computation graph. This theoretical base prepares them to write gradient code without relying on library abstractions.

### Phase 2: Manual Backpropagation Implementation

In **Phase 03 | Lesson 03**, students write the `forward()` and `backward()` functions manually. The code stores intermediate activations during the forward pass—specifically inputs, weights, biases, and layer outputs—in a cache. The `backward()` function then traverses these stored values in reverse, applying the chain rule to compute **∂L/∂w** for every parameter.

The implementation in [`phases/03-deep-learning-core/03-backpropagation/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/03-backpropagation/code/main.py) handles a two-layer network solving XOR, using sigmoid activations and binary cross-entropy loss. Students must explicitly calculate derivatives of the sigmoid (σ'(x) = σ(x)(1-σ(x))) and propagate gradients through each layer manually.

### Phase 3: The Mini-Framework Refactor

After mastering the manual process, **Lesson 10** consolidates these concepts into a reusable module. Located in [`phases/03-deep-learning-core/10-mini-framework/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/10-mini-framework/code/main.py), this lesson abstracts the forward and backward passes into classes while remaining framework-free. This step teaches software engineering patterns—encapsulation, modularity, and clean APIs—before introducing external dependencies.

### Phase 4: Introduction to PyTorch

Only after the manual implementation is solidified does the curriculum introduce PyTorch in **Phase 03 | Lesson 11**. The lesson at [`phases/03-deep-learning-core/11-intro-to-pytorch/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/11-intro-to-pytorch/code/main.py) demonstrates how `torch.autograd` automates the same chain-rule calculations students previously coded by hand. Students learn that `loss.backward()` simply automates the gradient computation they performed manually in earlier lessons.

## Code Comparison: Manual vs. Automatic Differentiation

The progression becomes clear when comparing the manual implementation against the PyTorch equivalent.

**Manual backpropagation** requires explicit caching and gradient calculation:

```python
import math, random

def forward(x, w1, b1, w2, b2):
    z1 = [sum(x_i * w_i for x_i, w_i in zip(x, col)) + b for col, b in zip(zip(*w1), b1)]
    a1 = [1 / (1 + math.exp(-z)) for z in z1]
    z2 = sum(a1_i * w_i for a1_i, w_i in zip(a1, w2)) + b2
    y_hat = 1 / (1 + math.exp(-z2))
    cache = (x, w1, b1, a1, w2, b2, y_hat)
    return y_hat, cache

def backward(y, cache):
    x, w1, b1, a1, w2, b2, y_hat = cache
    dL_dy = -(y / y_hat) + ((1 - y) / (1 - y_hat))
    dy_dz2 = y_hat * (1 - y_hat)
    dL_dz2 = dL_dy * dy_dz2
    dw2 = [dL_dz2 * a for a in a1]
    db2 = dL_dz2
    dz2_da1 = w2
    dL_da1 = dL_dz2 * dz2_da1
    da1_dz1 = [a * (1 - a) for a in a1]
    dL_dz1 = [dL_da1_i * da1_dz1_i for dL_da1_i, da1_dz1_i in zip(dL_da1, da1_dz1)]
    dw1 = [[dL_dz1_j * x_i for x_i in x] for dL_dz1_j in dL_dz1]
    db1 = dL_dz1
    return dw1, db1, dw2, db2

```

**PyTorch implementation** abstracts this into `torch.nn` modules:

```python
import torch
import torch.nn as nn

class TwoLayerNet(nn.Module):
    def __init__(self, in_dim, hidden_dim):
        super().__init__()
        self.fc1 = nn.Linear(in_dim, hidden_dim)
        self.fc2 = nn.Linear(hidden_dim, 1)

    def forward(self, x):
        a1 = torch.sigmoid(self.fc1(x))
        y_hat = torch.sigmoid(self.fc2(a1))
        return y_hat

net = TwoLayerNet(2, 4)
criterion = nn.BCELoss()
optimizer = torch.optim.SGD(net.parameters(), lr=0.1)

x = torch.tensor([[0., 1.], [1., 0.], [1., 1.], [0., 0.]])
y = torch.tensor([[1.], [1.], [0.], [0.]])

for epoch in range(500):
    optimizer.zero_grad()
    preds = net(x)
    loss = criterion(preds, y)
    loss.backward()  # Automatic backpropagation

    optimizer.step()

```

## Key Implementation Files

The repository structure reveals the deliberate progression:

- [`phases/03-deep-learning-core/03-backpropagation/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/03-backpropagation/code/main.py) – Python implementation with explicit `forward()` and `backward()` functions
- `phases/03-deep-learning-core/03-backpropagation/code/main.jl` – Julia implementation demonstrating language-agnostic concepts
- [`phases/03-deep-learning-core/10-mini-framework/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/10-mini-framework/code/main.py) – Refactored mini-framework with modular network classes
- [`phases/03-deep-learning-core/11-intro-to-pytorch/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/11-intro-to-pytorch/code/main.py) – PyTorch `nn.Module` implementation using `torch.autograd`

Each file builds upon the previous, with the manual versions requiring students to manage the computation graph and gradient storage manually, while the PyTorch version delegates these tasks to `autograd`.

## Summary

- **AI Engineering From Scratch** employs a build-first methodology where students implement backpropagation manually before using PyTorch
- The curriculum progresses from mathematical theory to manual Python implementation, then to a framework-free mini-framework, and finally to PyTorch
- Key skills developed include explicit chain-rule application, gradient flow visualization, and memory management through manual caching
- Source files in `phases/03-deep-learning-core/` demonstrate the transition from hand-coded `backward()` functions to `loss.backward()`
- By Lesson 11, students understand that PyTorch's autograd is an automation tool for the mathematics they previously implemented explicitly

## Frequently Asked Questions

### Why learn backpropagation manually before PyTorch?

Manual implementation forces you to trace every gradient calculation through the chain rule, revealing how information flows backward through the network. When you later call `loss.backward()` in PyTorch, you recognize the automation as a convenience rather than magic, enabling better debugging and architecture design.

### What programming languages does AI Engineering From Scratch use for backpropagation?

The repository provides implementations in both **Python** and **Julia**, with parallel codebases in [`main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/main.py) and `main.jl` files. This dual-language approach reinforces that backpropagation is a mathematical concept independent of any specific deep learning framework.

### Which lesson introduces PyTorch?

PyTorch first appears in **Phase 03 | Lesson 11** titled "Intro to PyTorch," located at `phases/03-deep-learning-core/11-intro-to-pytorch/`. This lesson follows ten prior lessons covering mathematical foundations, manual implementations, and framework construction.

### How does manual backpropagation differ from PyTorch autograd?

Manual backpropagation requires you to explicitly cache intermediate values during the forward pass and compute gradients using the chain rule in the `backward()` function. PyTorch's autograd automatically constructs a dynamic computation graph during `forward()` and computes gradients when you call `backward()`, handling the chain rule and memory management internally.