How AI Engineering From Scratch Teaches Backpropagation Before Using PyTorch

AI Engineering From Scratch teaches backpropagation by having students implement the chain rule and backward pass manually in pure Python or Julia, building a mini-framework from scratch before introducing PyTorch's autograd in Lesson 11.

The repository follows a build-first pedagogy where learners code every component of a neural network by hand. This approach ensures that by the time students encounter AI Engineering From Scratch backpropagation concepts using PyTorch, they understand the underlying mathematics of automatic differentiation and gradient flow.

The Five-Phase Learning Progression

Phase 1: Mathematical Foundations

Students begin with calculus fundamentals, focusing on the chain rule and how gradients propagate through a computation graph. This theoretical base prepares them to write gradient code without relying on library abstractions.

Phase 2: Manual Backpropagation Implementation

In Phase 03 | Lesson 03, students write the forward() and backward() functions manually. The code stores intermediate activations during the forward pass—specifically inputs, weights, biases, and layer outputs—in a cache. The backward() function then traverses these stored values in reverse, applying the chain rule to compute ∂L/∂w for every parameter.

The implementation in phases/03-deep-learning-core/03-backpropagation/code/main.py handles a two-layer network solving XOR, using sigmoid activations and binary cross-entropy loss. Students must explicitly calculate derivatives of the sigmoid (σ'(x) = σ(x)(1-σ(x))) and propagate gradients through each layer manually.

Phase 3: The Mini-Framework Refactor

After mastering the manual process, Lesson 10 consolidates these concepts into a reusable module. Located in phases/03-deep-learning-core/10-mini-framework/code/main.py, this lesson abstracts the forward and backward passes into classes while remaining framework-free. This step teaches software engineering patterns—encapsulation, modularity, and clean APIs—before introducing external dependencies.

Phase 4: Introduction to PyTorch

Only after the manual implementation is solidified does the curriculum introduce PyTorch in Phase 03 | Lesson 11. The lesson at phases/03-deep-learning-core/11-intro-to-pytorch/code/main.py demonstrates how torch.autograd automates the same chain-rule calculations students previously coded by hand. Students learn that loss.backward() simply automates the gradient computation they performed manually in earlier lessons.

Code Comparison: Manual vs. Automatic Differentiation

The progression becomes clear when comparing the manual implementation against the PyTorch equivalent.

Manual backpropagation requires explicit caching and gradient calculation:

import math, random

def forward(x, w1, b1, w2, b2):
    z1 = [sum(x_i * w_i for x_i, w_i in zip(x, col)) + b for col, b in zip(zip(*w1), b1)]
    a1 = [1 / (1 + math.exp(-z)) for z in z1]
    z2 = sum(a1_i * w_i for a1_i, w_i in zip(a1, w2)) + b2
    y_hat = 1 / (1 + math.exp(-z2))
    cache = (x, w1, b1, a1, w2, b2, y_hat)
    return y_hat, cache

def backward(y, cache):
    x, w1, b1, a1, w2, b2, y_hat = cache
    dL_dy = -(y / y_hat) + ((1 - y) / (1 - y_hat))
    dy_dz2 = y_hat * (1 - y_hat)
    dL_dz2 = dL_dy * dy_dz2
    dw2 = [dL_dz2 * a for a in a1]
    db2 = dL_dz2
    dz2_da1 = w2
    dL_da1 = dL_dz2 * dz2_da1
    da1_dz1 = [a * (1 - a) for a in a1]
    dL_dz1 = [dL_da1_i * da1_dz1_i for dL_da1_i, da1_dz1_i in zip(dL_da1, da1_dz1)]
    dw1 = [[dL_dz1_j * x_i for x_i in x] for dL_dz1_j in dL_dz1]
    db1 = dL_dz1
    return dw1, db1, dw2, db2

PyTorch implementation abstracts this into torch.nn modules:

import torch
import torch.nn as nn

class TwoLayerNet(nn.Module):
    def __init__(self, in_dim, hidden_dim):
        super().__init__()
        self.fc1 = nn.Linear(in_dim, hidden_dim)
        self.fc2 = nn.Linear(hidden_dim, 1)

    def forward(self, x):
        a1 = torch.sigmoid(self.fc1(x))
        y_hat = torch.sigmoid(self.fc2(a1))
        return y_hat

net = TwoLayerNet(2, 4)
criterion = nn.BCELoss()
optimizer = torch.optim.SGD(net.parameters(), lr=0.1)

x = torch.tensor([[0., 1.], [1., 0.], [1., 1.], [0., 0.]])
y = torch.tensor([[1.], [1.], [0.], [0.]])

for epoch in range(500):
    optimizer.zero_grad()
    preds = net(x)
    loss = criterion(preds, y)
    loss.backward()  # Automatic backpropagation

    optimizer.step()

Key Implementation Files

The repository structure reveals the deliberate progression:

• phases/03-deep-learning-core/03-backpropagation/code/main.py – Python implementation with explicit forward() and backward() functions
• phases/03-deep-learning-core/03-backpropagation/code/main.jl – Julia implementation demonstrating language-agnostic concepts
• phases/03-deep-learning-core/10-mini-framework/code/main.py – Refactored mini-framework with modular network classes
• phases/03-deep-learning-core/11-intro-to-pytorch/code/main.py – PyTorch nn.Module implementation using torch.autograd

Each file builds upon the previous, with the manual versions requiring students to manage the computation graph and gradient storage manually, while the PyTorch version delegates these tasks to autograd.

Summary

• AI Engineering From Scratch employs a build-first methodology where students implement backpropagation manually before using PyTorch
• The curriculum progresses from mathematical theory to manual Python implementation, then to a framework-free mini-framework, and finally to PyTorch
• Key skills developed include explicit chain-rule application, gradient flow visualization, and memory management through manual caching
• Source files in phases/03-deep-learning-core/ demonstrate the transition from hand-coded backward() functions to loss.backward()
• By Lesson 11, students understand that PyTorch's autograd is an automation tool for the mathematics they previously implemented explicitly

Frequently Asked Questions

Why learn backpropagation manually before PyTorch?

Manual implementation forces you to trace every gradient calculation through the chain rule, revealing how information flows backward through the network. When you later call loss.backward() in PyTorch, you recognize the automation as a convenience rather than magic, enabling better debugging and architecture design.

What programming languages does AI Engineering From Scratch use for backpropagation?

The repository provides implementations in both Python and Julia, with parallel codebases in main.py and main.jl files. This dual-language approach reinforces that backpropagation is a mathematical concept independent of any specific deep learning framework.

Which lesson introduces PyTorch?

PyTorch first appears in Phase 03 | Lesson 11 titled "Intro to PyTorch," located at phases/03-deep-learning-core/11-intro-to-pytorch/. This lesson follows ten prior lessons covering mathematical foundations, manual implementations, and framework construction.

How does manual backpropagation differ from PyTorch autograd?

Manual backpropagation requires you to explicitly cache intermediate values during the forward pass and compute gradients using the chain rule in the backward() function. PyTorch's autograd automatically constructs a dynamic computation graph during forward() and computes gradients when you call backward(), handling the chain rule and memory management internally.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}