How Backpropagation Is Implemented From Scratch in the AI-Engineering-From-Scratch Curriculum

The AI-Engineering-From-Scratch curriculum implements backpropagation through a minimal autograd engine that wraps scalars in Value objects, builds a computational graph via operator overloading, and executes reverse-mode automatic differentiation using topological sorting to compute gradients for every parameter in a single backward pass.

The backpropagation from scratch lesson (Phase 03, Lesson 03) in the rohitg00/ai-engineering-from-scratch repository provides a complete pure-Python implementation of reverse-mode automatic differentiation. Located in phases/03-deep-learning-core/03-backpropagation/code/main.py, this educational module reconstructs the core mechanics behind modern deep learning frameworks like PyTorch, demonstrating exactly how gradients flow from a loss value back to every weight in a neural network without external dependencies.

The Value Node: Foundation of the Autograd Engine

Storing Data, Gradients, and Graph Edges

At the heart of the implementation is the Value class defined in phases/03-deep-learning-core/03-backpropagation/code/main.py (lines 5-15). Every scalar quantity in the network—whether an input, weight, bias, or intermediate activation—is encapsulated in a Value object that tracks four critical pieces of state:

• data: The raw scalar value (e.g., the result of a multiplication).
• grad: The accumulated gradient, initialized to 0.0, representing the partial derivative of the final loss with respect to this node.
• _prev: A set of Value objects that are the parents (children in the computational graph) of this operation.
• _op: A string label indicating which operation produced this node (e.g., '+', '*', or 'sigmoid').

This structure allows the engine to treat every arithmetic operation as a node in a directed acyclic graph (DAG), where edges represent data dependencies required for gradient computation.

Building the Computational Graph

Operator Overloading and Local Gradient Rules

To construct the graph dynamically, the curriculum overloads Python's arithmetic operators. In phases/03-deep-learning-core/03-backpropagation/code/main.py (lines 16-38), the __add__ and __mul__ methods do not merely return numbers; they create new Value nodes and attach a custom _backward closure to each.

For addition, the _backward closure implements the local gradient rule: the gradient flows equally to both inputs. For multiplication, it implements the product rule, scaling each input's gradient by the other input's data.

Similarly, the sigmoid activation (lines 50-59) creates a new Value and defines its _backward to propagate gradients using the derivative formula sigmoid(x) * (1 - sigmoid(x)). These closures are stored but not executed during the forward pass; they act as deferred instructions for the reverse pass.

Executing Reverse-Mode Differentiation

Topological Sort and the Backward Pass

The actual backpropagation from scratch algorithm is implemented in the backward() method (lines 61-75 of main.py). Because gradients must flow from outputs to inputs, the engine must process nodes in reverse topological order—ensuring that a node's gradient is fully accumulated before it propagates gradients to its parents.

The method implements this via a recursive depth-first search:

1. Build a topologically sorted list (topo) starting from the loss node.
2. Reverse this list so the loss comes first.
3. Iterate through the reversed list, calling each node's stored _backward() closure.

This single pass efficiently computes dL/dw for every parameter w in the network, exactly mirroring the reverse-mode autodiff algorithm used in production frameworks.

Neural Network Architecture on Top of Value

Composing Neurons, Layers, and Networks

The curriculum demonstrates how to build higher-level abstractions using the Value primitive. In phases/03-deep-learning-core/03-backpropagation/code/main.py (lines 83-130), three classes provide a familiar API:

• Neuron: Holds a list of Value-wrapped weights and a bias. Its __call__ method computes the dot product of inputs and weights, adds the bias, and applies a nonlinearity.
• Layer: Groups multiple Neuron instances and processes them in parallel.
• Network: Stacks Layer objects to form a multi-layer perceptron.

Because every weight is a Value and every operation uses the overloaded operators, calling Network(x) automatically constructs the full computational graph required for backpropagation, with the final output being a single Value object representing the prediction.

Training with Backpropagation: The XOR Example

End-to-End Gradient Computation

The repository includes a complete training demonstration in phases/03-deep-learning-core/03-backpropagation/code/main.py (lines 36-70) that solves the XOR problem. The training loop follows the standard four-step pattern:

1. Forward pass: Compute predictions using net(x).
2. Loss calculation: Compute mean squared error (MSE) using mse_loss(pred, target), which returns a Value object representing the loss.
3. Backward pass: Call loss.backward() to trigger the topological sort and propagate gradients back through the entire graph.
4. Parameter update: Perform a simple SGD step: p.data -= lr * p.grad for every parameter p, followed by zero_grad() to reset gradients for the next iteration.

# Example: Training loop structure from the curriculum

net = Network([2, 4, 1])  # 2 inputs, 4 hidden, 1 output

# Single training step

x = [Value(0.0), Value(1.0)]  # XOR input (0, 1)

pred = net(x)
loss = mse_loss(pred, 1.0)    # Target is 1

loss.backward()               # Compute all gradients

# Update weights manually

for p in net.parameters():
    p.data -= 0.1 * p.grad

This implementation proves that the from-scratch autograd engine can successfully train a network to learn non-linear decision boundaries, including the circular classification dataset also provided in the lesson.

Summary

• The Value class in phases/03-deep-learning-core/03-backpropagation/code/main.py (lines 5-15) encapsulates scalars, gradients, and parent pointers to form the nodes of a computational graph.
• Operator overloading on __add__, __mul__, and sigmoid (lines 16-59) attaches _backward closures that encode local gradient rules without executing them immediately.
• Topological sorting in the backward() method (lines 61-75) ensures gradients propagate correctly from the loss node to every input via reverse-mode automatic differentiation.
• The Neuron, Layer, and Network classes (lines 83-130) demonstrate how to compose the autograd primitives into a trainable neural network architecture.
• The XOR training loop (lines 36-70) validates the implementation by successfully optimizing parameters using the computed gradients.

Frequently Asked Questions

What file contains the backpropagation implementation?

The core implementation resides in phases/03-deep-learning-core/03-backpropagation/code/main.py within the repository. This file contains the Value class, operator definitions, topological backward pass, and neural network classes.

How does the engine handle the chain rule?

The engine implements the chain rule through the combination of _backward closures and topological ordering. Each operation stores a closure that computes local gradients (the derivative of the operation's output with respect to its inputs). During the backward pass, these closures multiply the incoming gradient (from upstream) by the local gradient, effectively applying the chain rule as gradients flow backward through the graph.

Where is the mathematical theory documented?

The conceptual explanation of the chain rule, computational graphs, and vanishing gradients is documented in phases/03-deep-learning-core/03-backpropagation/docs/en.md. This markdown file accompanies the code and provides the mathematical derivation for a two-layer network.

Does this implementation support vectorized operations?

No, this educational implementation focuses on scalar Value objects for clarity. Each neuron computes weighted sums and activations using individual scalar operations rather than matrix multiplications. While less efficient than vectorized frameworks like NumPy or PyTorch, this scalar approach makes the gradient flow explicit and easier to debug for learning purposes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}