# How Backpropagation Is Implemented From Scratch in the AI-Engineering-From-Scratch Curriculum

> Learn how backpropagation is implemented from scratch with a minimal autograd engine. Explore computational graphs and reverse-mode automatic differentiation for efficient gradient calculation.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: deep-dive
- Published: 2026-08-30

---

**The AI-Engineering-From-Scratch curriculum implements backpropagation through a minimal autograd engine that wraps scalars in `Value` objects, builds a computational graph via operator overloading, and executes reverse-mode automatic differentiation using topological sorting to compute gradients for every parameter in a single backward pass.**

The **backpropagation from scratch** lesson (Phase 03, Lesson 03) in the `rohitg00/ai-engineering-from-scratch` repository provides a complete pure-Python implementation of reverse-mode automatic differentiation. Located in [`phases/03-deep-learning-core/03-backpropagation/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/03-backpropagation/code/main.py), this educational module reconstructs the core mechanics behind modern deep learning frameworks like PyTorch, demonstrating exactly how gradients flow from a loss value back to every weight in a neural network without external dependencies.

## The Value Node: Foundation of the Autograd Engine

### Storing Data, Gradients, and Graph Edges

At the heart of the implementation is the `Value` class defined in [`phases/03-deep-learning-core/03-backpropagation/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/03-backpropagation/code/main.py) (lines 5-15). Every scalar quantity in the network—whether an input, weight, bias, or intermediate activation—is encapsulated in a `Value` object that tracks four critical pieces of state:

- **`data`**: The raw scalar value (e.g., the result of a multiplication).
- **`grad`**: The accumulated gradient, initialized to 0.0, representing the partial derivative of the final loss with respect to this node.
- **`_prev`**: A set of `Value` objects that are the parents (children in the computational graph) of this operation.
- **`_op`**: A string label indicating which operation produced this node (e.g., `'+'`, `'*'`, or `'sigmoid'`).

This structure allows the engine to treat every arithmetic operation as a node in a directed acyclic graph (DAG), where edges represent data dependencies required for gradient computation.

## Building the Computational Graph

### Operator Overloading and Local Gradient Rules

To construct the graph dynamically, the curriculum overloads Python's arithmetic operators. In [`phases/03-deep-learning-core/03-backpropagation/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/03-backpropagation/code/main.py) (lines 16-38), the `__add__` and `__mul__` methods do not merely return numbers; they create new `Value` nodes and attach a custom `_backward` closure to each.

For addition, the `_backward` closure implements the local gradient rule: the gradient flows equally to both inputs. For multiplication, it implements the product rule, scaling each input's gradient by the other input's data.

Similarly, the `sigmoid` activation (lines 50-59) creates a new `Value` and defines its `_backward` to propagate gradients using the derivative formula `sigmoid(x) * (1 - sigmoid(x))`. These closures are stored but not executed during the forward pass; they act as deferred instructions for the reverse pass.

## Executing Reverse-Mode Differentiation

### Topological Sort and the Backward Pass

The actual **backpropagation from scratch** algorithm is implemented in the `backward()` method (lines 61-75 of [`main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/main.py)). Because gradients must flow from outputs to inputs, the engine must process nodes in reverse topological order—ensuring that a node's gradient is fully accumulated before it propagates gradients to its parents.

The method implements this via a recursive depth-first search:

1. Build a topologically sorted list (`topo`) starting from the loss node.
2. Reverse this list so the loss comes first.
3. Iterate through the reversed list, calling each node's stored `_backward()` closure.

This single pass efficiently computes `dL/dw` for every parameter `w` in the network, exactly mirroring the reverse-mode autodiff algorithm used in production frameworks.

## Neural Network Architecture on Top of Value

### Composing Neurons, Layers, and Networks

The curriculum demonstrates how to build higher-level abstractions using the `Value` primitive. In [`phases/03-deep-learning-core/03-backpropagation/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/03-backpropagation/code/main.py) (lines 83-130), three classes provide a familiar API:

- **`Neuron`**: Holds a list of `Value`-wrapped weights and a bias. Its `__call__` method computes the dot product of inputs and weights, adds the bias, and applies a nonlinearity.
- **`Layer`**: Groups multiple `Neuron` instances and processes them in parallel.
- **`Network`**: Stacks `Layer` objects to form a multi-layer perceptron.

Because every weight is a `Value` and every operation uses the overloaded operators, calling `Network(x)` automatically constructs the full computational graph required for backpropagation, with the final output being a single `Value` object representing the prediction.

## Training with Backpropagation: The XOR Example

### End-to-End Gradient Computation

The repository includes a complete training demonstration in [`phases/03-deep-learning-core/03-backpropagation/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/03-backpropagation/code/main.py) (lines 36-70) that solves the XOR problem. The training loop follows the standard four-step pattern:

1. **Forward pass**: Compute predictions using `net(x)`.
2. **Loss calculation**: Compute mean squared error (MSE) using `mse_loss(pred, target)`, which returns a `Value` object representing the loss.
3. **Backward pass**: Call `loss.backward()` to trigger the topological sort and propagate gradients back through the entire graph.
4. **Parameter update**: Perform a simple SGD step: `p.data -= lr * p.grad` for every parameter `p`, followed by `zero_grad()` to reset gradients for the next iteration.

```python

# Example: Training loop structure from the curriculum

net = Network([2, 4, 1])  # 2 inputs, 4 hidden, 1 output

# Single training step

x = [Value(0.0), Value(1.0)]  # XOR input (0, 1)

pred = net(x)
loss = mse_loss(pred, 1.0)    # Target is 1

loss.backward()               # Compute all gradients

# Update weights manually

for p in net.parameters():
    p.data -= 0.1 * p.grad

```

This implementation proves that the from-scratch autograd engine can successfully train a network to learn non-linear decision boundaries, including the circular classification dataset also provided in the lesson.

## Summary

- The **`Value` class** in [`phases/03-deep-learning-core/03-backpropagation/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/03-backpropagation/code/main.py) (lines 5-15) encapsulates scalars, gradients, and parent pointers to form the nodes of a computational graph.
- **Operator overloading** on `__add__`, `__mul__`, and `sigmoid` (lines 16-59) attaches `_backward` closures that encode local gradient rules without executing them immediately.
- **Topological sorting** in the `backward()` method (lines 61-75) ensures gradients propagate correctly from the loss node to every input via reverse-mode automatic differentiation.
- The **`Neuron`**, **`Layer`**, and **`Network`** classes (lines 83-130) demonstrate how to compose the autograd primitives into a trainable neural network architecture.
- The **XOR training loop** (lines 36-70) validates the implementation by successfully optimizing parameters using the computed gradients.

## Frequently Asked Questions

### What file contains the backpropagation implementation?

The core implementation resides in [`phases/03-deep-learning-core/03-backpropagation/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/03-backpropagation/code/main.py) within the repository. This file contains the `Value` class, operator definitions, topological backward pass, and neural network classes.

### How does the engine handle the chain rule?

The engine implements the chain rule through the combination of `_backward` closures and topological ordering. Each operation stores a closure that computes local gradients (the derivative of the operation's output with respect to its inputs). During the backward pass, these closures multiply the incoming gradient (from upstream) by the local gradient, effectively applying the chain rule as gradients flow backward through the graph.

### Where is the mathematical theory documented?

The conceptual explanation of the chain rule, computational graphs, and vanishing gradients is documented in [`phases/03-deep-learning-core/03-backpropagation/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/03-backpropagation/docs/en.md). This markdown file accompanies the code and provides the mathematical derivation for a two-layer network.

### Does this implementation support vectorized operations?

No, this educational implementation focuses on scalar `Value` objects for clarity. Each neuron computes weighted sums and activations using individual scalar operations rather than matrix multiplications. While less efficient than vectorized frameworks like NumPy or PyTorch, this scalar approach makes the gradient flow explicit and easier to debug for learning purposes.