# How to Implement Backpropagation from Scratch Using Python

> Learn to implement backpropagation from scratch in Python with NumPy. Build a computational graph and traverse it to compute derivatives for every parameter.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: how-to-guide
- Published: 2026-08-28

---

**You can implement backpropagation from scratch by building a computational graph with a `Value` class that tracks both data and gradients, then traversing the graph in reverse topological order to apply the chain rule and compute derivatives for every parameter.**

The repository `rohitg00/ai-engineering-from-scratch` provides a complete educational implementation of reverse-mode automatic differentiation—the algorithm powering modern neural networks—using only pure Python. This from-scratch approach demonstrates how deep learning frameworks compute gradients without relying on TensorFlow, PyTorch, or NumPy, storing scalar values and their derivatives in custom objects that form a dynamic computation graph.

## The Value Class: Building the Computational Graph

The core of the engine resides in [`phases/03-deep-learning-core/03-backpropagation/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/03-backpropagation/code/main.py), where the **`Value`** class serves as the fundamental unit of computation. Each instance wraps a scalar value while maintaining the metadata required for gradient computation.

### Forward Values and Gradient Storage

Every `Value` object stores four critical attributes:
- `data`: The scalar result of the forward pass
- `grad`: The accumulated gradient (initialized to 0)
- `_backward`: A closure containing the local gradient computation
- `_prev`: A set of parent `Value` objects forming the computation graph edges

```python
class Value:
    def __init__(self, data, _children=(), _op=''):
        self.data = data
        self.grad = 0
        self._backward = lambda: None
        self._prev = set(_children)
        self._op = _op  # Stores the operation type for debugging

```

### Operator Overloading and Local Gradients

Arithmetic operations construct the computation graph dynamically. When multiplying two `Value` objects, the implementation stores references to the operands and defines how to propagate gradients backward through the chain rule:

```python
def __mul__(self, other):
    other = other if isinstance(other, Value) else Value(other)
    out = Value(self.data * other.data, (self, other), '*')
    
    def _backward():
        # Chain rule: dL/dx = dL/dz * dz/dx

        self.grad += other.data * out.grad
        other.grad += self.data * out.grad
    out._backward = _backward
    
    return out

```

## Reverse-Mode Automatic Differentiation

Backpropagation is mathematically equivalent to **reverse-mode automatic differentiation**, which computes the gradient of a scalar output with respect to all inputs in a single pass.

### Topological Sorting for Correct Ordering

The `backward()` method constructs a topologically sorted list of all nodes in the computation graph, ensuring that a node is processed only after all its children have received their gradients:

```python
def backward(self):
    topo = []
    visited = set()
    def build_topo(v):
        if v not in visited:
            visited.add(v)
            for child in v._prev:
                build_topo(child)
            topo.append(v)
    build_topo(self)
    
    self.grad = 1  # Seed gradient at the output node

    for node in reversed(topo):
        node._backward()

```

### Gradient Propagation via the Chain Rule

When `loss.backward()` is invoked, the algorithm applies the chain rule recursively: if $z = f(x, y)$ and we know $\frac{\partial L}{\partial z}$, we compute $\frac{\partial L}{\partial x} = \frac{\partial L}{\partial z} \cdot \frac{\partial z}{\partial x}$. The `_backward` closures store the local Jacobian $\frac{\partial z}{\partial x}$, while `out.grad` carries the upstream gradient $\frac{\partial L}{\partial z}$.

## Neural Network Architecture

The engine constructs networks using three composable classes built entirely from `Value` objects, avoiding matrix libraries in favor of scalar operations that expose the underlying mechanics.

### Neuron, Layer, and Network Classes

The **`Neuron`** class maintains a list of weight `Value` objects and a bias `Value`, computing the dot product of inputs with weights plus bias:

```python
class Neuron:
    def __init__(self, nin):
        self.w = [Value(random.uniform(-1,1)) for _ in range(nin)]
        self.b = Value(random.uniform(-1,1))
    
    def __call__(self, x):
        # Weighted sum plus bias

        act = sum((wi*xi for wi,xi in zip(self.w, x)), self.b)
        return act.sigmoid()  # Activation function

```

A **`Layer`** aggregates multiple neurons into a list, and a **`Network`** chains layers sequentially, feeding the output list of one layer as the input to the next.

## Training Loop Implementation

The complete training cycle demonstrates the integration of forward propagation, backpropagation, and gradient descent.

### Zeroing Gradients and Computing Loss

Before each backward pass, the `Network.zero_grad()` method clears accumulated gradients from previous iterations to prevent incorrect accumulation:

```python
def zero_grad(self):
    for p in self.parameters():
        p.grad = 0

```

The forward pass computes predictions and loss (mean squared error via `mse_loss`), then initiates backpropagation:

```python
y_pred = [net(x) for x in xs]
total_loss = mse_loss(y_pred, ys)
total_loss.backward()  # Populates .grad for every parameter

```

### Gradient Descent Parameter Updates

After `backward()` computes all gradients, standard gradient descent updates each parameter:

```python
learning_rate = 0.1
for p in net.parameters():
    p.data -= learning_rate * p.grad

```

## Practical Examples

The repository validates the engine with two classic classification tasks that require non-linear decision boundaries.

### XOR Truth Table Classification

The XOR demo trains a network with architecture `[2, 4, 1]` (2 inputs, 4 hidden units, 1 output) to solve the non-linear XOR problem:

```python
from phases/03-deep-learning-core/03-backpropagation/code/main import train_xor

train_xor()

# Output demonstrates convergence:

# [0.0, 0.0] -> 0.0147 (rounded: 0, expected 0)

# [0.0, 1.0] -> 0.9843 (rounded: 1, expected 1)

```

### Circle Boundary Detection

The circle classifier uses architecture `[2, 8, 1]` to learn a circular decision boundary, separating points inside (`label 1`) from outside (`label 0`) a unit circle:

```python
from phases/03-deep-learning-core/03-backpropagation/code.main import train_circle

train_circle()

# After 2000 epochs, the network accurately classifies points

```

## Summary

- **Pure Python Implementation**: The entire engine in [`phases/03-deep-learning-core/03-backpropagation/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/03-backpropagation/code/main.py) implements backpropagation without NumPy or deep learning frameworks, using only scalar `Value` objects.
- **Computational Graph**: The `Value` class builds a directed acyclic graph (DAG) through operator overloading, storing parent references in `_prev` and gradient functions in `_backward`.
- **Topological Traversal**: The `backward()` method uses depth-first search to order nodes such that gradients flow from outputs to inputs according to the chain rule.
- **Modular Design**: `Neuron`, `Layer`, and `Network` classes compose `Value` objects into trainable architectures supporting arbitrary depth and width.
- **End-to-End Training**: The standard ML training loop—`zero_grad()`, forward pass, `backward()`, and parameter update—works identically to PyTorch but with transparent, educational code.

## Frequently Asked Questions

### Why doesn't this implementation use NumPy?

This implementation uses pure Python to maximize educational transparency. While NumPy provides vectorized operations that would improve performance, scalar operations in the `Value` class make the chain rule and gradient flow explicit and debuggable. You can trace individual gradient values through every operation, which becomes opaque when using matrix multiplication kernels.

### How does the topological sort handle complex graph structures?

The `build_topo` recursive function guarantees that a node appears in the list only after all its children (nodes that depend on it) have been processed. This works for any directed acyclic graph structure, including networks with skip connections or shared weights, because it respects the data dependency direction encoded in `_prev`.

### Can I extend this engine with modern activations like ReLU or LayerNorm?

Yes, you can extend the `Value` class by adding methods that implement both the forward operation and its local gradient. For ReLU, define a method that returns `self` if `self.data > 0` else `Value(0)`, with a `_backward` closure that passes `out.grad` through only if the input was positive. LayerNorm would require implementing mean and variance computations as `Value` operations.

### What is the difference between `grad` and `_backward`?

`grad` stores the accumulated gradient value $\frac{\partial L}{\partial v}$ for that specific node, while `_backward` is a function (closure) that defines how to propagate the gradient to parent nodes. When `backward()` runs, it calls `_backward` on each node to execute the local chain rule application, adding the result to each parent's `grad` attribute.