How to Implement Backpropagation from Scratch Using Python

You can implement backpropagation from scratch by building a computational graph with a Value class that tracks both data and gradients, then traversing the graph in reverse topological order to apply the chain rule and compute derivatives for every parameter.

The repository rohitg00/ai-engineering-from-scratch provides a complete educational implementation of reverse-mode automatic differentiation—the algorithm powering modern neural networks—using only pure Python. This from-scratch approach demonstrates how deep learning frameworks compute gradients without relying on TensorFlow, PyTorch, or NumPy, storing scalar values and their derivatives in custom objects that form a dynamic computation graph.

The Value Class: Building the Computational Graph

The core of the engine resides in phases/03-deep-learning-core/03-backpropagation/code/main.py, where the Value class serves as the fundamental unit of computation. Each instance wraps a scalar value while maintaining the metadata required for gradient computation.

Forward Values and Gradient Storage

Every Value object stores four critical attributes:

• data: The scalar result of the forward pass
• grad: The accumulated gradient (initialized to 0)
• _backward: A closure containing the local gradient computation
• _prev: A set of parent Value objects forming the computation graph edges

class Value:
    def __init__(self, data, _children=(), _op=''):
        self.data = data
        self.grad = 0
        self._backward = lambda: None
        self._prev = set(_children)
        self._op = _op  # Stores the operation type for debugging

Operator Overloading and Local Gradients

Arithmetic operations construct the computation graph dynamically. When multiplying two Value objects, the implementation stores references to the operands and defines how to propagate gradients backward through the chain rule:

def __mul__(self, other):
    other = other if isinstance(other, Value) else Value(other)
    out = Value(self.data * other.data, (self, other), '*')
    
    def _backward():
        # Chain rule: dL/dx = dL/dz * dz/dx

        self.grad += other.data * out.grad
        other.grad += self.data * out.grad
    out._backward = _backward
    
    return out

Reverse-Mode Automatic Differentiation

Backpropagation is mathematically equivalent to reverse-mode automatic differentiation, which computes the gradient of a scalar output with respect to all inputs in a single pass.

Topological Sorting for Correct Ordering

The backward() method constructs a topologically sorted list of all nodes in the computation graph, ensuring that a node is processed only after all its children have received their gradients:

def backward(self):
    topo = []
    visited = set()
    def build_topo(v):
        if v not in visited:
            visited.add(v)
            for child in v._prev:
                build_topo(child)
            topo.append(v)
    build_topo(self)
    
    self.grad = 1  # Seed gradient at the output node

    for node in reversed(topo):
        node._backward()

Gradient Propagation via the Chain Rule

When loss.backward() is invoked, the algorithm applies the chain rule recursively: if $z = f(x, y)$ and we know $\frac{\partial L}{\partial z}$, we compute $\frac{\partial L}{\partial x} = \frac{\partial L}{\partial z} \cdot \frac{\partial z}{\partial x}$. The _backward closures store the local Jacobian $\frac{\partial z}{\partial x}$, while out.grad carries the upstream gradient $\frac{\partial L}{\partial z}$.

Neural Network Architecture

The engine constructs networks using three composable classes built entirely from Value objects, avoiding matrix libraries in favor of scalar operations that expose the underlying mechanics.

Neuron, Layer, and Network Classes

The Neuron class maintains a list of weight Value objects and a bias Value, computing the dot product of inputs with weights plus bias:

class Neuron:
    def __init__(self, nin):
        self.w = [Value(random.uniform(-1,1)) for _ in range(nin)]
        self.b = Value(random.uniform(-1,1))
    
    def __call__(self, x):
        # Weighted sum plus bias

        act = sum((wi*xi for wi,xi in zip(self.w, x)), self.b)
        return act.sigmoid()  # Activation function

A Layer aggregates multiple neurons into a list, and a Network chains layers sequentially, feeding the output list of one layer as the input to the next.

Training Loop Implementation

The complete training cycle demonstrates the integration of forward propagation, backpropagation, and gradient descent.

Zeroing Gradients and Computing Loss

Before each backward pass, the Network.zero_grad() method clears accumulated gradients from previous iterations to prevent incorrect accumulation:

def zero_grad(self):
    for p in self.parameters():
        p.grad = 0

The forward pass computes predictions and loss (mean squared error via mse_loss), then initiates backpropagation:

y_pred = [net(x) for x in xs]
total_loss = mse_loss(y_pred, ys)
total_loss.backward()  # Populates .grad for every parameter

Gradient Descent Parameter Updates

After backward() computes all gradients, standard gradient descent updates each parameter:

learning_rate = 0.1
for p in net.parameters():
    p.data -= learning_rate * p.grad

Practical Examples

The repository validates the engine with two classic classification tasks that require non-linear decision boundaries.

XOR Truth Table Classification

The XOR demo trains a network with architecture [2, 4, 1] (2 inputs, 4 hidden units, 1 output) to solve the non-linear XOR problem:

from phases/03-deep-learning-core/03-backpropagation/code/main import train_xor

train_xor()

# Output demonstrates convergence:

# [0.0, 0.0] -> 0.0147 (rounded: 0, expected 0)

# [0.0, 1.0] -> 0.9843 (rounded: 1, expected 1)

Circle Boundary Detection

The circle classifier uses architecture [2, 8, 1] to learn a circular decision boundary, separating points inside (label 1) from outside (label 0) a unit circle:

from phases/03-deep-learning-core/03-backpropagation/code.main import train_circle

train_circle()

# After 2000 epochs, the network accurately classifies points

Summary

• Pure Python Implementation: The entire engine in phases/03-deep-learning-core/03-backpropagation/code/main.py implements backpropagation without NumPy or deep learning frameworks, using only scalar Value objects.
• Computational Graph: The Value class builds a directed acyclic graph (DAG) through operator overloading, storing parent references in _prev and gradient functions in _backward.
• Topological Traversal: The backward() method uses depth-first search to order nodes such that gradients flow from outputs to inputs according to the chain rule.
• Modular Design: Neuron, Layer, and Network classes compose Value objects into trainable architectures supporting arbitrary depth and width.
• End-to-End Training: The standard ML training loop—zero_grad(), forward pass, backward(), and parameter update—works identically to PyTorch but with transparent, educational code.

Frequently Asked Questions

Why doesn't this implementation use NumPy?

This implementation uses pure Python to maximize educational transparency. While NumPy provides vectorized operations that would improve performance, scalar operations in the Value class make the chain rule and gradient flow explicit and debuggable. You can trace individual gradient values through every operation, which becomes opaque when using matrix multiplication kernels.

How does the topological sort handle complex graph structures?

The build_topo recursive function guarantees that a node appears in the list only after all its children (nodes that depend on it) have been processed. This works for any directed acyclic graph structure, including networks with skip connections or shared weights, because it respects the data dependency direction encoded in _prev.

Can I extend this engine with modern activations like ReLU or LayerNorm?

Yes, you can extend the Value class by adding methods that implement both the forward operation and its local gradient. For ReLU, define a method that returns self if self.data > 0 else Value(0), with a _backward closure that passes out.grad through only if the input was positive. LayerNorm would require implementing mean and variance computations as Value operations.

What is the difference between grad and _backward?

grad stores the accumulated gradient value $\frac{\partial L}{\partial v}$ for that specific node, while _backward is a function (closure) that defines how to propagate the gradient to parent nodes. When backward() runs, it calls _backward on each node to execute the local chain rule application, adding the result to each parent's grad attribute.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}