Building Backpropagation from First Principles without Frameworks: A Complete Guide

You can build a fully functional reverse-mode automatic differentiation engine by implementing a Value class that tracks operations, builds a computational graph, and traverses it backwards using topological sorting to accumulate gradients—no PyTorch or TensorFlow required.

The rohitg00/ai-engineering-from-scratch repository provides a comprehensive walkthrough of building backpropagation from first principles without frameworks in Phase 03, Lesson 03. This educational module demonstrates how to construct every component of a neural network's learning algorithm—from computational graph nodes to gradient propagation—using only fundamental data structures, giving you complete visibility into how automatic differentiation actually works before touching any high-level library.

The Mathematical Foundation: Chain Rule and Computational Graphs

At the heart of backpropagation lies the chain rule, which states that the gradient of the loss with respect to any weight equals the product of local derivatives traced back through the computational graph. The lesson narrative in phases/03-deep-learning-core/03-backpropagation/docs/en.md explains how this principle enables efficient gradient computation for arbitrary network architectures.

Tracing Gradients Through the Network

Each operation in a neural network creates a node in a directed acyclic graph. When the loss is computed at the output, gradients flow backwards along these edges, multiplying local derivatives at each step. This reverse-mode accumulation allows you to calculate all parameter gradients in a single pass after the forward computation completes.

The Value Node Architecture

The engine centers on a Value class defined in phases/03-deep-learning-core/03-backpropagation/code/main.jl (lines 45‑53). Each instance stores:

• data: The forward-pass numerical value
• grad: The accumulated gradient (initialized to 0.0)
• _backward: A closure implementing the local gradient rule
• _children: References to parent nodes in the graph
• _op: The operation string for debugging

class Value:
    def __init__(self, data, children=(), op=''):
        self.data = data          # forward value

        self.grad = 0.0           # accumulated gradient

        self._backward = lambda: None
        self._children = set(children)
        self._op = op

Implementing the Autograd Engine from Scratch

Building backpropagation from first principles requires implementing primitive operations with attached backward functions, then orchestrating their execution in the correct order.

Primitive Operations and Backward Functions

Every arithmetic operation creates a new Value node and defines how to route gradients to its inputs. For multiplication, implemented in main.jl (lines 76‑85), the backward function distributes the upstream gradient weighted by the opposite operand’s data:

def __mul__(self, other):
    other = other if isinstance(other, Value) else Value(other)
    out = Value(self.data * other.data, (self, other), '*')
    def _backward():
        self.grad += other.data * out.grad
        other.grad += self.data * out.grad
    out._backward = _backward
    return out

Addition passes the gradient through unchanged to both operands, while activation functions like sigmoid incorporate their specific derivatives (capped at 0.25, which the documentation notes causes vanishing gradients in deep networks).

Topological Sort for Reverse Mode

The backward method in main.jl (lines 26‑42) performs a depth‑first search to generate a reverse‑topologically sorted list of nodes. This guarantees that when a node’s _backward function executes, all downstream gradients have already accumulated in out.grad:

def backward(self):
    topo = []
    visited = set()
    def build_topo(v):
        if v not in visited:
            visited.add(v)
            for child in v._children:
                build_topo(child)
            topo.append(v)
    build_topo(self)
    
    self.grad = 1.0
    for node in reversed(topo):
        node._backward()

Layer Abstractions (Neuron, Layer, Network)

The implementation provides composable building blocks in main.jl (lines 48‑99). A Neuron holds Value weights and a bias, a Layer contains multiple neurons, and a Network stacks layers. The forward pass uses overloaded Value operators, automatically constructing the graph for later differentiation:

class Neuron:
    def __init__(self, nin):
        self.w = [Value(random.uniform(-1,1)) for _ in range(nin)]
        self.b = Value(random.uniform(-1,1))
    
    def __call__(self, x):
        act = sum((wi*xi for wi, xi in zip(self.w, x)), self.b)
        return act.sigmoid()

Training Neural Networks with Manual Backpropagation

Once the engine is built, training proceeds identically to framework-based workflows, only with explicit gradient management.

Solving the XOR Problem

The lesson demonstrates training on the XOR dataset using the hand‑crafted engine in main.jl (lines 106‑142). The training loop manually zeros gradients, calls backward(), and applies SGD updates:

net = Network([2, 4, 1])                     # 2‑in, 4‑hidden, 1‑out

xor_data = [([0,0],0), ([0,1],1), ([1,0],1), ([1,1],0)]
lr = 1.0
for epoch in range(1000):
    total = Value(0.0)
    for x_vals, y in xor_data:
        x = [Value(v) for v in x_vals]
        pred = net(x)
        loss = (pred + Value(-y)) * (pred + Value(-y))   # MSE

        total = total + loss
    net.zero_grad()
    total.backward()
    for p in net.parameters():
        p.data -= lr * p.grad

Manual SGD Updates

Unlike PyTorch’s optimizer.step(), this implementation exposes the parameter update explicitly (p.data -= lr * p.grad). This transparency reveals exactly how gradients affect the weights during learning.

From Scratch to PyTorch: Understanding the Abstraction

The lesson concludes with a PyTorch implementation in main.jl (lines 200‑224) that solves the identical XOR problem. The comparison demonstrates that loss.backward() and optimizer.step() perform the same topological traversal and gradient accumulation hidden behind the API:

import torch, torch.nn as nn
model = nn.Sequential(nn.Linear(2,4), nn.Sigmoid(),
                      nn.Linear(4,1), nn.Sigmoid())
optimizer = torch.optim.SGD(model.parameters(), lr=1.0)
criterion = nn.MSELoss()

for epoch in range(1000):
    pred = model(X)
    loss = criterion(pred, y)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

Understanding the hand‑written engine clarifies what PyTorch abstracts: the computational graph construction, the topological sort, and the per‑parameter gradient accumulation.

Debugging Vanishing Gradients

The documentation in docs/en.md specifically analyzes why sigmoid activations struggle in deep networks—their derivative reaches a maximum of 0.25, causing gradients to shrink exponentially during backpropagation. This insight becomes tangible when you implement the sigmoid backward function and observe the gradient values directly in the Value.grad fields.

Summary

• The Value class in phases/03-deep-learning-core/03-backpropagation/code/main.jl (lines 45‑53) serves as the fundamental unit of the computational graph, recording operations via _backward closures.
• Reverse‑mode autodiff requires a topological sort (lines 26‑42) to ensure gradients propagate correctly from outputs to inputs in O(n) time.
• Primitive operations implement local gradient rules; multiplication routes other.data * out.grad to its left operand, as shown in lines 76‑85.
• Manual SGD updates (p.data -= lr * p.grad) execute the same parameter adjustments as PyTorch’s optimizer, but with full visibility into the gradient values.
• Building backpropagation from first principles without frameworks exposes the mechanics of vanishing gradients and validates that production frameworks perform identical mathematical operations under the hood.

Frequently Asked Questions

How does reverse-mode autodiff differ from forward-mode?

Reverse-mode autodiff computes gradients from outputs to inputs in a single backward pass, making it efficient for scalar‑to‑vector gradients (loss to parameters). Forward-mode propagates derivatives from inputs to outputs and becomes prohibitively expensive for neural networks with many parameters. The implementation uses reverse-mode via the topological sort in main.jl to ensure all parameter gradients are computed after one forward and one backward pass.

Why does the sigmoid activation cause vanishing gradients in this implementation?

The sigmoid derivative caps at 0.25, as documented in the "Vanishing Gradients" section of docs/en.md. When you chain multiple sigmoid layers, you multiply these small values repeatedly during backpropagation, causing gradients to vanish exponentially as they travel backward through the network. This becomes visible when printing Value.grad fields during training of deep architectures.

Can this hand-written engine handle deep networks?

Yes, though with significant performance overhead compared to optimized frameworks. The topological sort in backward() correctly handles arbitrary graph depth and branching, but the Python interpreter overhead makes it impractical for production-scale models. It excels for educational purposes and for verifying gradient computations in small networks before implementing them in PyTorch or JAX.

How does the topological sort ensure correct gradient accumulation?

The depth-first search in main.jl (lines 26‑42) generates a list where every node appears after all its children. When reversed, this list processes nodes from the loss backward to the inputs. By the time a node’s _backward function executes, out.grad already contains the complete derivative of the final loss with respect to that node’s output, ensuring gradients accumulate correctly before propagating to parents.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}