# Building Backpropagation from First Principles without Frameworks: A Complete Guide

> Implement backpropagation from scratch. Build a Value class to track operations and compute gradients without frameworks like PyTorch or TensorFlow. A complete guide to reverse-mode autodiff.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: deep-dive
- Published: 2026-07-25

---

**You can build a fully functional reverse-mode automatic differentiation engine by implementing a `Value` class that tracks operations, builds a computational graph, and traverses it backwards using topological sorting to accumulate gradients—no PyTorch or TensorFlow required.**

The `rohitg00/ai-engineering-from-scratch` repository provides a comprehensive walkthrough of building backpropagation from first principles without frameworks in Phase 03, Lesson 03. This educational module demonstrates how to construct every component of a neural network's learning algorithm—from computational graph nodes to gradient propagation—using only fundamental data structures, giving you complete visibility into how automatic differentiation actually works before touching any high-level library.

## The Mathematical Foundation: Chain Rule and Computational Graphs

At the heart of backpropagation lies the **chain rule**, which states that the gradient of the loss with respect to any weight equals the product of local derivatives traced back through the computational graph. The lesson narrative in [`phases/03-deep-learning-core/03-backpropagation/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/03-backpropagation/docs/en.md) explains how this principle enables efficient gradient computation for arbitrary network architectures.

### Tracing Gradients Through the Network

Each operation in a neural network creates a node in a directed acyclic graph. When the loss is computed at the output, gradients flow backwards along these edges, multiplying local derivatives at each step. This reverse-mode accumulation allows you to calculate all parameter gradients in a single pass after the forward computation completes.

### The `Value` Node Architecture

The engine centers on a `Value` class defined in `phases/03-deep-learning-core/03-backpropagation/code/main.jl` (lines 45‑53). Each instance stores:

- `data`: The forward-pass numerical value
- `grad`: The accumulated gradient (initialized to 0.0)
- `_backward`: A closure implementing the local gradient rule
- `_children`: References to parent nodes in the graph
- `_op`: The operation string for debugging

```python
class Value:
    def __init__(self, data, children=(), op=''):
        self.data = data          # forward value

        self.grad = 0.0           # accumulated gradient

        self._backward = lambda: None
        self._children = set(children)
        self._op = op

```

## Implementing the Autograd Engine from Scratch

Building backpropagation from first principles requires implementing primitive operations with attached backward functions, then orchestrating their execution in the correct order.

### Primitive Operations and Backward Functions

Every arithmetic operation creates a new `Value` node and defines how to route gradients to its inputs. For multiplication, implemented in `main.jl` (lines 76‑85), the backward function distributes the upstream gradient weighted by the opposite operand’s data:

```python
def __mul__(self, other):
    other = other if isinstance(other, Value) else Value(other)
    out = Value(self.data * other.data, (self, other), '*')
    def _backward():
        self.grad += other.data * out.grad
        other.grad += self.data * out.grad
    out._backward = _backward
    return out

```

Addition passes the gradient through unchanged to both operands, while activation functions like sigmoid incorporate their specific derivatives (capped at 0.25, which the documentation notes causes vanishing gradients in deep networks).

### Topological Sort for Reverse Mode

The `backward` method in `main.jl` (lines 26‑42) performs a depth‑first search to generate a reverse‑topologically sorted list of nodes. This guarantees that when a node’s `_backward` function executes, all downstream gradients have already accumulated in `out.grad`:

```python
def backward(self):
    topo = []
    visited = set()
    def build_topo(v):
        if v not in visited:
            visited.add(v)
            for child in v._children:
                build_topo(child)
            topo.append(v)
    build_topo(self)
    
    self.grad = 1.0
    for node in reversed(topo):
        node._backward()

```

### Layer Abstractions (Neuron, Layer, Network)

The implementation provides composable building blocks in `main.jl` (lines 48‑99). A `Neuron` holds `Value` weights and a bias, a `Layer` contains multiple neurons, and a `Network` stacks layers. The forward pass uses overloaded `Value` operators, automatically constructing the graph for later differentiation:

```python
class Neuron:
    def __init__(self, nin):
        self.w = [Value(random.uniform(-1,1)) for _ in range(nin)]
        self.b = Value(random.uniform(-1,1))
    
    def __call__(self, x):
        act = sum((wi*xi for wi, xi in zip(self.w, x)), self.b)
        return act.sigmoid()

```

## Training Neural Networks with Manual Backpropagation

Once the engine is built, training proceeds identically to framework-based workflows, only with explicit gradient management.

### Solving the XOR Problem

The lesson demonstrates training on the XOR dataset using the hand‑crafted engine in `main.jl` (lines 106‑142). The training loop manually zeros gradients, calls `backward()`, and applies SGD updates:

```python
net = Network([2, 4, 1])                     # 2‑in, 4‑hidden, 1‑out

xor_data = [([0,0],0), ([0,1],1), ([1,0],1), ([1,1],0)]
lr = 1.0
for epoch in range(1000):
    total = Value(0.0)
    for x_vals, y in xor_data:
        x = [Value(v) for v in x_vals]
        pred = net(x)
        loss = (pred + Value(-y)) * (pred + Value(-y))   # MSE

        total = total + loss
    net.zero_grad()
    total.backward()
    for p in net.parameters():
        p.data -= lr * p.grad

```

### Manual SGD Updates

Unlike PyTorch’s `optimizer.step()`, this implementation exposes the parameter update explicitly (`p.data -= lr * p.grad`). This transparency reveals exactly how gradients affect the weights during learning.

## From Scratch to PyTorch: Understanding the Abstraction

The lesson concludes with a PyTorch implementation in `main.jl` (lines 200‑224) that solves the identical XOR problem. The comparison demonstrates that `loss.backward()` and `optimizer.step()` perform the same topological traversal and gradient accumulation hidden behind the API:

```python
import torch, torch.nn as nn
model = nn.Sequential(nn.Linear(2,4), nn.Sigmoid(),
                      nn.Linear(4,1), nn.Sigmoid())
optimizer = torch.optim.SGD(model.parameters(), lr=1.0)
criterion = nn.MSELoss()

for epoch in range(1000):
    pred = model(X)
    loss = criterion(pred, y)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

```

Understanding the hand‑written engine clarifies what PyTorch abstracts: the computational graph construction, the topological sort, and the per‑parameter gradient accumulation.

## Debugging Vanishing Gradients

The documentation in [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md) specifically analyzes why sigmoid activations struggle in deep networks—their derivative reaches a maximum of 0.25, causing gradients to shrink exponentially during backpropagation. This insight becomes tangible when you implement the sigmoid backward function and observe the gradient values directly in the `Value.grad` fields.

## Summary

- The `Value` class in `phases/03-deep-learning-core/03-backpropagation/code/main.jl` (lines 45‑53) serves as the fundamental unit of the computational graph, recording operations via `_backward` closures.
- **Reverse‑mode autodiff** requires a topological sort (lines 26‑42) to ensure gradients propagate correctly from outputs to inputs in O(n) time.
- Primitive operations implement local gradient rules; multiplication routes `other.data * out.grad` to its left operand, as shown in lines 76‑85.
- Manual SGD updates (`p.data -= lr * p.grad`) execute the same parameter adjustments as PyTorch’s optimizer, but with full visibility into the gradient values.
- Building backpropagation from first principles without frameworks exposes the mechanics of vanishing gradients and validates that production frameworks perform identical mathematical operations under the hood.

## Frequently Asked Questions

### How does reverse-mode autodiff differ from forward-mode?

Reverse-mode autodiff computes gradients from outputs to inputs in a single backward pass, making it efficient for scalar‑to‑vector gradients (loss to parameters). Forward-mode propagates derivatives from inputs to outputs and becomes prohibitively expensive for neural networks with many parameters. The implementation uses reverse-mode via the topological sort in `main.jl` to ensure all parameter gradients are computed after one forward and one backward pass.

### Why does the sigmoid activation cause vanishing gradients in this implementation?

The sigmoid derivative caps at 0.25, as documented in the "Vanishing Gradients" section of [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md). When you chain multiple sigmoid layers, you multiply these small values repeatedly during backpropagation, causing gradients to vanish exponentially as they travel backward through the network. This becomes visible when printing `Value.grad` fields during training of deep architectures.

### Can this hand-written engine handle deep networks?

Yes, though with significant performance overhead compared to optimized frameworks. The topological sort in `backward()` correctly handles arbitrary graph depth and branching, but the Python interpreter overhead makes it impractical for production-scale models. It excels for educational purposes and for verifying gradient computations in small networks before implementing them in PyTorch or JAX.

### How does the topological sort ensure correct gradient accumulation?

The depth-first search in `main.jl` (lines 26‑42) generates a list where every node appears after all its children. When reversed, this list processes nodes from the loss backward to the inputs. By the time a node’s `_backward` function executes, `out.grad` already contains the complete derivative of the final loss with respect to that node’s output, ensuring gradients accumulate correctly before propagating to parents.