# How Backpropagation Is Implemented in Micrograd: A Deep Dive into the Value Class

> Discover how micrograd implements backpropagation using its Value class. Learn about reverse-mode automatic differentiation and computation history in this deep dive.

- Repository: [Andrej/nn-zero-to-hero](https://github.com/karpathy/nn-zero-to-hero)
- Tags: deep-dive
- Published: 2026-05-23

---

**Micrograd implements reverse-mode automatic differentiation by wrapping scalar values in `Value` objects that record computation history and execute local gradient functions in topological order.**

Micrograd, the educational autograd engine built during Andrej Karpathy's "Neural Networks: Zero to Hero" course, demonstrates how backpropagation is implemented in micrograd from first principles using pure Python. This minimalist scalar-only framework constructs a dynamic computation graph through the `Value` class to enable gradient computation without external dependencies. The complete implementation resides in `lectures/micrograd/micrograd_lecture_first_half_roughly.ipynb`.

## The Value Object: Building Blocks of the Computation Graph

Each scalar in micrograd is wrapped in a `Value` instance that tracks five critical attributes:

- `data`: The forward-pass numeric value
- `grad`: The accumulated gradient ∂L/∂self
- `_prev`: A set of parent `Value` objects used to compute this value
- `_op`: A string describing the operation (e.g., `+`, `*`, `tanh`)
- `_backward`: A closure that propagates the output gradient to its parents

This structure transforms static numbers into nodes of a dynamic directed acyclic graph (DAG). When operations execute, they create new `Value` instances linking back to their operands via `_prev`, establishing the computation topology required for reverse-mode differentiation.

## Forward Pass: Recording Local Gradients

During the forward pass, each operation instantiates a new `Value` and records a **local backward function** encoding the partial derivatives. In `lectures/micrograd/micrograd_lecture_first_half_roughly.ipynb`, addition is implemented as:

```python
out = Value(self.data + other.data, (self, other), '+')

def _backward():
    self.grad += out.grad      # dL/dself = dL/dout * 1

    other.grad += out.grad
out._backward = _backward

```

Multiplication applies the chain rule using the sibling's data as the local Jacobian:

```python
def _backward():
    self.grad += other.data * out.grad   # dL/dself = dL/dout * other.data

    other.grad += self.data * out.grad   # dL/dother = dL/dout * self.data

out._backward = _backward

```

For non-linearities like `tanh`, the derivative `1 - tanh²(x)` is multiplied by the upstream gradient, encoding the local Jacobian directly into the closure without symbolic computation.

## Reverse-Mode Autodiff via Topological Sort

The `backward()` method implements reverse-mode automatic differentiation by traversing the graph in reverse topological order. This ensures that a node's gradient is fully accumulated before flowing to its parents. The algorithm follows three distinct phases:

1. **Build topological order**: Recursively visit all children before appending the current node to the `topo` list
2. **Seed the gradient**: Set the loss node's gradient to `1.0` (dL/dL = 1)
3. **Execute backward passes**: Iterate through the reversed `topo` list, invoking each node's `_backward` closure

```python
def backward(self):
    topo = []
    visited = set()
    def build_topo(v):
        if v not in visited:
            visited.add(v)
            for child in v._prev:
                build_topo(child)
            topo.append(v)
    build_topo(self)
    
    self.grad = 1.0
    for node in reversed(topo):
        node._backward()

```

This topological guarantee ensures that when a parent node executes its local gradient function, `out.grad` already contains the complete upstream gradient from all downstream paths.

## Practical Example: Backprop Through a Neuron

The following example from `lectures/micrograd/micrograd_lecture_second_half_roughly.ipynb` demonstrates backpropagation through a single neuron with weights, bias, and hyperbolic tangent activation:

```python

# Inputs and parameters

x1 = Value(2.0, label='x1')
x2 = Value(0.0, label='x2')
w1 = Value(-3.0, label='w1')
w2 = Value(1.0, label='w2')
b  = Value(6.8813735870195432, label='b')

# Forward pass: tanh(x1*w1 + x2*w2 + b)

n = x1 * w1 + x2 * w2 + b
o = n.tanh()
loss = (o - Value(1.0)) ** 2

# Backpropagation

loss.backward()
print(f"∂loss/∂w1 = {w1.grad:.4f}")  # Output: 0.5000

print(f"∂loss/∂b  = {b.grad:.4f}")   # Output: 0.5000

```

When `loss.backward()` executes, micrograd automatically computes gradients for every parameter by traversing the graph from the loss node back to the leaves, applying the chain rule through multiplication and addition gates.

## Summary

- **Dynamic Graph Construction**: Each operation creates `Value` nodes with `_prev` references and local `_backward` functions that encode partial derivatives using Python closures
- **Topological Ordering**: The `build_topo` helper ensures gradients flow from outputs to inputs in correct dependency order, executing children before parents
- **Gradient Accumulation**: Gradients are accumulated using `+=` rather than assignment to support branching computation graphs where one value contributes to multiple downstream nodes
- **Pure Python Implementation**: The entire backpropagation mechanism resides in `lectures/micrograd/micrograd_lecture_first_half_roughly.ipynb` without external autograd dependencies, making the algorithm transparent and educational

## Frequently Asked Questions

### What makes micrograd different from PyTorch's autograd?

Micrograd implements scalar-valued reverse-mode autodiff using only Python data structures and closures, whereas PyTorch operates on tensors with optimized C++ backends. Both systems build computation graphs dynamically and use topological sorting for gradient flow, but micrograd's implementation serves purely educational purposes with explicit `_backward` lambdas that demonstrate the chain rule explicitly.

### Why does micrograd use topological sorting for backpropagation?

Topological sorting guarantees that a node's gradient is fully computed before propagating to its parents. In the `backward()` method, the `build_topo` function creates a list where children appear before parents; reversing this list ensures that `node._backward()` executes only after `out.grad` contains the complete upstream gradient from all downstream paths, satisfying the dependencies required by the chain rule.

### How does micrograd handle the chain rule for non-linear operations?

Each non-linear operation stores its specific derivative calculation inside the `_backward` closure. For `tanh`, the implementation multiplies the incoming gradient by `1 - tanh²(x)` (where `x` is the pre-activation value stored in the parent's `data` attribute), automatically applying the chain rule during the backward pass without requiring symbolic differentiation.

### Can micrograd handle multiple outputs or branching graphs?

Yes. Because gradients accumulate using `+=` rather than assignment, micrograd correctly handles branching computation graphs where one `Value` contributes to multiple downstream nodes. The `_backward` functions increment `self.grad` for each edge, ensuring gradients sum across all paths as required by the multivariate chain rule when a node has multiple children in the computation graph.