How Backpropagation Is Implemented in Micrograd: A Deep Dive into the Value Class

Micrograd implements reverse-mode automatic differentiation by wrapping scalar values in Value objects that record computation history and execute local gradient functions in topological order.

Micrograd, the educational autograd engine built during Andrej Karpathy's "Neural Networks: Zero to Hero" course, demonstrates how backpropagation is implemented in micrograd from first principles using pure Python. This minimalist scalar-only framework constructs a dynamic computation graph through the Value class to enable gradient computation without external dependencies. The complete implementation resides in lectures/micrograd/micrograd_lecture_first_half_roughly.ipynb.

The Value Object: Building Blocks of the Computation Graph

Each scalar in micrograd is wrapped in a Value instance that tracks five critical attributes:

• data: The forward-pass numeric value
• grad: The accumulated gradient ∂L/∂self
• _prev: A set of parent Value objects used to compute this value
• _op: A string describing the operation (e.g., +, *, tanh)
• _backward: A closure that propagates the output gradient to its parents

This structure transforms static numbers into nodes of a dynamic directed acyclic graph (DAG). When operations execute, they create new Value instances linking back to their operands via _prev, establishing the computation topology required for reverse-mode differentiation.

Forward Pass: Recording Local Gradients

During the forward pass, each operation instantiates a new Value and records a local backward function encoding the partial derivatives. In lectures/micrograd/micrograd_lecture_first_half_roughly.ipynb, addition is implemented as:

out = Value(self.data + other.data, (self, other), '+')

def _backward():
    self.grad += out.grad      # dL/dself = dL/dout * 1

    other.grad += out.grad
out._backward = _backward

Multiplication applies the chain rule using the sibling's data as the local Jacobian:

def _backward():
    self.grad += other.data * out.grad   # dL/dself = dL/dout * other.data

    other.grad += self.data * out.grad   # dL/dother = dL/dout * self.data

out._backward = _backward

For non-linearities like tanh, the derivative 1 - tanh²(x) is multiplied by the upstream gradient, encoding the local Jacobian directly into the closure without symbolic computation.

Reverse-Mode Autodiff via Topological Sort

The backward() method implements reverse-mode automatic differentiation by traversing the graph in reverse topological order. This ensures that a node's gradient is fully accumulated before flowing to its parents. The algorithm follows three distinct phases:

1. Build topological order: Recursively visit all children before appending the current node to the topo list
2. Seed the gradient: Set the loss node's gradient to 1.0 (dL/dL = 1)
3. Execute backward passes: Iterate through the reversed topo list, invoking each node's _backward closure

def backward(self):
    topo = []
    visited = set()
    def build_topo(v):
        if v not in visited:
            visited.add(v)
            for child in v._prev:
                build_topo(child)
            topo.append(v)
    build_topo(self)
    
    self.grad = 1.0
    for node in reversed(topo):
        node._backward()

This topological guarantee ensures that when a parent node executes its local gradient function, out.grad already contains the complete upstream gradient from all downstream paths.

Practical Example: Backprop Through a Neuron

The following example from lectures/micrograd/micrograd_lecture_second_half_roughly.ipynb demonstrates backpropagation through a single neuron with weights, bias, and hyperbolic tangent activation:

# Inputs and parameters

x1 = Value(2.0, label='x1')
x2 = Value(0.0, label='x2')
w1 = Value(-3.0, label='w1')
w2 = Value(1.0, label='w2')
b  = Value(6.8813735870195432, label='b')

# Forward pass: tanh(x1*w1 + x2*w2 + b)

n = x1 * w1 + x2 * w2 + b
o = n.tanh()
loss = (o - Value(1.0)) ** 2

# Backpropagation

loss.backward()
print(f"∂loss/∂w1 = {w1.grad:.4f}")  # Output: 0.5000

print(f"∂loss/∂b  = {b.grad:.4f}")   # Output: 0.5000

When loss.backward() executes, micrograd automatically computes gradients for every parameter by traversing the graph from the loss node back to the leaves, applying the chain rule through multiplication and addition gates.

Summary

• Dynamic Graph Construction: Each operation creates Value nodes with _prev references and local _backward functions that encode partial derivatives using Python closures
• Topological Ordering: The build_topo helper ensures gradients flow from outputs to inputs in correct dependency order, executing children before parents
• Gradient Accumulation: Gradients are accumulated using += rather than assignment to support branching computation graphs where one value contributes to multiple downstream nodes
• Pure Python Implementation: The entire backpropagation mechanism resides in lectures/micrograd/micrograd_lecture_first_half_roughly.ipynb without external autograd dependencies, making the algorithm transparent and educational

Frequently Asked Questions

What makes micrograd different from PyTorch's autograd?

Micrograd implements scalar-valued reverse-mode autodiff using only Python data structures and closures, whereas PyTorch operates on tensors with optimized C++ backends. Both systems build computation graphs dynamically and use topological sorting for gradient flow, but micrograd's implementation serves purely educational purposes with explicit _backward lambdas that demonstrate the chain rule explicitly.

Why does micrograd use topological sorting for backpropagation?

Topological sorting guarantees that a node's gradient is fully computed before propagating to its parents. In the backward() method, the build_topo function creates a list where children appear before parents; reversing this list ensures that node._backward() executes only after out.grad contains the complete upstream gradient from all downstream paths, satisfying the dependencies required by the chain rule.

How does micrograd handle the chain rule for non-linear operations?

Each non-linear operation stores its specific derivative calculation inside the _backward closure. For tanh, the implementation multiplies the incoming gradient by 1 - tanh²(x) (where x is the pre-activation value stored in the parent's data attribute), automatically applying the chain rule during the backward pass without requiring symbolic differentiation.

Can micrograd handle multiple outputs or branching graphs?

Yes. Because gradients accumulate using += rather than assignment, micrograd correctly handles branching computation graphs where one Value contributes to multiple downstream nodes. The _backward functions increment self.grad for each edge, ensuring gradients sum across all paths as required by the multivariate chain rule when a node has multiple children in the computation graph.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}