Understanding the Computational Graph Structure in micrograd Value Objects

The micrograd Value class implements a directed acyclic graph (DAG) where each node stores parent references in _prev, records the operation in _op, and encapsulates local gradient logic in a _backward closure to enable reverse-mode automatic differentiation.

The educational autograd engine in the karpathy/nn-zero-to-hero repository demonstrates how modern deep learning frameworks track computations. The Value object serves as the fundamental unit of a computational graph structure in micrograd Value objects, recording scalar values alongside their lineage to calculate gradients via backpropagation.

Core Attributes of the Value Node

As implemented in lectures/micrograd/micrograd_lecture_first_half_roughly.ipynb (lines 71–81), each Value instance maintains six critical attributes that define its role in the graph:

• data — The scalar numeric value (float).
• grad — Accumulated gradient initialized to 0.0, representing the derivative of the final output with respect to this node.
• _prev — A Python set containing parent Value objects used as operands to compute the current node.
• _op — A string identifier (e.g., '+', '*', 'tanh') describing the operation that created this node.
• label — An optional string for human-readable identification during visualization.
• _backward — A closure attached during the forward pass that knows how to propagate gradients backward to parent nodes.

Graph Construction During Forward Pass

Operations like addition and multiplication construct the graph dynamically. When you execute c = a + b, the __add__ method creates a new Value containing the result data while establishing edges back to its parents.

In lectures/micrograd/micrograd_lecture_first_half_roughly.ipynb, the binary operation implementation follows this pattern:

out = Value(self.data + other.data, (self, other), '+')

The second argument (self, other) populates _prev, and the third argument '+' populates _op. Immediately after construction, micrograd attaches a _backward function that implements the local gradient for that specific operation:

def _backward():
    self.grad += out.grad          # dL/da

    other.grad += out.grad         # dL/db

out._backward = _backward

Similarly, multiplication defines its own _backward closure that scales the upstream gradient by the opposing operand’s value, implementing the product rule.

Topological Ordering and the Backward Pass

Calling some_value.backward() triggers reverse-mode differentiation through a three-phase process defined in the notebook (line 16):

1. Topological sort — A depth-first traversal builds a list topo containing all reachable nodes, ensuring that children appear before their parents in the ordering.
2. Seed gradient — The target node’s gradient is set to 1.0 (representing ∂L/∂L).
3. Reverse traversal — Nodes are visited in reverse topological order, invoking each _backward closure to accumulate gradients into _prev parents via the chain rule.

This traversal guarantees that when a node’s _backward runs, its own grad already contains the complete upstream derivative, allowing it to correctly distribute gradients to its parents.

Visualizing the Computational Graph

The notebook provides trace and draw_dot utilities that walk the _prev links to collect nodes and edges. These functions generate Graphviz diagrams showing each node’s label, data value, and current gradient, making the abstract DAG concrete for debugging and education.

To render a graph visualization:

from graphviz import Digraph

dot = draw_dot(L)          # L is the final Value node

dot.render('micrograd_graph', view=True)

Complete Example: Building and Traversing the Graph

The following example constructs the expression L = (a × b + c) × f and computes gradients:

# Build the computational graph

a = Value(2.0, label='a')
b = Value(-3.0, label='b')
c = Value(10.0, label='c')
f = Value(-2.0, label='f')

e = a * b; e.label = 'e'      # Intermediate node

d = e + c; d.label = 'd'      # Addition node

L = d * f; L.label = 'L'      # Final output

# Forward pass

print(f"L.data = {L.data}")   # Output: 40.0

# Backward pass

L.backward()

# Inspected gradients

print(f"∂L/∂a = {a.grad}")    # -20.0

print(f"∂L/∂b = {b.grad}")    # 40.0

print(f"∂L/∂c = {c.grad}")    # -2.0

print(f"∂L/∂f = {f.grad}")    # 8.0

This demonstrates how the computational graph structure maintains the necessary information to analytically compute ∂L/∂a and other partial derivatives through recursive application of the chain rule.

Summary

• The computational graph structure in micrograd Value objects forms a directed acyclic graph using the _prev attribute to store parent references.
• Each node encapsulates its local gradient calculation in a _backward closure, enabling modular backpropagation.
• The backward() method performs a reverse topological sort to ensure gradients propagate from children to parents in the correct order.
• Visualization utilities in lectures/micrograd/micrograd_lecture_first_half_roughly.ipynb leverage the _prev links to render the complete graph structure.

Frequently Asked Questions

What does the _prev attribute store in micrograd Value objects?

The _prev attribute stores a Python set containing the parent Value nodes that served as operands in the operation that created the current node. For example, when computing c = a + b, the new Value c holds its _prev set containing both a and b, establishing the directed edge in the computational graph.

How does micrograd perform the backward pass?

The backward() method first performs a topological sort to establish a linear ordering where children precede parents. It then sets the output gradient to 1.0 and iterates through the sorted nodes in reverse, calling each stored _backward closure to accumulate gradients into parent nodes according to the chain rule.

Is the computational graph in micrograd a DAG?

Yes, the computational graph structure is strictly a directed acyclic graph (DAG). The _prev references always point from result nodes to their operand nodes, and since values flow forward while gradients flow backward, no cycles can exist in the computation history.

How can I visualize the micrograd computational graph?

Use the draw_dot function defined in lectures/micrograd/micrograd_lecture_first_half_roughly.ipynb. Pass the final output Value node to this function, which traverses _prev references to collect all nodes and edges, then returns a Graphviz Digraph object that can be rendered to SVG or PNG.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}