When Does the AI Engineering Curriculum Introduce Deep Learning and Backpropagation from Scratch?

The rohitg00/ai-engineering-from-scratch curriculum introduces deep learning and backpropagation from scratch in Phase 3 – Deep Learning Core, specifically Lesson 03, where learners build a complete neural-network autograd engine without using PyTorch, TensorFlow, or any external ML libraries.

The journey from mathematical foundations to training neural networks requires a structured understanding of how gradients flow through computational graphs. In the rohitg00/ai-engineering-from-scratch repository, the transition from traditional machine learning to deep learning occurs in Phase 3, where students implement backpropagation by hand. This phase marks the critical moment when theoretical calculus concepts transform into executable code that can train multi-layer networks on non-linear problems.

Phase 3: The Deep Learning Transition

The curriculum organizes content into distinct phases, with deep learning fundamentals emerging only after establishing mathematical and classical ML prerequisites.

Prerequisites: Completing Phase 1 and Phase 2

Before encountering deep learning and backpropagation from scratch, learners must finish Phase 1 (Math Foundations) and Phase 2 (ML Fundamentals). These earlier phases cover linear algebra, calculus, probability, and supervised learning algorithms, providing the necessary theoretical substrate for understanding gradient descent and automatic differentiation.

Lesson Location: 03-backpropagation

The specific lesson resides at phases/03-deep-learning-core/03-backpropagation/. This directory contains both the theoretical documentation in docs/en.md and the reference implementation in code/main.py, which serves as the canonical example of building a deep learning framework from zero dependencies.

Building the Autograd Engine from Scratch

The core objective of Lesson 03 is constructing a computational graph that tracks operations and automatically computes gradients via reverse-mode automatic differentiation.

The Value Class: Nodes in the Computational Graph

According to the source code in phases/03-deep-learning-core/03-backpropagation/code/main.py, the engine centers around a Value class that encapsulates scalar data, gradients, and backward propagation functions. Each instance stores its children nodes and the operation that produced it, forming a directed acyclic graph.

class Value:
    def __init__(self, data, children=(), op=''):
        self.data = data
        self.grad = 0.0
        self._backward = lambda: None
        self._children = set(children)
        self._op = op

    def __repr__(self):
        return f"Value(data={self.data:.4f}, grad={self.grad:.4f})"

Implementing Gradient Rules for Operations

The implementation defines how gradients flow through basic arithmetic. The __add__ and __mul__ methods create new Value nodes while capturing closure-based backward functions that apply the chain rule.

def __add__(self, other):
    other = other if isinstance(other, Value) else Value(other)
    out = Value(self.data + other.data, (self, other), '+')
    def _backward():
        self.grad += out.grad
        other.grad += out.grad
    out._backward = _backward
    return out

def __mul__(self, other):
    other = other if isinstance(other, Value) else Value(other)
    out = Value(self.data * other.data, (self, other), '*')
    def _backward():
        self.grad += other.data * out.grad
        other.grad += self.data * out.grad
    out._backward = _backward
    return out

Non-Linear Activations: Sigmoid Implementation

To train neural networks on non-linear decision boundaries like circles or XOR patterns, the curriculum implements the sigmoid activation with safe numerical clipping and its derivative built into the backward pass.

def sigmoid(self):
    x = max(-500, min(500, self.data))
    s = 1.0 / (1.0 + math.exp(-x))
    out = Value(s, (self,), 'sigmoid')
    def _backward():
        self.grad += (s * (1 - s)) * out.grad
    out._backward = _backward
    return out

Executing the Backward Pass

The curriculum teaches reverse-mode automatic differentiation through a topological sort that ensures gradients propagate from output to inputs in the correct dependency order.

Topological Sort and Gradient Accumulation

The backward() method performs a post-order traversal to build a topological ordering of the computational graph, initializes the output gradient to 1.0, and then calls each node's stored _backward function in reverse order.

def backward(self):
    topo, visited = [], set()
    def build(v):
        if v not in visited:
            visited.add(v)
            for child in v._children:
                build(child)
            topo.append(v)
    build(self)
    self.grad = 1.0
    for node in reversed(topo):
        node._backward()

Defining Loss Functions

The lesson implements Mean Squared Error (MSE) loss using the same Value primitives, ensuring the loss computation itself becomes part of the differentiable graph.

def mse_loss(predicted, target):
    diff = predicted + Value(-target)
    return diff * diff

Training Networks on Real Problems

The pedagogical goal culminates in training a multi-layer perceptron without importing torch or tensorflow.

XOR Classification: The Non-Linear Test Case

The reference implementation in main.py includes a complete training loop for the XOR problem, using a network architecture with 2 inputs, 4 hidden units, and 1 output. This demonstrates that the from-scratch engine can successfully optimize parameters through backpropagation.

net = Network([2, 4, 1])          # 2-in, hidden 4, 1-out

xor_data = [([0,0],0),([0,1],1),([1,0],1),([1,1],0)]
learning_rate = 1.0
for epoch in range(1000):
    total_loss = Value(0.0)
    for inputs, target in xor_data:
        x = [Value(i) for i in inputs]
        pred = net(x)
        loss = mse_loss(pred, target)
        total_loss = total_loss + loss
    net.zero_grad()
    total_loss.backward()
    for p in net.parameters():
        p.data -= learning_rate * p.grad
    if epoch % 100 == 0:
        print(f"Epoch {epoch:4d} | Loss: {total_loss.data:.6f}")

Key Files in the Curriculum

The implementation spans specific files within the Phase 3 directory structure:

• phases/03-deep-learning-core/03-backpropagation/docs/en.md — Contains the full lesson text with theoretical explanations of the chain rule, computational graphs, and step-by-step derivations of backward passes.
• phases/03-deep-learning-core/03-backpropagation/code/main.py — Provides the executable reference implementation including the Value class, activation functions, loss definitions, network architecture, and training loops.
• phases/03-deep-learning-core/README.md — Offers the Phase 3 overview and navigation links to all deep learning lessons, contextualizing where backpropagation fits within the broader deep learning core.

Summary

• The curriculum introduces deep learning and backpropagation from scratch in Phase 3 – Deep Learning Core, Lesson 03.
• Prerequisites include completion of Phase 1 (Math Foundations) and Phase 2 (ML Fundamentals).
• The Value class in phases/03-deep-learning-core/03-backpropagation/code/main.py implements a complete autograd engine using reverse-mode automatic differentiation.
• The lesson builds a computational graph dynamically, stores local gradient functions in _backward closures, and executes them via topological sort.
• Learners train a multi-layer network on the XOR problem without importing external ML frameworks, verifying the engine works correctly.

Frequently Asked Questions

What prerequisites are needed before studying backpropagation from scratch?

Learners must complete Phase 1 (Math Foundations) covering calculus and linear algebra, and Phase 2 (ML Fundamentals) covering basic supervised learning. These provide the mathematical literacy required to understand partial derivatives and the chain rule implementations in the Value class.

Which files contain the actual backpropagation implementation?

The core implementation lives in phases/03-deep-learning-core/03-backpropagation/code/main.py, which contains the Value class, gradient operations, and training loops. The theoretical documentation explaining the mathematics resides in phases/03-deep-learning-core/03-backpropagation/docs/en.md.

What neural network architectures are built using this from-scratch engine?

The curriculum implements a fully-connected multi-layer perceptron (MLP) with configurable layer dimensions. The specific example trains a network with architecture [2, 4, 1] (2 inputs, 4 hidden units, 1 output) on the XOR classification problem, though the engine supports arbitrary depth and width.

Does this lesson use PyTorch, TensorFlow, or other frameworks?

No. The entire objective of Lesson 03 is to build the autograd engine from zero dependencies. The main.py file implements automatic differentiation, backpropagation, and neural network training using only Python standard library modules like math, demonstrating how modern frameworks function under the hood.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}