# When Does the AI Engineering Curriculum Introduce Deep Learning and Backpropagation from Scratch?

> Discover when Phase 3 of the AI Engineering From Scratch curriculum introduces deep learning and backpropagation. Build a neural-network autograd engine from scratch without libraries.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: getting-started
- Published: 2026-09-02

---

**The rohitg00/ai-engineering-from-scratch curriculum introduces deep learning and backpropagation from scratch in Phase 3 – Deep Learning Core, specifically Lesson 03, where learners build a complete neural-network autograd engine without using PyTorch, TensorFlow, or any external ML libraries.**

The journey from mathematical foundations to training neural networks requires a structured understanding of how gradients flow through computational graphs. In the `rohitg00/ai-engineering-from-scratch` repository, the transition from traditional machine learning to deep learning occurs in Phase 3, where students implement backpropagation by hand. This phase marks the critical moment when theoretical calculus concepts transform into executable code that can train multi-layer networks on non-linear problems.

## Phase 3: The Deep Learning Transition

The curriculum organizes content into distinct phases, with deep learning fundamentals emerging only after establishing mathematical and classical ML prerequisites.

### Prerequisites: Completing Phase 1 and Phase 2

Before encountering deep learning and backpropagation from scratch, learners must finish **Phase 1 (Math Foundations)** and **Phase 2 (ML Fundamentals)**. These earlier phases cover linear algebra, calculus, probability, and supervised learning algorithms, providing the necessary theoretical substrate for understanding gradient descent and automatic differentiation.

### Lesson Location: 03-backpropagation

The specific lesson resides at `phases/03-deep-learning-core/03-backpropagation/`. This directory contains both the theoretical documentation in [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md) and the reference implementation in [`code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/code/main.py), which serves as the canonical example of building a deep learning framework from zero dependencies.

## Building the Autograd Engine from Scratch

The core objective of Lesson 03 is constructing a **computational graph** that tracks operations and automatically computes gradients via reverse-mode automatic differentiation.

### The Value Class: Nodes in the Computational Graph

According to the source code in [`phases/03-deep-learning-core/03-backpropagation/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/03-backpropagation/code/main.py), the engine centers around a `Value` class that encapsulates scalar data, gradients, and backward propagation functions. Each instance stores its children nodes and the operation that produced it, forming a directed acyclic graph.

```python
class Value:
    def __init__(self, data, children=(), op=''):
        self.data = data
        self.grad = 0.0
        self._backward = lambda: None
        self._children = set(children)
        self._op = op

    def __repr__(self):
        return f"Value(data={self.data:.4f}, grad={self.grad:.4f})"

```

### Implementing Gradient Rules for Operations

The implementation defines how gradients flow through basic arithmetic. The `__add__` and `__mul__` methods create new `Value` nodes while capturing closure-based backward functions that apply the chain rule.

```python
def __add__(self, other):
    other = other if isinstance(other, Value) else Value(other)
    out = Value(self.data + other.data, (self, other), '+')
    def _backward():
        self.grad += out.grad
        other.grad += out.grad
    out._backward = _backward
    return out

def __mul__(self, other):
    other = other if isinstance(other, Value) else Value(other)
    out = Value(self.data * other.data, (self, other), '*')
    def _backward():
        self.grad += other.data * out.grad
        other.grad += self.data * out.grad
    out._backward = _backward
    return out

```

### Non-Linear Activations: Sigmoid Implementation

To train neural networks on non-linear decision boundaries like circles or XOR patterns, the curriculum implements the sigmoid activation with safe numerical clipping and its derivative built into the backward pass.

```python
def sigmoid(self):
    x = max(-500, min(500, self.data))
    s = 1.0 / (1.0 + math.exp(-x))
    out = Value(s, (self,), 'sigmoid')
    def _backward():
        self.grad += (s * (1 - s)) * out.grad
    out._backward = _backward
    return out

```

## Executing the Backward Pass

The curriculum teaches reverse-mode automatic differentiation through a topological sort that ensures gradients propagate from output to inputs in the correct dependency order.

### Topological Sort and Gradient Accumulation

The `backward()` method performs a post-order traversal to build a topological ordering of the computational graph, initializes the output gradient to 1.0, and then calls each node's stored `_backward` function in reverse order.

```python
def backward(self):
    topo, visited = [], set()
    def build(v):
        if v not in visited:
            visited.add(v)
            for child in v._children:
                build(child)
            topo.append(v)
    build(self)
    self.grad = 1.0
    for node in reversed(topo):
        node._backward()

```

### Defining Loss Functions

The lesson implements Mean Squared Error (MSE) loss using the same `Value` primitives, ensuring the loss computation itself becomes part of the differentiable graph.

```python
def mse_loss(predicted, target):
    diff = predicted + Value(-target)
    return diff * diff

```

## Training Networks on Real Problems

The pedagogical goal culminates in training a multi-layer perceptron without importing `torch` or `tensorflow`.

### XOR Classification: The Non-Linear Test Case

The reference implementation in [`main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/main.py) includes a complete training loop for the XOR problem, using a network architecture with 2 inputs, 4 hidden units, and 1 output. This demonstrates that the from-scratch engine can successfully optimize parameters through backpropagation.

```python
net = Network([2, 4, 1])          # 2-in, hidden 4, 1-out

xor_data = [([0,0],0),([0,1],1),([1,0],1),([1,1],0)]
learning_rate = 1.0
for epoch in range(1000):
    total_loss = Value(0.0)
    for inputs, target in xor_data:
        x = [Value(i) for i in inputs]
        pred = net(x)
        loss = mse_loss(pred, target)
        total_loss = total_loss + loss
    net.zero_grad()
    total_loss.backward()
    for p in net.parameters():
        p.data -= learning_rate * p.grad
    if epoch % 100 == 0:
        print(f"Epoch {epoch:4d} | Loss: {total_loss.data:.6f}")

```

## Key Files in the Curriculum

The implementation spans specific files within the Phase 3 directory structure:

- **[`phases/03-deep-learning-core/03-backpropagation/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/03-backpropagation/docs/en.md)** — Contains the full lesson text with theoretical explanations of the chain rule, computational graphs, and step-by-step derivations of backward passes.
- **[`phases/03-deep-learning-core/03-backpropagation/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/03-backpropagation/code/main.py)** — Provides the executable reference implementation including the `Value` class, activation functions, loss definitions, network architecture, and training loops.
- **[`phases/03-deep-learning-core/README.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/README.md)** — Offers the Phase 3 overview and navigation links to all deep learning lessons, contextualizing where backpropagation fits within the broader deep learning core.

## Summary

- The curriculum introduces **deep learning and backpropagation from scratch** in **Phase 3 – Deep Learning Core, Lesson 03**.
- Prerequisites include completion of Phase 1 (Math Foundations) and Phase 2 (ML Fundamentals).
- The `Value` class in [`phases/03-deep-learning-core/03-backpropagation/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/03-backpropagation/code/main.py) implements a complete autograd engine using reverse-mode automatic differentiation.
- The lesson builds a computational graph dynamically, stores local gradient functions in `_backward` closures, and executes them via topological sort.
- Learners train a multi-layer network on the **XOR problem** without importing external ML frameworks, verifying the engine works correctly.

## Frequently Asked Questions

### What prerequisites are needed before studying backpropagation from scratch?

Learners must complete Phase 1 (Math Foundations) covering calculus and linear algebra, and Phase 2 (ML Fundamentals) covering basic supervised learning. These provide the mathematical literacy required to understand partial derivatives and the chain rule implementations in the `Value` class.

### Which files contain the actual backpropagation implementation?

The core implementation lives in [`phases/03-deep-learning-core/03-backpropagation/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/03-backpropagation/code/main.py), which contains the `Value` class, gradient operations, and training loops. The theoretical documentation explaining the mathematics resides in [`phases/03-deep-learning-core/03-backpropagation/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/03-deep-learning-core/03-backpropagation/docs/en.md).

### What neural network architectures are built using this from-scratch engine?

The curriculum implements a fully-connected multi-layer perceptron (MLP) with configurable layer dimensions. The specific example trains a network with architecture `[2, 4, 1]` (2 inputs, 4 hidden units, 1 output) on the XOR classification problem, though the engine supports arbitrary depth and width.

### Does this lesson use PyTorch, TensorFlow, or other frameworks?

No. The entire objective of Lesson 03 is to build the autograd engine from zero dependencies. The [`main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/main.py) file implements automatic differentiation, backpropagation, and neural network training using only Python standard library modules like `math`, demonstrating how modern frameworks function under the hood.