How AI Engineering from Scratch Teaches Neural Networks From First Principles

AI Engineering from Scratch employs a "build-it-use-it" pedagogical cycle that teaches neural networks by progressively assembling mathematical building blocks from the ground up, requiring learners to implement forward and backward passes by hand before touching modern frameworks.

The open-source curriculum at rohitg00/ai-engineering-from-scratch takes a deterministic approach to deep learning education. Rather than importing pre-built models, learners write every algorithm from scratch using only the Python standard library. This methodology ensures that every graduate understands the mathematical operations powering modern AI systems.

The Pedagogical Pipeline: From Perceptron to Production

The repository organizes neural network instruction into nine distinct phases, each documented in phases/03-deep-learning-core/. Each phase adds exactly one new concept, preventing the cognitive overload common in framework-first tutorials.

Phase 1: The Perceptron as the Atomic Unit

According to phases/03-deep-learning-core/01-the-perceptron/docs/en.md, the curriculum begins with the perceptron as the fundamental neural unit. Learners implement a weighted sum plus step function to understand linear decision boundaries. The lesson covers the perceptron learning rule and its convergence proof, establishing the geometric intuition that underlies all subsequent deep learning.

Phase 2: Multi-Layer Networks and Depth

The progression moves to phases/03-deep-learning-core/02-multi-layer-networks/docs/en.md, where learners stack perceptrons to create deep architectures. This phase exposes why single-layer networks cannot model non-linear functions like XOR. The curriculum demonstrates matrix multiplication as the dominant computational cost in any network, introducing the universal approximation theorem (Cybenko, 1989) to justify depth.

Phase 3: Backpropagation From the Chain Rule

In phases/03-deep-learning-core/03-backpropagation/docs/en.md, the curriculum derives backpropagation from the chain rule of calculus. Learners implement the "gradient-chain" intuition manually, seeing how gradients flow backward through each layer. The lesson emphasizes that a network without backpropagation cannot learn, and students write the full backward pass by hand before using automatic differentiation.

Phase 4: Non-Linear Activation Functions

The repository addresses the limitations of linearity in phases/03-deep-learning-core/04-activation-functions/docs/en.md. After establishing linear layers, learners add ReLU, sigmoid, and Swish activations. The lesson explains why non-linearities break the linearity of stacked layers and discusses the vanishing-gradient problem that motivated modern activation designs.

Phase 5: Weight Initialization Strategies

According to phases/03-deep-learning-core/08-weight-initialization/docs/en.md, the curriculum explores why naive random initialization causes exploding or vanishing gradients. Learners implement Xavier/Glorot and He initialization schemes, conducting hands-on experiments that visualize loss-curve differences between strategies. This phase connects the mathematics of variance preservation to training stability.

Phase 6: Regularization and Optimizer Design

The repository covers practical training techniques in phases/03-deep-learning-core/06-optimizers/docs/en.md and phases/03-deep-learning-core/07-regularization/docs/en.md. Learners add L2 regularization and dropout to prevent overfitting, then implement SGD, momentum, and Adam optimizers. Each optimizer is shown as a modification to the update rule derived in the backpropagation phase.

Phase 7: Building a Mini-Framework

To cement the theory, phases/03-deep-learning-core/10-mini-framework/docs/en.md guides learners through creating a tiny neural-network library using only Python's standard library. Students implement Modules, Sequential containers, Linear layers, and loss functions. This mini-framework is then used to reproduce previous experiments, proving the entire stack works end-to-end.

Phase 8: Translating to PyTorch and JAX

Once the bare-bones framework is verified, the curriculum maps these concepts onto production libraries. phases/03-deep-learning-core/11-intro-to-pytorch/docs/en.md shows the translation from hand-written code to PyTorch tensors, while phases/03-deep-learning-core/12-intro-to-jax/docs/en.md demonstrates functional-style neural network building with JAX/Flax.

Phase 9: Debugging and Scaling Laws

The final phase in phases/03-deep-learning-core/13-debugging-neural-networks/docs/en.md teaches systematic debugging through gradient checks and loss-curve diagnostics. Learners explore scaling laws that explain why modern models require billions of parameters, completing the bridge from first principles to state-of-the-art practice.

Hands-On Implementation Examples

The following self-contained snippets mirror exercises from the curriculum and run with only the Python standard library.

Perceptron Learning the AND Gate

This example from the perceptron phase implements the learning rule for a linearly separable Boolean function:

import random

def step(x):
    return 1 if x > 0 else 0

# Randomly initialise weights & bias

w = [random.uniform(-1, 1) for _ in range(2)]
b = random.uniform(-1, 1)
lr = 0.1

# Training data for AND

X = [(0,0), (0,1), (1,0), (1,1)]
y = [0, 0, 0, 1]

for epoch in range(10):
    for xi, yi in zip(X, y):
        # Forward pass

        z = sum(wi * xi_i for wi, xi_i in zip(w, xi)) + b
        y_hat = step(z)
        # Perceptron update rule

        error = yi - y_hat
        w = [wi + lr * error * xi_i for wi, xi_i in zip(w, xi)]
        b += lr * error
    
    # Calculate mean squared error

    loss = sum((yi - step(sum(wi * xi_i for wi, xi_i in zip(w, xi)) + b)) ** 2 
               for xi, yi in zip(X, y)) / 4
    print(f"epoch {epoch} loss {loss:.3f}")

print("Trained weights:", w, "bias:", b)

Two-Layer Network With Manual Backpropagation

This implementation from the backpropagation phase solves the non-linear XOR problem without any framework:

import random

def relu(x): 
    return max(0.0, x)

def drelu(x): 
    return 1.0 if x > 0 else 0.0

# Initialise small weights to prevent saturation

W1 = [[random.uniform(-0.1, 0.1) for _ in range(2)] for _ in range(3)]  # 2→3

b1 = [0.0] * 3
W2 = [random.uniform(-0.1, 0.1) for _ in range(3)]  # 3→1

b2 = 0.0
lr = 0.01

def forward(x):
    h = [relu(sum(wi * xj for wi, xj in zip(row, x)) + bi) 
         for row, bi in zip(W1, b1)]
    out = sum(wi * hi for wi, hi in zip(W2, h)) + b2
    return h, out

def backward(x, y, h, out):
    # MSE loss derivative

    dL_dout = 2 * (out - y)
    # Output layer gradients

    dL_dW2 = [dL_dout * hi for hi in h]
    dL_db2 = dL_dout
    # Hidden layer gradients via chain rule

    dL_dh = [dL_dout * w2i for w2i in W2]
    dL_dz1 = [dh_i * drelu(z) for dh_i, z in zip(dL_dh, h)]
    dL_dW1 = [[dL_dz1_j * x_i for x_i in x] for dL_dz1_j in dL_dz1]
    dL_db1 = dL_dz1
    return dL_dW1, dL_db1, dL_dW2, dL_db2

# Train on XOR

X = [(0,0), (0,1), (1,0), (1,1)]
y = [0, 1, 1, 0]

for epoch in range(5000):
    total_loss = 0.0
    for xi, yi in zip(X, y):
        h, out = forward(xi)
        loss = (out - yi) ** 2
        total_loss += loss
        dW1, db1, dW2, db2 = backward(xi, yi, h, out)
        # Gradient descent update

        W1 = [[w - lr * gw for w, gw in zip(row, drow)] 
              for row, drow in zip(W1, dW1)]
        b1 = [b - lr * gb for b, gb in zip(b1, db1)]
        W2 = [w - lr * gw for w, gw in zip(W2, dW2)]
        b2 -= lr * db2
    
    if epoch % 500 == 0:
        print(f"epoch {epoch} loss {total_loss/4:.4f}")

# Verify predictions

for xi in X:
    _, out = forward(xi)
    print(xi, "→", round(out))

Summary

• AI Engineering from Scratch teaches neural networks through a deterministic nine-phase pipeline located in phases/03-deep-learning-core/.
• Each phase introduces exactly one concept: perceptrons, depth, backpropagation, activations, initialization, optimizers, framework building, and production migration.
• Learners implement forward and backward passes manually using only Python's standard library before accessing PyTorch or JAX.
• The curriculum derives all algorithms from first principles, including the perceptron learning rule, chain rule backpropagation, and variance-preserving weight initialization.
• A capstone mini-framework project proves that students can construct the entire deep learning stack from scratch.

Frequently Asked Questions

What is the "build-it-use-it" cycle in AI Engineering from Scratch?

The "build-it-use-it" cycle refers to the curriculum's requirement that learners implement every algorithm manually before using a library implementation. According to the source documentation, you must build the forward and backward passes by hand in pure Python, then use your implementation to train models, before finally migrating to PyTorch or JAX. This ensures you understand the mathematical operations rather than just calling APIs.

Why does the curriculum start with the perceptron instead of modern architectures?

The perceptron, documented in phases/03-deep-learning-core/01-the-perceptron/docs/en.md, serves as the "atom" of neural networks. Starting with this single-unit model allows learners to understand the geometry of decision boundaries and the mechanics of weight updates without the complexity of backpropagation. This foundation makes the transition to multi-layer networks in phase two conceptually straightforward rather than mysterious.

How does the mini-framework phase prepare students for PyTorch?

The mini-framework phase in phases/03-deep-learning-core/10-mini-framework/docs/en.md requires implementing a complete neural network library with Modules, Sequential containers, and Linear layers using only standard Python. When students later transition to PyTorch in phases/03-deep-learning-core/11-intro-to-pytorch/docs/en.md, they recognize every class and method because they have already written equivalent functionality themselves. This mapping makes framework documentation transparent rather than opaque.

Which mathematical prerequisites are required for the backpropagation lessons?

The backpropagation lessons in phases/03-deep-learning-core/03-backpropagation/docs/en.md derive the algorithm from the multivariable chain rule. You need to understand partial derivatives and how gradients flow through composite functions. The curriculum does not require linear algebra beyond matrix multiplication, as it builds intuition for Jacobian matrices through the manual computation of layer-to-layer gradients rather than abstract tensor calculus.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}