# How the AI Engineering from Scratch Curriculum Transitions from Raw Math to Production-Ready Frameworks

> Master AI engineering by smoothly transitioning from raw math to production-ready frameworks with our Build/Use curriculum. Understand linear algebra before PyTorch optimizations.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: deep-dive
- Published: 2026-07-30

---

**The repository implements a deliberate "Build/Use" pedagogical split that progresses from pure Python mathematical implementations to PyTorch distributed training, ensuring you understand the underlying linear algebra before using framework optimizations.**

The `rohitg00/ai-engineering-from-scratch` repository structures its curriculum as a progressive ladder that transitions learners from handwritten mathematical derivations to enterprise-grade AI systems. This approach mandates that you implement every algorithmic component—attention mechanisms, gradient checkpointing, and distributed training—using only the Python standard library and NumPy before ever invoking a PyTorch abstraction.

## Phase 1: Foundations with Pure Python and NumPy

The curriculum begins with **raw mathematical transparency**, explicitly avoiding framework magic to expose the underlying formulas.

In **Phase 12 – Multimodal AI**, the "Flamingo-Gated-Cross-Attention" lesson demonstrates this philosophy clearly. According to the lesson documentation at [`phases/12-multimodal-ai/04-flamingo-gated-cross-attention/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/12-multimodal-ai/04-flamingo-gated-cross-attention/docs/en.md), the implementation constraints specify: *"Pure Python. No numpy, no torch – the point is to see the loss math and the argmax pattern."*

These early lessons rely on:
- Explicit `for` loops and list comprehensions for matrix operations
- NumPy arrays for basic linear algebra without broadcasting abstractions
- Manual implementation of attention scores using dot products and softmax

This foundation ensures you see every floating-point operation that later gets optimized into a single CUDA kernel call.

## Phase 2: Intermediate Integration with Hybrid NumPy and PyTorch

As the curriculum advances, lessons intentionally blend NumPy prototypes with PyTorch tensors to demonstrate interoperability between educational and production code.

The **Phase 10 – Gradient Checkpointing** lesson at [`phases/10-llms-from-scratch/34-gradient-checkpointing/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/10-llms-from-scratch/34-gradient-checkpointing/code/main.py) implements a hybrid approach. It generates toy data using NumPy arrays, then transitions to `torch.Tensor` objects when scaling to GPU-accelerated training. This structure proves that the mathematical logic remains identical—you simply swap `np.dot()` for `torch.matmul()` while gaining automatic differentiation and device placement.

This phase teaches **framework transplantation**: recognizing that a NumPy implementation of `softmax` or `layer_norm` can be replaced with `torch.nn.functional` equivalents without changing the underlying mathematics.

## Phase 3: Production-Grade PyTorch and Distributed Training

The capstone phases encapsulate the same mathematical operations inside production-ready abstractions, adding distributed training and fault tolerance.

**Phase 19 – End-to-End Distributed Train** at [`phases/19-capstone-projects/81-end-to-end-distributed-train/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/19-capstone-projects/81-end-to-end-distributed-train/code/main.py) demonstrates the final evolution. The lesson starts with a simple `torch.nn.Module` definition of a transformer block—mathematically identical to your Phase 12 hand-coded attention—but wraps it with:
- `torch.distributed` for multi-node parallelism
- Gradient sharding via `torch.nn.parallel.DistributedDataParallel`
- Fault-tolerant checkpointing using `torch.save()` and `torch.load()` schedules

By this point, you have already implemented the attention mechanism manually; you now see how `nn.MultiheadAttention` optimizes that same math across CUDA streams and network interfaces.

## The "Build/Use" Curriculum Philosophy

The [`AGENTS.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/AGENTS.md) manual codifies the repository's core pedagogical rule: **you must build an algorithm from raw math before you use a framework implementation**.

This philosophy manifests in the repository's metadata structure. Each lesson's [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md) front-matter explicitly lists the languages used (e.g., "Python (torch, numpy)"), while the accompanying [`code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/code/main.py) files contain header comments linking back to the mathematical documentation. This systematic traceability ensures you can always locate the pure-math origin of any production abstraction.

## Code Comparison: Raw Math vs. Production Framework

Below is the same linear transformation implemented first with raw NumPy (exposing the matrix multiplication) and then with PyTorch (leveraging optimized kernels and autograd).

Pure NumPy implementation:

```python
import numpy as np

def linear(x, w, b):
    """Compute y = x·wᵀ + b using explicit NumPy operations."""
    return np.dot(x, w.T) + b

# Example usage: batch_size=4, input_dim=10, output_dim=6

x = np.random.randn(4, 10)
w = np.random.randn(6, 10)
b = np.random.randn(6)
y = linear(x, w, b)  # Shape: (4, 6)

```

PyTorch production implementation:

```python
import torch
import torch.nn as nn

class Linear(nn.Module):
    """Optimized wrapper with automatic differentiation."""
    def __init__(self, in_dim: int, out_dim: int):
        super().__init__()
        self.linear = nn.Linear(in_dim, out_dim)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        return self.linear(x)

# Example usage with identical mathematical semantics

x = torch.randn(4, 10)
layer = Linear(10, 6)
y = layer(x)  # Shape: (4, 6), gradients tracked automatically

```

The NumPy version mirrors the mathematical definition `y = xW^T + b`; the PyTorch version utilizes highly-optimized CuBLAS kernels and integrates seamlessly with distributed training utilities—exactly the progression mapped across the curriculum phases.

## Summary

- **Start with transparency**: Early phases (12, 10) mandate pure Python/NumPy implementations to expose attention mechanisms and checkpointing logic without framework abstraction.
- **Bridge with hybrid code**: Intermediate lessons blend NumPy and PyTorch to demonstrate that mathematical logic transcends implementation details.
- **End with production systems**: Capstone projects (Phase 19) wrap the same mathematics in `torch.distributed` pipelines, sharding, and fault-tolerant checkpointing.
- **Follow the "Build/Use" split**: The [`AGENTS.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/AGENTS.md) curriculum philosophy ensures you implement algorithms manually before calling optimized framework methods.

## Frequently Asked Questions

### Why does the curriculum start with pure Python instead of PyTorch immediately?

Starting with pure Python and NumPy forces you to implement the mathematical operations—matrix multiplications, softmax normalization, and gradient calculations—using explicit loops and basic array operations. This ensures you understand *why* `torch.nn.Linear` produces specific outputs, not just *how* to call it, according to the pedagogical guidelines in [`AGENTS.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/AGENTS.md).

### How does Phase 10 demonstrate the transition between NumPy and PyTorch?

Phase 10's "Gradient Checkpointing" lesson at [`phases/10-llms-from-scratch/34-gradient-checkpointing/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/10-llms-from-scratch/34-gradient-checkpointing/code/main.py) generates synthetic training data using NumPy arrays, then transitions the computation graph to PyTorch tensors midway through the script. This shows that you can prototype with NumPy and migrate to PyTorch with minimal code changes while preserving mathematical correctness.

### What production features are introduced in the final capstone phases?

**Phase 19 – End-to-End Distributed Train** introduces `torch.distributed` for multi-GPU coordination, gradient sharding through `DistributedDataParallel`, and fault-tolerant checkpointing logic. These features wrap the same attention and linear layer mathematics you implemented manually in earlier phases, now optimized for cluster-scale training.

### How does the repository track which implementation style each lesson uses?

Every lesson directory contains a [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md) file with YAML front-matter listing the specific languages and libraries used (e.g., "Python (torch, numpy)"). The corresponding [`code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/code/main.py) files include header comments linking back to the documentation, creating a traceable path from raw mathematical implementations to production framework code.