How AI Engineering from Scratch Compares to PyTorch and scikit‑learn

AI Engineering from Scratch teaches the complete AI stack by building every component from first principles before introducing third‑party libraries, creating a direct mental model that maps hand‑crafted implementations to PyTorch and scikit‑learn abstractions.

The rohitg00/ai-engineering-from-scratch repository follows a strict “Build It / Use It” pattern: you first implement low‑level primitives in pure Python, then map each to its industrial‑strength counterpart. This approach bridges the gap between mathematical theory and production code, revealing exactly what high‑level frameworks hide behind their APIs.

The Build It / Use It Pedagogy

The curriculum is organized into Build It lessons that construct tensors, back‑propagation, optimizers, and data loaders from scratch, followed by Use It lessons that translate these primitives to PyTorch or scikit‑learn equivalents. This methodology appears in core files such as phases/03-deep-learning-core/10-mini-framework/docs/en.md and phases/03-deep-learning-core/11-intro-to-pytorch/docs/en.md.

The pedagogical goal is understanding how the math works and why each abstraction exists, rather than simply calling high‑level APIs.

From‑Scratch Implementation vs. Production Frameworks

Hand‑Crafted Mini‑Framework

In phases/03-deep-learning-core/10-mini-framework/docs/en.md, the curriculum implements a mini‑framework in approximately 500 lines of pure Python. This library defines:

• Module base class with abstract forward and backward methods
• Linear layers with manual weight initialization and gradient computation
• Activation functions (ReLU, Sigmoid) and regularization (Dropout, BatchNorm)
• Optimizers (SGD, Adam) with explicit state tracking and parameter updates
• DataLoader for batch iteration and shuffling

Every tensor operation and gradient calculation is explicit. For example, the Linear layer manually computes forward passes and backward derivatives without autograd assistance.

PyTorch Equivalents

The Introduction to PyTorch lesson in phases/03-deep-learning-core/11-intro-to-pytorch/docs/en.md maps the mini‑framework classes directly to PyTorch primitives:

• Module → torch.nn.Module (same interface for forward, parameters, train/eval modes)
• Linear → torch.nn.Linear (automatic parameter registration, no manual gradient bookkeeping)
• Dropout/BatchNorm → Native layers with automatic train/eval mode handling
• Adam → torch.optim.Adam (adaptive moments without manual state tracking)
• DataLoader → torch.utils.data.DataLoader (multi‑process loading and GPU acceleration)

The implementation shifts from manual backward methods to tape‑based automatic differentiation, eliminating explicit gradient bookkeeping.

scikit‑learn Abstractions

For classical machine learning, lessons like phases/02-ml-fundamentals/02-linear-regression/docs/en.md demonstrate how the same problem—such as linear regression—can be solved three ways: NumPy implementation, scikit‑learn one‑liner (LinearRegression().fit(X, y)), and discussion of edge cases the library handles automatically (numerical stability, regularization).

Side‑by‑Side Technical Comparison

Aspect	AI Engineering from Scratch	PyTorch	scikit‑learn
Goal	Understand mathematical foundations and infrastructure	Production‑ready, GPU‑accelerated deep learning	High‑level API for classical ML pipelines
Implementation	~500 LOC hand‑written mini‑framework (Module, Linear, Sequential)	torch.nn.Module, torch.nn.Sequential, autograd	Ready‑made estimators (LinearRegression, LogisticRegression)
Autograd	Manual backward methods per layer	Automatic differentiation via loss.backward()	No explicit gradient handling; algorithms pre‑implemented
Device Support	CPU‑only pure Python	Transparent CPU/GPU via torch.device and .to()	CPU‑only; delegates to NumPy/SciPy
Performance	~300 seconds/epoch (MNIST MLP on CPU)	~0.5 seconds/epoch (CPU), ~5 seconds/epoch (GPU)	Competitive for small‑to‑medium datasets via vectorized NumPy
Educational Value	Explicit forward/right‑hand side of every equation; visible weight initialization and gradient flow	Immediate comparison reveals hidden mechanics (autograd, device placement)	Understanding when to replace custom code with battle‑tested implementations

Practical Code Comparison: Circle Classification

The following examples solve identical binary classification problems (circle data) at three abstraction levels, demonstrating the mental mapping taught in the curriculum.

1. Hand‑Crafted Framework

import random, math

# Module, Linear, ReLU, Sigmoid, BCELoss, Adam, DataLoader defined in lesson

def make_circle_data(n=500):
    random.seed(42)
    data = []
    for _ in range(n):
        x = random.uniform(-2, 2)
        y = random.uniform(-2, 2)
        label = 1.0 if x*x + y*y < 1.5 else 0.0
        data.append(([x, y], [label]))
    return data

model = Sequential(
    Linear(2, 16), ReLU(),
    Linear(16, 8), ReLU(),
    Linear(8, 1), Sigmoid(),
)
criterion = BCELoss()
optimizer = Adam(model.parameters(), lr=0.01)
loader = DataLoader(make_circle_data(), batch_size=16, shuffle=True)

model.train()
for epoch in range(100):
    for inputs, targets in loader:
        optimizer.zero_grad()
        loss = 0
        grads = []
        for x, t in zip(inputs, targets):
            pred = model.forward(x)
            loss += criterion(pred, t)
            grads.append(criterion.backward())
        for grad in grads:
            model.backward(grad)
        optimizer.step()

2. PyTorch Implementation

import torch, torch.nn as nn, torch.optim as optim
from torch.utils.data import DataLoader, TensorDataset

def make_circle_data_tensor():
    torch.manual_seed(42)
    X = torch.empty(500, 2).uniform_(-2, 2)
    y = (X[:,0]**2 + X[:,1]**2 < 1.5).float().unsqueeze(1)
    return TensorDataset(X, y)

model = nn.Sequential(
    nn.Linear(2, 16), nn.ReLU(),
    nn.Linear(16, 8), nn.ReLU(),
    nn.Linear(8, 1), nn.Sigmoid(),
)
criterion = nn.BCELoss()
optimizer = optim.Adam(model.parameters(), lr=0.01)
loader = DataLoader(make_circle_data_tensor(), batch_size=16, shuffle=True)

model.train()
for epoch in range(100):
    for x, t in loader:
        optimizer.zero_grad()
        pred = model(x)
        loss = criterion(pred, t)
        loss.backward()
        optimizer.step()

3. scikit‑learn Approach

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

def make_circle_data_np():
    np.random.seed(42)
    X = np.random.uniform(-2, 2, size=(500, 2))
    y = (X[:,0]**2 + X[:,1]**2 < 1.5).astype(int)
    return X, y

X, y = make_circle_data_np()
clf = LogisticRegression(max_iter=1000)
clf.fit(X, y)

pred = clf.predict(X)
print('Training accuracy:', accuracy_score(y, pred))

The progression illustrates how AI Engineering from Scratch makes every forward and backward step explicit, while PyTorch hides gradients behind autograd and scikit‑learn abstracts the entire pipeline into a single estimator call.

When to Use Each Approach

Use the mini‑framework when you need to understand gradient flow, debug vanishing gradients, or implement custom layers not available in standard libraries. The explicit code in phases/03-deep-learning-core/10-mini-framework/docs/en.md reveals exactly why optimizer.zero_grad() matters and how weight initialization affects convergence.

Use PyTorch when you need GPU acceleration, automatic differentiation, or production deployment. As shown in phases/03-deep-learning-core/11-intro-to-pytorch/docs/en.md, PyTorch provides optimized C++/CUDA kernels that run orders of magnitude faster than pure Python implementations.

Use scikit‑learn for classical machine learning tasks on small‑to‑medium datasets where you need robust, battle‑tested implementations of algorithms like linear regression, logistic regression, or TF‑IDF vectorization (covered in phases/05-nlp-foundations-to-advanced/02-bag-of-words-tfidf/docs/en.md).

Summary

• AI Engineering from Scratch builds a mental model by implementing tensors, back‑propagation, and optimizers in pure Python before introducing external libraries.
• The Build It / Use It pattern creates one‑to‑one mappings between custom Module classes and torch.nn.Module, or between NumPy implementations and scikit‑learn estimators.
• Performance trade‑offs are explicit: the educational framework runs ~300 seconds per MNIST epoch versus PyTorch’s ~0.5 seconds on CPU, illustrating the cost of abstraction.
• Autograd mechanics are demystified by writing manual backward methods first, making loss.backward() in PyTorch transparent rather than magical.
• File references such as phases/03-deep-learning-core/10-mini-framework/docs/en.md provide the complete source code for the hand‑crafted framework and its PyTorch equivalents.

Frequently Asked Questions

Does AI Engineering from Scratch replace PyTorch and scikit‑learn?

No. The curriculum teaches you the foundations so you can use PyTorch and scikit‑learn effectively. By implementing Linear, BatchNorm, and Adam from scratch in phases/03-deep-learning-core/10-mini-framework/docs/en.md, you gain the intuition to debug framework code and understand what happens inside torch.optim.Adam or sklearn.linear_model.LinearRegression.

How does the mini‑framework handle automatic differentiation?

It does not. You write explicit backward methods for every layer that compute gradients manually using the chain rule. This contrasts with PyTorch’s tape‑based autograd, which records operations during the forward pass and replays them during loss.backward(). The manual approach appears in the hand‑crafted Module base class defined in the deep learning core lessons.

Can I use the mini‑framework for production workloads?

No. The pure Python implementation lacks GPU support and optimized kernels, resulting in performance approximately 600× slower than PyTorch on CPU for MNIST classification. The framework serves educational purposes only, explicitly designed to answer “what does the library hide?” before you transition to industrial frameworks.

What is the “Build It / Use It” pattern?

This pedagogical structure requires you to construct a component—such as a DataLoader or TF‑IDF vectorizer—from first principles, then immediately compare it to the PyTorch or scikit‑learn equivalent. Lessons like phases/05-nlp-foundations-to-advanced/02-bag-of-words-tfidf/docs/en.md show the from‑scratch implementation followed by the three‑line scikit‑learn solution, highlighting numerical stability and edge cases the library handles automatically.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}