How to Implement ML Algorithms from Scratch: The Complete AI Engineering from Scratch Curriculum

You can implement ML algorithms from scratch by progressing through the phased lessons in the rohitg00/ai-engineering-from-scratch repository, which teaches you to code perceptrons, automatic differentiation engines, transformers, and mini-GPTs using pure Python, Rust, TypeScript, or Julia before introducing any external frameworks.

The rohitg00/ai-engineering-from-scratch curriculum provides a systematic approach to implement ML algorithms from scratch by deconstructing modern AI into foundational mathematical components. Each lesson follows a "Build-It/Use-It" pattern where you first hand-code algorithms from first principles, then immediately apply them to solve concrete problems like XOR classification or language modeling. This methodology ensures you understand the underlying mechanics of gradient flow, matrix operations, and attention mechanisms rather than simply calling library functions.

Curriculum Architecture and Learning Phases

The repository organizes content into sequential phases, with each phase containing focused lessons that build upon previous implementations. Every lesson provides four core artifacts: docs/en.md for theory, code/ for reference implementations, tests/ for deterministic validation, and quiz.json for concept verification.

The progression moves from mathematical primitives to production-grade architectures:

• Phase 01-02: Linear algebra foundations and sampling methods, including the reparameterization trick for stochastic nodes
• Phase 03: Deep learning core components, starting with single neurons and culminating in a custom reverse-mode autodiff engine
• Phase 07: Transformer deep-dive, implementing scaled dot-product attention and multi-head mechanisms
• Phase 10: Large language model construction, culminating in a pre-trained mini-GPT with your custom backpropagation stack

Building Core Components From First Principles

The Perceptron and Learning Rules

In phases/03-deep-learning-core/01-the-perceptron/code/perceptron.py, you implement the foundational Perceptron class that demonstrates how single neurons learn through error correction. This lesson establishes the weight update rule and exposes why linear separability fails on XOR problems, motivating the need for multi-layer architectures.

class Perceptron:
    def __init__(self, n_inputs, learning_rate=0.1):
        self.weights = [0.0] * n_inputs
        self.bias = 0.0
        self.lr = learning_rate

    def predict(self, inputs):
        total = sum(w * x for w, x in zip(self.weights, inputs)) + self.bias
        return 1 if total >= 0 else 0

    def train(self, training_data, epochs=100):
        for epoch in range(epochs):
            errors = 0
            for inputs, target in training_data:
                pred = self.predict(inputs)
                error = target - pred
                if error:
                    errors += 1
                    for i in range(len(self.weights)):
                        self.weights[i] += self.lr * error * inputs[i]
                    self.bias += self.lr * error
            if errors == 0:
                print(f"Converged at epoch {epoch+1}")
                break

The train() method iteratively adjusts weights using the perceptron learning rule until convergence, providing the algorithmic foundation before you stack these units into deep networks.

Custom Automatic Differentiation Engine

Phase 03-03 requires you to build a Value class in phases/03-deep-learning-core/03-backpropagation/code/main.py that implements reverse-mode automatic differentiation. This custom autograd engine computes gradients through computational graphs using topological sorting, enabling you to train neural networks without PyTorch or TensorFlow.

class Value:
    def __init__(self, data, children=(), op=''):
        self.data = data
        self.grad = 0.0
        self._backward = lambda: None
        self._children = set(children)
        self._op = op

    def __add__(self, other):
        other = other if isinstance(other, Value) else Value(other)
        out = Value(self.data + other.data, (self, other), '+')
        def _backward():
            self.grad += out.grad
            other.grad += out.grad
        out._backward = _backward
        return out

    def backward(self):
        topo = []
        visited = set()
        def build_topo(v):
            if v not in visited:
                visited.add(v)
                for child in v._children:
                    build_topo(child)
                topo.append(v)
        build_topo(self)
        self.grad = 1.0
        for v in reversed(topo):
            v._backward()

The backward() method performs a topological sort via build_topo() to ensure gradients flow correctly through the computational graph, applying the chain rule in reverse topological order. This engine powers all subsequent training loops in the curriculum.

Transformer Architectures and Self-Attention

Moving to phases/07-transformers-deep-dive/02-self-attention-from-scratch/code/self_attention.py, you implement scaled dot-product attention using pure NumPy or native language operations. This lesson dissects the query-key-value mechanism that underpins modern LLMs, teaching you how attention weights calculate context-aware representations without matrix multiplication black boxes.

Mini-GPT Training Pipeline

In phases/10-llms-from-scratch/04-pre-training-mini-gpt/code/main.py, you assemble previous components into a complete language model. The implementation leverages your custom Value class for backpropagation while managing multi-head attention blocks and feed-forward networks:

def train_gpt(tokens, vocab_size, n_layers=2, n_heads=4, d_model=64):
    model = Transformer(vocab_size, n_layers, n_heads, d_model)
    optimizer = AdamW(model.parameters(), lr=1e-3)
    for epoch in range(num_epochs):
        for batch in DataLoader(tokens, batch_size=32):
            logits = model(batch.inputs)
            loss = cross_entropy(logits, batch.targets)
            loss.backward()
            optimizer.step()
            optimizer.zero_grad()

This training loop demonstrates how to integrate your custom autograd engine with optimizer states and batch processing, bridging the gap between algorithmic understanding and production ML workflows.

Implementation Roadmap

To effectively implement ML algorithms from scratch using this curriculum:

1. Start with Phase 01 to establish mathematical intuition for linear algebra operations and probability sampling that underpin all ML systems.

2. Complete Phase 03 sequentially by first implementing the Perceptron class, then extending it to multi-layer networks, and finally replacing manual derivatives with your Value autograd engine to solve XOR and circle classification problems.

3. Validate with unit tests after each lesson using python -m unittest discover in the respective tests/ directories to ensure your implementations match deterministic expectations.

4. Progress through Phase 07 to implement attention mechanisms, then apply these in Phase 10 to train your mini-GPT on real text corpora using the backpropagation engine you built in Phase 03.

5. Review docs/en.md files to understand the mathematical theory behind each implementation before examining the code/ reference solutions.

Summary

• The rohitg00/ai-engineering-from-scratch curriculum teaches you to implement ML algorithms from scratch through progressive phases covering math foundations, deep learning cores, and transformer architectures.
• You will build a custom autograd engine in phases/03-deep-learning-core/03-backpropagation/code/main.py that performs reverse-mode automatic differentiation via topological sorting.
• The perceptron implementation in phases/03-deep-learning-core/01-the-perceptron/code/perceptron.py establishes the foundational learning rules before you scale to multi-layer networks.
• Each lesson provides deterministic unit tests, theoretical documentation, and quizzes to verify your understanding before proceeding.
• The curriculum culminates in training a mini-GPT using your custom backpropagation stack, demonstrating how first-principles implementations scale to production-grade AI systems.

Frequently Asked Questions

Do I need prior machine learning experience to implement these algorithms from scratch?

No prior ML framework experience is required, but you should understand basic Python programming and high school-level calculus. The curriculum specifically targets Phase 01 (math foundations) and Phase 02 (linear algebra) to build the necessary mathematical intuition before you encounter neural networks. Each docs/en.md file explains the theoretical concepts while the code/ directory provides working implementations for reference.

Which programming languages does the curriculum support?

The reference implementations primarily use Python, with alternative versions available in Rust, TypeScript, and Julia for specific lessons like the self-attention mechanism. The core autograd engine and perceptron implementations are provided in Python to maximize accessibility, while the multi-language support demonstrates how these algorithms translate across different memory models and performance characteristics.

How does the custom backpropagation engine handle complex computational graphs?

The Value class in phases/03-deep-learning-core/03-backpropagation/code/main.py uses reverse-mode automatic differentiation with a topological sort implemented in the build_topo() function. When you call backward(), the engine traverses the computational graph from output to inputs, accumulating gradients via the chain rule stored in _backward closures. This approach efficiently computes gradients for arbitrary neural network architectures without external dependencies.

Can I use these implementations for production machine learning systems?

While the curriculum implementations are pedagogically minimal, they follow the same API patterns and mathematical correctness as production frameworks like PyTorch. The Transformer and mini-GPT implementations demonstrate production patterns including proper initialization, optimizer integration, and batch processing. However, for production deployment, you should transition to optimized frameworks, as these scratch implementations prioritize educational clarity over GPU acceleration or distributed training optimizations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}