Phase 7 Transformers Deep Dive vs Phase 10 LLMs from Scratch: 4 Critical Differences

Phase 7 dissects transformer building blocks like self-attention and KV-caching in isolation, while Phase 10 assembles these components into a complete production LLM pipeline spanning tokenization to deployment.

The rohitg00/ai-engineering-from-scratch repository structures its curriculum to first teach the mathematical foundations of transformers before scaling to industrial systems. Understanding the distinction between Phase 7 Transformers Deep Dive and Phase 10 LLMs from Scratch (also labeled "Phase IO") helps learners choose the right entry point for their engineering goals.

Core Philosophy: Components vs. Systems

Phase 7 follows a "build-it-first, use-it-later" approach. It contains 16 focused lessons that isolate individual transformer mechanisms, implementing them in pure Python (and occasionally Rust or Julia) before comparing them to library equivalents.

Phase 10 adopts a "from-scratch-to-library" methodology across 24-plus lessons. After demonstrating low-level concepts, it immediately transitions to realistic training pipelines using PyTorch, HuggingFace, DeepSpeed, and other production frameworks.

Scope and Depth Comparison

Phase 7: Mathematical Foundations

This phase emphasizes depth over breadth. You implement scaled dot-product attention, multi-head attention, and positional encodings (sinusoidal, RoPE, ALiBi) from first principles. The capstone projects remain pedagogical—small, self-contained modules that illustrate single concepts.

Key topics include:

• Self-attention mathematics and code
• KV-cache optimization and Flash Attention algorithms
• Mixture-of-Experts (MoE) routing mechanisms
• Scaling laws and speculative decoding theory

Phase 10: Full-Stack Engineering

Phase 10 prioritizes the breadth of the LLM lifecycle. Rather than isolated algorithms, you build complete artifacts: tokenizers, trained model checkpoints, evaluation suites, and deployment-ready inference servers.

The curriculum covers:

• Byte-Pair Encoding (BPE) and WordPiece tokenizers
• Distributed training with FSDP and DeepSpeed
• Instruction-tuning, RLHF, and Direct Preference Optimization (DPO)
• Quantization strategies (INT8, GPTQ, AWQ)
• Advanced architectures like Jamba (hybrid SSM-Transformer) and DeepSeek-V3

Code Implementation Styles

Phase 7: Educational Modularity

In phases/07-transformers-deep-dive/02-self-attention-from-scratch/code/self_attention.py, the repository provides a minimal implementation that prioritizes clarity over performance:

# Phase 7 – Self-Attention from Scratch (Python)

# File: phases/07-transformers-deep-dive/02-self-attention-from-scratch/code/self_attention.py

def self_attention(Q, K, V):
    """Simple scaled dot‑product self‑attention."""
    dk = Q.shape[-1]
    scores = Q @ K.transpose(-2, -1) / (dk ** 0.5)
    weights = scores.softmax(dim=-1)
    return weights @ V

Similarly, positional encoding implementations reside in phases/07-transformers-deep-dive/04-positional-encoding/code/main.py, while KV-cache demonstrations appear in phases/07-transformers-deep-dive/12-kv-cache-flash-attention/code/main.py.

Phase 10: Production Pipelines

Phase 10 begins with foundational utilities like the BPE tokenizer in phases/10-llms-from-scratch/01-tokenizers/code/bpe.py:

# Phase 10 – Byte‑Pair Encoding Tokenizer (Python)

# File: phases/10-llms-from-scratch/01-tokenizers/code/bpe.py

class BPETokenizer:
    def __init__(self, vocab, merges):
        self.vocab = vocab
        self.merges = merges

    def encode(self, text):
        # Split into characters, then apply merges greedily

        tokens = list(text)
        for pair in self.merges:
            tokens = self._merge_pair(tokens, pair)
        return [self.vocab[t] for t in tokens]

The phase progresses to full training loops. The file phases/10-llms-from-scratch/04-pre-training-mini-gpt/code/main.py contains a complete script for training a 124M-parameter GPT:

# Phase 10 – Mini‑GPT Pre‑Training Loop (Python)

# File: phases/10-llms-from-scratch/04-pre-training-mini-gpt/code/main.py

def train_gpt(model, dataloader, optimizer, epochs=1):
    model.train()
    for epoch in range(epochs):
        for batch in dataloader:
            inputs, targets = batch, batch
            logits = model(inputs)
            loss = torch.nn.functional.cross_entropy(logits.view(-1, logits.size(-1)), targets.view(-1))
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

Critical Source Files reference

Phase	Lesson	File Path	Description
Phase 7	Self-Attention	phases/07-transformers-deep-dive/02-self-attention-from-scratch/code/self_attention.py	Minimal scaled dot-product implementation
Phase 7	Positional Encoding	phases/07-transformers-deep-dive/04-positional-encoding/code/main.py	Sinusoidal, RoPE, and ALiBi variants
Phase 7	KV-Cache	phases/07-transformers-deep-dive/12-kv-cache-flash-attention/code/main.py	Inference optimization techniques
Phase 10	Tokenizers	phases/10-llms-from-scratch/01-tokenizers/code/bpe.py	BPE algorithm from scratch
Phase 10	Pre-Training	phases/10-llms-from-scratch/04-pre-training-mini-gpt/code/main.py	End-to-end 124M GPT training
Phase 10	Speculative Decoding	phases/10-llms-from-scratch/15-speculative-decoding-eagle3/code/main.py	Draft-then-verify (EAGLE-3) implementation
Phase 10	Jamba Architecture	phases/10-llms-from-scratch/21-jamba-hybrid-ssm-transformer/code/main.py	Hybrid State Space Model transformer

Learning Outcomes

Completing Phase 7 Transformers Deep Dive enables you to read and modify core transformer operations, understand the mathematical justification for attention mechanisms, and experiment with architectural variants like MoE or sparse attention.

Finishing Phase 10 LLMs from Scratch prepares you to assemble complete training pipelines, implement distributed training strategies, fine-tune models via RLHF or DPO, and deploy quantized models with optimized inference kernels like Flash Attention.

Summary

• Phase 7 provides the nuts and bolts of transformer mathematics, implementing attention and encoding mechanisms in isolation.
• Phase 10 demonstrates how to assemble those nuts and bolts into production LLM systems, covering the full lifecycle from tokenization to deployment.
• Phase 7 contains 16 lessons focusing on deep technical dissection; Phase 10 contains 24-plus lessons emphasizing systems engineering.
• Code in Phase 7 lives in educational modules like self_attention.py, while Phase 10 produces full-stack artifacts including bpe.py and distributed training scripts.

Frequently Asked Questions

Should I complete Phase 7 before starting Phase 10?

Yes. According to the repository structure, Phase 10 assumes mastery of attention mechanisms, positional encodings, and scaling laws covered in Phase 7. The pedagogical continuity ensures you understand why transformers work before learning how to scale them across GPU clusters.

Why is Phase 10 sometimes called "Phase IO"?

"Phase IO" uses the Roman numeral ten (IO) to represent Phase 10. The repository uses both terms interchangeably when referencing content in phases/10-llms-from-scratch/, particularly in documentation and file annotations.

Does Phase 10 re-implement everything from scratch or use libraries?

Phase 10 employs a hybrid strategy. Initial lessons implement components like BPE tokenizers from scratch in bpe.py, but quickly transition to industrial libraries such as PyTorch Distributed, DeepSpeed, and HuggingFace Transformers for realistic training scenarios.

Which phase covers Flash Attention?

Both phases address Flash Attention, but with different objectives. Phase 7 implements the algorithm pedagogically in 12-kv-cache-flash-attention/code/main.py to explain memory-efficient attention mechanics. Phase 10 integrates optimized Flash Attention kernels into production inference pipelines for serving large models.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}