# Phase 7 Transformers Deep Dive vs Phase 10 LLMs from Scratch: 4 Critical Differences

> Explore critical differences between Phase 7 Transformers Deep Dive and Phase 10 LLMs from Scratch. Understand transformer building blocks versus a complete LLM pipeline.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: deep-dive
- Published: 2026-08-29

---

**Phase 7 dissects transformer building blocks like self-attention and KV-caching in isolation, while Phase 10 assembles these components into a complete production LLM pipeline spanning tokenization to deployment.**

The `rohitg00/ai-engineering-from-scratch` repository structures its curriculum to first teach the mathematical foundations of transformers before scaling to industrial systems. Understanding the distinction between **Phase 7 Transformers Deep Dive** and **Phase 10 LLMs from Scratch** (also labeled "Phase IO") helps learners choose the right entry point for their engineering goals.

## Core Philosophy: Components vs. Systems

Phase 7 follows a "build-it-first, use-it-later" approach. It contains 16 focused lessons that isolate individual transformer mechanisms, implementing them in pure Python (and occasionally Rust or Julia) before comparing them to library equivalents.

Phase 10 adopts a "from-scratch-to-library" methodology across 24-plus lessons. After demonstrating low-level concepts, it immediately transitions to realistic training pipelines using PyTorch, HuggingFace, DeepSpeed, and other production frameworks.

## Scope and Depth Comparison

### Phase 7: Mathematical Foundations

This phase emphasizes depth over breadth. You implement scaled dot-product attention, multi-head attention, and positional encodings (sinusoidal, RoPE, ALiBi) from first principles. The capstone projects remain pedagogical—small, self-contained modules that illustrate single concepts.

Key topics include:
- Self-attention mathematics and code
- KV-cache optimization and Flash Attention algorithms
- Mixture-of-Experts (MoE) routing mechanisms
- Scaling laws and speculative decoding theory

### Phase 10: Full-Stack Engineering

Phase 10 prioritizes the breadth of the LLM lifecycle. Rather than isolated algorithms, you build complete artifacts: tokenizers, trained model checkpoints, evaluation suites, and deployment-ready inference servers.

The curriculum covers:
- Byte-Pair Encoding (BPE) and WordPiece tokenizers
- Distributed training with FSDP and DeepSpeed
- Instruction-tuning, RLHF, and Direct Preference Optimization (DPO)
- Quantization strategies (INT8, GPTQ, AWQ)
- Advanced architectures like Jamba (hybrid SSM-Transformer) and DeepSeek-V3

## Code Implementation Styles

### Phase 7: Educational Modularity

In [`phases/07-transformers-deep-dive/02-self-attention-from-scratch/code/self_attention.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/02-self-attention-from-scratch/code/self_attention.py), the repository provides a minimal implementation that prioritizes clarity over performance:

```python

# Phase 7 – Self-Attention from Scratch (Python)

# File: phases/07-transformers-deep-dive/02-self-attention-from-scratch/code/self_attention.py

def self_attention(Q, K, V):
    """Simple scaled dot‑product self‑attention."""
    dk = Q.shape[-1]
    scores = Q @ K.transpose(-2, -1) / (dk ** 0.5)
    weights = scores.softmax(dim=-1)
    return weights @ V

```

Similarly, positional encoding implementations reside in [`phases/07-transformers-deep-dive/04-positional-encoding/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/04-positional-encoding/code/main.py), while KV-cache demonstrations appear in [`phases/07-transformers-deep-dive/12-kv-cache-flash-attention/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/12-kv-cache-flash-attention/code/main.py).

### Phase 10: Production Pipelines

Phase 10 begins with foundational utilities like the BPE tokenizer in [`phases/10-llms-from-scratch/01-tokenizers/code/bpe.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/10-llms-from-scratch/01-tokenizers/code/bpe.py):

```python

# Phase 10 – Byte‑Pair Encoding Tokenizer (Python)

# File: phases/10-llms-from-scratch/01-tokenizers/code/bpe.py

class BPETokenizer:
    def __init__(self, vocab, merges):
        self.vocab = vocab
        self.merges = merges

    def encode(self, text):
        # Split into characters, then apply merges greedily

        tokens = list(text)
        for pair in self.merges:
            tokens = self._merge_pair(tokens, pair)
        return [self.vocab[t] for t in tokens]

```

The phase progresses to full training loops. The file [`phases/10-llms-from-scratch/04-pre-training-mini-gpt/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/10-llms-from-scratch/04-pre-training-mini-gpt/code/main.py) contains a complete script for training a 124M-parameter GPT:

```python

# Phase 10 – Mini‑GPT Pre‑Training Loop (Python)

# File: phases/10-llms-from-scratch/04-pre-training-mini-gpt/code/main.py

def train_gpt(model, dataloader, optimizer, epochs=1):
    model.train()
    for epoch in range(epochs):
        for batch in dataloader:
            inputs, targets = batch, batch
            logits = model(inputs)
            loss = torch.nn.functional.cross_entropy(logits.view(-1, logits.size(-1)), targets.view(-1))
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

```

## Critical Source Files reference

| Phase | Lesson | File Path | Description |
|-------|--------|-----------|-------------|
| **Phase 7** | Self-Attention | [`phases/07-transformers-deep-dive/02-self-attention-from-scratch/code/self_attention.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/02-self-attention-from-scratch/code/self_attention.py) | Minimal scaled dot-product implementation |
| **Phase 7** | Positional Encoding | [`phases/07-transformers-deep-dive/04-positional-encoding/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/04-positional-encoding/code/main.py) | Sinusoidal, RoPE, and ALiBi variants |
| **Phase 7** | KV-Cache | [`phases/07-transformers-deep-dive/12-kv-cache-flash-attention/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/12-kv-cache-flash-attention/code/main.py) | Inference optimization techniques |
| **Phase 10** | Tokenizers | [`phases/10-llms-from-scratch/01-tokenizers/code/bpe.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/10-llms-from-scratch/01-tokenizers/code/bpe.py) | BPE algorithm from scratch |
| **Phase 10** | Pre-Training | [`phases/10-llms-from-scratch/04-pre-training-mini-gpt/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/10-llms-from-scratch/04-pre-training-mini-gpt/code/main.py) | End-to-end 124M GPT training |
| **Phase 10** | Speculative Decoding | [`phases/10-llms-from-scratch/15-speculative-decoding-eagle3/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/10-llms-from-scratch/15-speculative-decoding-eagle3/code/main.py) | Draft-then-verify (EAGLE-3) implementation |
| **Phase 10** | Jamba Architecture | [`phases/10-llms-from-scratch/21-jamba-hybrid-ssm-transformer/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/10-llms-from-scratch/21-jamba-hybrid-ssm-transformer/code/main.py) | Hybrid State Space Model transformer |

## Learning Outcomes

Completing **Phase 7 Transformers Deep Dive** enables you to read and modify core transformer operations, understand the mathematical justification for attention mechanisms, and experiment with architectural variants like MoE or sparse attention.

Finishing **Phase 10 LLMs from Scratch** prepares you to assemble complete training pipelines, implement distributed training strategies, fine-tune models via RLHF or DPO, and deploy quantized models with optimized inference kernels like Flash Attention.

## Summary

- **Phase 7** provides the *nuts and bolts* of transformer mathematics, implementing attention and encoding mechanisms in isolation.
- **Phase 10** demonstrates *how to assemble those nuts and bolts* into production LLM systems, covering the full lifecycle from tokenization to deployment.
- Phase 7 contains 16 lessons focusing on deep technical dissection; Phase 10 contains 24-plus lessons emphasizing systems engineering.
- Code in Phase 7 lives in educational modules like [`self_attention.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/self_attention.py), while Phase 10 produces full-stack artifacts including [`bpe.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/bpe.py) and distributed training scripts.

## Frequently Asked Questions

### Should I complete Phase 7 before starting Phase 10?

Yes. According to the repository structure, Phase 10 assumes mastery of attention mechanisms, positional encodings, and scaling laws covered in Phase 7. The pedagogical continuity ensures you understand *why* transformers work before learning *how to scale* them across GPU clusters.

### Why is Phase 10 sometimes called "Phase IO"?

"Phase IO" uses the Roman numeral ten (IO) to represent Phase 10. The repository uses both terms interchangeably when referencing content in `phases/10-llms-from-scratch/`, particularly in documentation and file annotations.

### Does Phase 10 re-implement everything from scratch or use libraries?

Phase 10 employs a hybrid strategy. Initial lessons implement components like BPE tokenizers from scratch in [`bpe.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/bpe.py), but quickly transition to industrial libraries such as PyTorch Distributed, DeepSpeed, and HuggingFace Transformers for realistic training scenarios.

### Which phase covers Flash Attention?

Both phases address Flash Attention, but with different objectives. Phase 7 implements the algorithm pedagogically in [`12-kv-cache-flash-attention/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/12-kv-cache-flash-attention/code/main.py) to explain memory-efficient attention mechanics. Phase 10 integrates optimized Flash Attention kernels into production inference pipelines for serving large models.