Transformers Deep Dive Phase 7: 16 Lessons Covering Self-Attention, BERT, GPT, and Modern Efficiency Techniques

Phase 7 of the AI Engineering from Scratch curriculum provides a comprehensive, hands-on exploration of transformer architectures, covering everything from basic self-attention implementations in NumPy to advanced inference optimizations like KV-Cache, Flash Attention, and speculative decoding.

The Transformers Deep Dive Phase 7 in the rohitg00/ai-engineering-from-scratch repository delivers a systematic breakdown of modern neural network architectures. Spanning sixteen progressive lessons, this phase moves from foundational mathematics to production-grade efficiency techniques, with every component implemented from scratch to ensure deep understanding of the underlying mechanics.

Core Architecture Foundations (Lessons 1–5)

Why Transformers Abandon Recurrence

Lesson 01 establishes the motivation for transformer architectures by contrasting them with RNN limitations. According to phases/07-transformers-deep-dive/01-why-transformers/docs/en.md, the curriculum explains how self-attention reduces serial depth from O(N) to O(1), enabling massive GPU parallelism at the cost of O(N²) memory complexity.

Self-Attention from Scratch

Lesson 02 implements the fundamental scaled dot-product attention mechanism using only NumPy. The operation projects tokens into Query (Q), Key (K), and Value (V) matrices, computing attention weights via softmax(QKᵀ / √d_k). This core mathematics appears in phases/07-transformers-deep-dive/02-self-attention-from-scratch/code/main.py.

Multi-Head Attention and Positional Encoding

Lessons 03 and 04 extend the basic mechanism. Multi-head attention splits Q/K/V projections into parallel heads, concatenating results to enrich representational capacity. Positional encoding injects sequence order through sinusoidal embeddings or learnable parameters, addressing the permutation-invariance of self-attention.

Complete Transformer Blocks

Lesson 05 assembles the full architecture in phases/07-transformers-deep-dive/05-full-transformer/code/main.py, stacking N identical layers comprising multi-head attention, feed-forward networks, and layer normalization.

Specialized Model Architectures (Lessons 6–10)

BERT and GPT Implementations

The phase contrasts bidirectional and autoregressive paradigms. Lesson 06 covers BERT's masked language modeling objective for contextual understanding, while Lesson 07 implements GPT's causal language modeling with autoregressive generation strategies including temperature scaling and top-k/top-p sampling.

Encoder-Decoder and Multimodal Transformers

Lessons 08 through 10 expand into specialized domains:

• T5/BART encoder-decoder architectures for sequence-to-sequence tasks using teacher-forcing
• Vision Transformers (ViT) applying patch-based attention to image classification
• Audio Transformers (Whisper) processing spectrogram inputs for multilingual speech-to-text

Efficiency Optimizations and Scaling (Lessons 11–13, 15–16)

Mixture-of-Experts and Memory Optimization

Lesson 11 implements Mixture-of-Experts (MoE) routing networks that scale parameter counts without linear compute increases by activating only relevant expert subnetworks per token.

Lesson 12 tackles inference bottlenecks through KV-Cache and Flash Attention. The KV-Cache stores previously computed key/value pairs to avoid redundant calculations during autoregressive generation, while Flash Attention uses tiled algorithms to reduce the quadratic memory complexity of attention.

Scaling Laws and Alternative Mechanisms

Lesson 13 explores empirical scaling laws governing the relationship between model size, data volume, and performance, incorporating insights from Chinchilla, Gopher, and PaLM-2 research.

Lesson 15 surveys efficient attention variants including Linear Attention, Performer, and Reformer that approximate full attention with sub-quadratic complexity. Lesson 16 introduces speculative decoding, which uses a lightweight draft model to predict multiple tokens ahead, dramatically reducing latency while maintaining output quality.

Implementation Code Examples

Scaled Dot-Product Attention

The foundational attention mechanism appears in phases/07-transformers-deep-dive/02-self-attention-from-scratch/code/main.py:

import numpy as np

def softmax(x):
    shifted = x - np.max(x, axis=-1, keepdims=True)
    exp_x = np.exp(shifted)
    return exp_x / np.sum(exp_x, axis=-1, keepdims=True)

def scaled_dot_product_attention(Q, K, V):
    dk = Q.shape[-1]
    scores = Q @ K.T / np.sqrt(dk)
    weights = softmax(scores)
    return weights @ V, weights

Multi-Head Attention Wrapper

From phases/07-transformers-deep-dive/03-multi-head-attention/code/main.py:

class MultiHeadAttention:
    def __init__(self, d_model, n_heads, seed=42):
        self.n_heads = n_heads
        self.dk = d_model // n_heads
        rng = np.random.default_rng(seed)
        self.Wq = rng.normal(0, 1, (d_model, d_model))
        self.Wk = rng.normal(0, 1, (d_model, d_model))
        self.Wv = rng.normal(0, 1, (d_model, d_model))
        self.Wo = rng.normal(0, 1, (d_model, d_model))

    def split_heads(self, X):
        return X.reshape(X.shape[0], self.n_heads, self.dk)

    def forward(self, X):
        Q = self.split_heads(X @ self.Wq)
        K = self.split_heads(X @ self.Wk)
        V = self.split_heads(X @ self.Wv)

        heads_out = []
        for h in range(self.n_heads):
            out, _ = scaled_dot_product_attention(Q[:, h, :], K[:, h, :], V[:, h, :])
            heads_out.append(out)

        concat = np.concatenate(heads_out, axis=-1)
        return concat @ self.Wo

KV-Cache for Fast Inference

The autoregressive optimization from phases/07-transformers-deep-dive/12-kv-cache-flash-attention/code/main.py:

class KVCache:
    def __init__(self):
        self.keys = []
        self.values = []

    def append(self, k, v):
        self.keys.append(k)
        self.values.append(v)

    def get(self):
        return np.concatenate(self.keys, axis=0), np.concatenate(self.values, axis=0)

Speculative Decoding Skeleton

The latency reduction technique from phases/07-transformers-deep-dive/16-speculative-decoding/code/main.py:

def speculative_decode(draft_model, target_model, prompt, max_len=50):
    tokens = draft_model.generate(prompt, max_len=10)
    verified = target_model.generate(prompt + tokens, max_len=1)
    return verified

Capstone Integration (Lesson 14)

The phase culminates in phases/07-transformers-deep-dive/14-build-a-transformer-capstone/code/main.py, where learners construct an end-to-end transformer from scratch using only fundamental libraries. This integration tests understanding of embedding layers, attention mechanisms, feed-forward networks, and training loops against synthetic datasets, reinforcing debugging skills for common initialization and gradient issues.

Summary

• Transformers Deep Dive Phase 7 comprises sixteen progressive lessons covering theoretical foundations through production optimizations.
• Core implementations include scaled dot-product attention, multi-head attention, and full transformer blocks written in NumPy.
• Specialized architectures span BERT, GPT, T5/BART, Vision Transformers, and Whisper audio models.
• Efficiency modules teach KV-Cache, Flash Attention, Mixture-of-Experts, scaling laws, and speculative decoding for high-performance inference.
• Every lesson provides runnable code located in phases/07-transformers-deep-dive/, with the capstone project integrating all components into a working model.

Frequently Asked Questions

What prerequisites are needed for Transformers Deep Dive Phase 7?

Learners should understand matrix operations and basic neural network concepts, including backpropagation and gradient descent. The implementations use NumPy and Python fundamentals, requiring no prior transformer experience as the phase builds from first principles in phases/07-transformers-deep-dive/01-why-transformers/.

How does the phase cover modern efficiency techniques like Flash Attention?

Lesson 12 specifically addresses KV-Cache and Flash Attention through practical implementations in phases/07-transformers-deep-dive/12-kv-cache-flash-attention/code/main.py, demonstrating how caching key/value pairs and tiled attention algorithms reduce the quadratic memory bottleneck during autoregressive generation.

Are the transformer implementations built from scratch or using libraries?

Every architectural component is implemented from scratch using only NumPy and basic Python in the early lessons, later transitioning to PyTorch for scaling. The MultiHeadAttention class and scaled_dot_product_attention function demonstrate this hands-on approach, ensuring learners understand the underlying mathematics rather than relying solely on high-level APIs.

What is the difference between the BERT and GPT implementations in this phase?

BERT (Lesson 06) implements bidirectional masked language modeling with token-level masking for contextual understanding, while GPT (Lesson 07) implements causal autoregressive modeling with triangular masking to prevent future token access, supported by generation strategies including temperature scaling and top-k/top-p sampling as detailed in their respective docs/en.md files.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}