# Transformers Deep Dive Phase 7: 16 Lessons Covering Self-Attention, BERT, GPT, and Modern Efficiency Techniques

> Explore transformer architectures in Phase 7, mastering self-attention, BERT, GPT, and efficiency techniques like KV-Cache and Flash Attention. Build practical skills.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: deep-dive
- Published: 2026-08-26

---

**Phase 7 of the AI Engineering from Scratch curriculum provides a comprehensive, hands-on exploration of transformer architectures, covering everything from basic self-attention implementations in NumPy to advanced inference optimizations like KV-Cache, Flash Attention, and speculative decoding.**

The **Transformers Deep Dive Phase 7** in the `rohitg00/ai-engineering-from-scratch` repository delivers a systematic breakdown of modern neural network architectures. Spanning sixteen progressive lessons, this phase moves from foundational mathematics to production-grade efficiency techniques, with every component implemented from scratch to ensure deep understanding of the underlying mechanics.

## Core Architecture Foundations (Lessons 1–5)

### Why Transformers Abandon Recurrence

Lesson 01 establishes the motivation for transformer architectures by contrasting them with RNN limitations. According to [`phases/07-transformers-deep-dive/01-why-transformers/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/01-why-transformers/docs/en.md), the curriculum explains how **self-attention** reduces serial depth from **O(N)** to **O(1)**, enabling massive GPU parallelism at the cost of **O(N²)** memory complexity.

### Self-Attention from Scratch

Lesson 02 implements the fundamental **scaled dot-product attention** mechanism using only NumPy. The operation projects tokens into **Query (Q)**, **Key (K)**, and **Value (V)** matrices, computing attention weights via `softmax(QKᵀ / √d_k)`. This core mathematics appears in [`phases/07-transformers-deep-dive/02-self-attention-from-scratch/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/02-self-attention-from-scratch/code/main.py).

### Multi-Head Attention and Positional Encoding

Lessons 03 and 04 extend the basic mechanism. **Multi-head attention** splits Q/K/V projections into parallel heads, concatenating results to enrich representational capacity. **Positional encoding** injects sequence order through sinusoidal embeddings or learnable parameters, addressing the permutation-invariance of self-attention.

### Complete Transformer Blocks

Lesson 05 assembles the full architecture in [`phases/07-transformers-deep-dive/05-full-transformer/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/05-full-transformer/code/main.py), stacking **N** identical layers comprising multi-head attention, feed-forward networks, and layer normalization.

## Specialized Model Architectures (Lessons 6–10)

### BERT and GPT Implementations

The phase contrasts bidirectional and autoregressive paradigms. **Lesson 06** covers **BERT's masked language modeling** objective for contextual understanding, while **Lesson 07** implements **GPT's causal language modeling** with autoregressive generation strategies including temperature scaling and top-k/top-p sampling.

### Encoder-Decoder and Multimodal Transformers

Lessons 08 through 10 expand into specialized domains:

- **T5/BART encoder-decoder architectures** for sequence-to-sequence tasks using teacher-forcing
- **Vision Transformers (ViT)** applying patch-based attention to image classification
- **Audio Transformers (Whisper)** processing spectrogram inputs for multilingual speech-to-text

## Efficiency Optimizations and Scaling (Lessons 11–13, 15–16)

### Mixture-of-Experts and Memory Optimization

**Lesson 11** implements **Mixture-of-Experts (MoE)** routing networks that scale parameter counts without linear compute increases by activating only relevant expert subnetworks per token.

**Lesson 12** tackles inference bottlenecks through **KV-Cache** and **Flash Attention**. The KV-Cache stores previously computed key/value pairs to avoid redundant calculations during autoregressive generation, while Flash Attention uses tiled algorithms to reduce the quadratic memory complexity of attention.

### Scaling Laws and Alternative Mechanisms

**Lesson 13** explores empirical **scaling laws** governing the relationship between model size, data volume, and performance, incorporating insights from Chinchilla, Gopher, and PaLM-2 research.

**Lesson 15** surveys efficient attention variants including **Linear Attention**, **Performer**, and **Reformer** that approximate full attention with sub-quadratic complexity. **Lesson 16** introduces **speculative decoding**, which uses a lightweight draft model to predict multiple tokens ahead, dramatically reducing latency while maintaining output quality.

## Implementation Code Examples

### Scaled Dot-Product Attention

The foundational attention mechanism appears in [`phases/07-transformers-deep-dive/02-self-attention-from-scratch/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/02-self-attention-from-scratch/code/main.py):

```python
import numpy as np

def softmax(x):
    shifted = x - np.max(x, axis=-1, keepdims=True)
    exp_x = np.exp(shifted)
    return exp_x / np.sum(exp_x, axis=-1, keepdims=True)

def scaled_dot_product_attention(Q, K, V):
    dk = Q.shape[-1]
    scores = Q @ K.T / np.sqrt(dk)
    weights = softmax(scores)
    return weights @ V, weights

```

### Multi-Head Attention Wrapper

From [`phases/07-transformers-deep-dive/03-multi-head-attention/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/03-multi-head-attention/code/main.py):

```python
class MultiHeadAttention:
    def __init__(self, d_model, n_heads, seed=42):
        self.n_heads = n_heads
        self.dk = d_model // n_heads
        rng = np.random.default_rng(seed)
        self.Wq = rng.normal(0, 1, (d_model, d_model))
        self.Wk = rng.normal(0, 1, (d_model, d_model))
        self.Wv = rng.normal(0, 1, (d_model, d_model))
        self.Wo = rng.normal(0, 1, (d_model, d_model))

    def split_heads(self, X):
        return X.reshape(X.shape[0], self.n_heads, self.dk)

    def forward(self, X):
        Q = self.split_heads(X @ self.Wq)
        K = self.split_heads(X @ self.Wk)
        V = self.split_heads(X @ self.Wv)

        heads_out = []
        for h in range(self.n_heads):
            out, _ = scaled_dot_product_attention(Q[:, h, :], K[:, h, :], V[:, h, :])
            heads_out.append(out)

        concat = np.concatenate(heads_out, axis=-1)
        return concat @ self.Wo

```

### KV-Cache for Fast Inference

The autoregressive optimization from [`phases/07-transformers-deep-dive/12-kv-cache-flash-attention/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/12-kv-cache-flash-attention/code/main.py):

```python
class KVCache:
    def __init__(self):
        self.keys = []
        self.values = []

    def append(self, k, v):
        self.keys.append(k)
        self.values.append(v)

    def get(self):
        return np.concatenate(self.keys, axis=0), np.concatenate(self.values, axis=0)

```

### Speculative Decoding Skeleton

The latency reduction technique from [`phases/07-transformers-deep-dive/16-speculative-decoding/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/16-speculative-decoding/code/main.py):

```python
def speculative_decode(draft_model, target_model, prompt, max_len=50):
    tokens = draft_model.generate(prompt, max_len=10)
    verified = target_model.generate(prompt + tokens, max_len=1)
    return verified

```

## Capstone Integration (Lesson 14)

The phase culminates in [`phases/07-transformers-deep-dive/14-build-a-transformer-capstone/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/14-build-a-transformer-capstone/code/main.py), where learners construct an **end-to-end transformer from scratch** using only fundamental libraries. This integration tests understanding of embedding layers, attention mechanisms, feed-forward networks, and training loops against synthetic datasets, reinforcing debugging skills for common initialization and gradient issues.

## Summary

- **Transformers Deep Dive Phase 7** comprises sixteen progressive lessons covering theoretical foundations through production optimizations.
- Core implementations include **scaled dot-product attention**, **multi-head attention**, and **full transformer blocks** written in NumPy.
- Specialized architectures span **BERT**, **GPT**, **T5/BART**, **Vision Transformers**, and **Whisper audio models**.
- Efficiency modules teach **KV-Cache**, **Flash Attention**, **Mixture-of-Experts**, **scaling laws**, and **speculative decoding** for high-performance inference.
- Every lesson provides runnable code located in `phases/07-transformers-deep-dive/`, with the capstone project integrating all components into a working model.

## Frequently Asked Questions

### What prerequisites are needed for Transformers Deep Dive Phase 7?

Learners should understand matrix operations and basic neural network concepts, including backpropagation and gradient descent. The implementations use NumPy and Python fundamentals, requiring no prior transformer experience as the phase builds from first principles in `phases/07-transformers-deep-dive/01-why-transformers/`.

### How does the phase cover modern efficiency techniques like Flash Attention?

Lesson 12 specifically addresses **KV-Cache** and **Flash Attention** through practical implementations in [`phases/07-transformers-deep-dive/12-kv-cache-flash-attention/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/12-kv-cache-flash-attention/code/main.py), demonstrating how caching key/value pairs and tiled attention algorithms reduce the quadratic memory bottleneck during autoregressive generation.

### Are the transformer implementations built from scratch or using libraries?

Every architectural component is implemented **from scratch** using only NumPy and basic Python in the early lessons, later transitioning to PyTorch for scaling. The `MultiHeadAttention` class and `scaled_dot_product_attention` function demonstrate this hands-on approach, ensuring learners understand the underlying mathematics rather than relying solely on high-level APIs.

### What is the difference between the BERT and GPT implementations in this phase?

**BERT** (Lesson 06) implements **bidirectional masked language modeling** with token-level masking for contextual understanding, while **GPT** (Lesson 07) implements **causal autoregressive modeling** with triangular masking to prevent future token access, supported by generation strategies including temperature scaling and top-k/top-p sampling as detailed in their respective [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md) files.