# How Transformers Are Built from the Ground Up in Phase 7: A Complete Technical Deep Dive

> Explore how transformers are built from the ground up in Phase 7 of AI Engineering from Scratch. Learn through pure NumPy implementations in 15 progressive lessons.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: deep-dive
- Published: 2026-09-02

---

**Phase 7 of the *AI Engineering from Scratch* curriculum reconstructs every transformer component from first principles—beginning with naive self‑attention and culminating in speculative decoding—using pure NumPy implementations across 15 progressive lessons.**

The `rohitg00/ai-engineering-from-scratch` repository dedicates Phase 7 to a rigorous, ground‑up construction of modern transformer architectures. Unlike tutorials that rely on pre‑built layers, this phase implements **scaled‑dot‑product attention**, **multi‑head projections**, and **positional encodings** from scratch, revealing the mathematical machinery behind models like BERT, GPT, and Whisper.

## Foundational Attention Mechanisms

### Why Transformers Replace Recurrence

The first lesson in [`phases/07-transformers-deep-dive/01-why-transformers/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/01-why-transformers/docs/en.md) establishes the computational bottleneck of RNNs: their sequential hidden‑state updates prevent parallelization across sequence positions. The solution is the **parallel‑attention paradigm**, which computes interactions between all tokens simultaneously using matrix multiplication.

### Self‑Attention from Scratch

In [`phases/07-transformers-deep-dive/02-self-attention-from-scratch/code/self_attention.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/02-self-attention-from-scratch/code/self_attention.py), the core operation is implemented as a pure NumPy function:

```python
def scaled_dot_product_attention(Q, K, V):
    dk = Q.shape[-1]
    scores = Q @ K.T / np.sqrt(dk)
    weights = softmax(scores)
    return weights @ V, weights

```

This routine computes `softmax(QKᵀ/√dₖ)`, the only operation that couples all token positions directly. By scaling with `√dₖ`, the implementation prevents dot‑product magnitudes from pushing the softmax into saturation zones, a critical detail for training stability.

### Multi‑Head Attention

The same file introduces the `MultiHeadSelfAttention` class, which partitions the model dimension `d_model` into `n_heads` independent sub‑spaces. Each head learns distinct relational patterns; their outputs are concatenated and projected back via `self.Wo`:

```python
class MultiHeadSelfAttention:
    def __init__(self, d_model, n_heads, seed=42):
        self.heads = [...]
        self.Wo = rng.normal(0, scale, (n_heads * d_model//n_heads, d_model))

    def forward(self, X):
        head_outputs = [head.forward(X)[0] for head in self.heads]
        concatenated = np.concatenate(head_outputs, axis=-1)
        return concatenated @ self.Wo

```

## Encoding Sequence and Depth

### Sinusoidal Positional Encoding

Since attention is permutation‑invariant, [`phases/07-transformers-deep-dive/04-positional-encoding/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/04-positional-encoding/code/main.py) injects absolute position information through sinusoidal signals:

```python
def positional_encoding(seq_len, d_model):
    pos = np.arange(seq_len)[:, None]
    i = np.arange(d_model)[None, :]
    angle_rates = 1 / np.power(10000, (2 * (i//2)) / d_model)
    angle_rads = pos * angle_rates
    pe = np.empty((seq_len, d_model))
    pe[:, 0::2] = np.sin(angle_rads[:, 0::2])
    pe[:, 1::2] = np.cos(angle_rads[:, 1::2])
    return pe

```

This uses wavelengths that form a geometric progression, allowing the model to learn relative positions via linear transformations of the encoding vectors.

### The Full Transformer Block

[`phases/07-transformers-deep-dive/05-full-transformer/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/05-full-transformer/code/main.py) assembles the complete encoder layer. The `TransformerBlock` class combines multi‑head attention, residual connections, layer normalization, and a position‑wise feed‑forward network:

```python
class TransformerBlock:
    def __init__(self, d_model, n_heads):
        self.mha = MultiHeadSelfAttention(d_model, n_heads)
        self.ffn = FeedForward(d_model)
        self.ln1 = LayerNorm(d_model)
        self.ln2 = LayerNorm(d_model)

    def forward(self, X):
        attn_out, _ = self.mha(self.ln1(X))
        X = X + attn_out                 # residual

        ffn_out = self.ffn(self.ln2(X))
        return X + ffn_out               # residual

```

Stacking `L` such blocks yields the full encoder stack used in BERT and the encoder‑decoder variants.

## Architectural Variants: From BERT to GPT

### BERT and Masked Language Modeling

[`phases/07-transformers-deep-dive/06-bert-masked-language-modeling/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/06-bert-masked-language-modeling/code/main.py) demonstrates how to convert the encoder into a bidirectional model. Random tokens are masked during training, and the model learns to reconstruct them using context from both directions—a process that requires no causal masking.

### GPT: Causal Language Modeling

Conversely, [`phases/07-transformers-deep-dive/07-gpt-causal-language-modeling/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/07-gpt-causal-language-modeling/code/main.py) implements **autoregressive generation** by applying a triangular mask to the attention scores. This ensures each position attends only to previous tokens, enabling next‑token prediction:

```python

# Conceptual example from the GPT lesson

mask = np.triu(np.ones((seq_len, seq_len)), k=1) * -np.inf
scores = (Q @ K.T / np.sqrt(dk)) + mask

```

### Encoder‑Decoder Cross‑Attention

[`phases/07-transformers-deep-dive/08-t5-bart-encoder-decoder/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/08-t5-bart-encoder-decoder/code/main.py) introduces **cross‑attention**, where the decoder’s query matrices attend to the encoder’s final hidden states. This mechanism powers sequence‑to‑sequence tasks like translation, combining the bidirectional encoding of the source with the left‑to‑right generation of the target.

## Cross‑Modal and Production Implementations

### Vision Transformers (ViT)

[`phases/07-transformers-deep-dive/09-vision-transformers/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/09-vision-transformers/code/main.py) adapts the same blocks to image patches. An image is split into fixed‑size patches, linearly embedded, and processed by the standard transformer stack, proving the architecture is modality‑agnostic.

### Audio Transformers (Whisper)

Similarly, [`phases/07-transformers-deep-dive/10-audio-transformers-whisper/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/10-audio-transformers-whisper/code/main.py) applies the attention stack to audio spectrograms after tokenisation, handling speech recognition with the same mathematical primitives used for text.

### KV‑Cache and Flash Attention

For inference optimization, [`phases/07-transformers-deep-dive/12-kv-cache-flash-attention/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/12-kv-cache-flash-attention/code/main.py) implements **KV‑caching** to eliminate redundant computation:

```python
def forward_with_cache(self, X, cache=None):
    Q = X @ self.Wq
    K = X @ self.Wk if cache is None else np.concatenate([cache['K'], X @ self.Wk], axis=0)
    V = X @ self.Wv if cache is None else np.concatenate([cache['V'], X @ self.Wv], axis=0)
    out, _ = scaled_dot_product_attention(Q, K, V)
    return out, {'K': K, 'V': V}

```

By storing previous `K` and `V` matrices, the decoder reuses them for each new token, reducing complexity from `O(N²)` per step to `O(N)`. The lesson also covers **Flash Attention**, which reorders memory access patterns to keep the quadratic term compute‑bound rather than memory‑bound on GPUs.

## Advanced Scaling and Inference Optimizations

### Linear, Sparse, and Rotary Attention Variants

[`phases/07-transformers-deep-dive/15-attention-variants/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/15-attention-variants/code/main.py) addresses the `O(N²)` memory bottleneck through several strategies:

- **Linear attention**: Computes `Q(KᵀV)` instead of `(QKᵀ)V`, reducing memory to linear complexity.
- **Rotary Position Embeddings (RoPE)**: Encodes absolute position directly into the dot‑product via rotation matrices, eliminating the need for additive positional encodings.

### Mixture‑of‑Experts (MoE)

[`phases/07-transformers-deep-dive/11-mixture-of-experts/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/11-mixture-of-experts/code/main.py) scales model capacity without linearly increasing compute. A **gating network** routes each token to a sparse subset of expert feed‑forward layers, allowing trillion‑parameter scale models during inference with only a fraction of the FLOPs activated per token.

### Scaling Laws and Speculative Decoding

[`phases/07-transformers-deep-dive/13-scaling-laws/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/13-scaling-laws/code/main.py) provides empirical formulas predicting the relationship between parameters, training tokens, and final loss. Complementing this, [`phases/07-transformers-deep-dive/16-speculative-decoding/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/16-speculative-decoding/code/main.py) implements **speculative decoding**, where a lightweight "draft" transformer proposes multiple candidate tokens; the full model verifies them in parallel, achieving 3×–6× speedups without altering the output distribution.

## Key Source Files in Phase 7

| Lesson | File Path | Core Concept |
|--------|-----------|--------------|
| Why Transformers | [`phases/07-transformers-deep-dive/01-why-transformers/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/01-why-transformers/docs/en.md) | Motivation for parallel attention |
| Self‑Attention | [`phases/07-transformers-deep-dive/02-self-attention-from-scratch/code/self_attention.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/02-self-attention-from-scratch/code/self_attention.py) | `scaled_dot_product_attention` and `MultiHeadSelfAttention` |
| Positional Encoding | [`phases/07-transformers-deep-dive/04-positional-encoding/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/04-positional-encoding/code/main.py) | Sinusoidal encoding generation |
| Full Transformer | [`phases/07-transformers-deep-dive/05-full-transformer/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/05-full-transformer/code/main.py) | `TransformerBlock` with residuals |
| BERT (MLM) | [`phases/07-transformers-deep-dive/06-bert-masked-language-modeling/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/06-bert-masked-language-modeling/code/main.py) | Bidirectional masked training |
| GPT (Causal) | [`phases/07-transformers-deep-dive/07-gpt-causal-language-modeling/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/07-gpt-causal-language-modeling/code/main.py) | Autoregressive generation loop |
| Encoder‑Decoder | [`phases/07-transformers-deep-dive/08-t5-bart-encoder-decoder/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/08-t5-bart-encoder-decoder/code/main.py) | Cross‑attention for seq‑2‑seq |
| Vision Transformer | [`phases/07-transformers-deep-dive/09-vision-transformers/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/09-vision-transformers/code/main.py) | Patch embedding for images |
| Audio Transformer | [`phases/07-transformers-deep-dive/10-audio-transformers-whisper/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/10-audio-transformers-whisper/code/main.py) | Audio tokenisation |
| KV‑Cache / Flash Attention | [`phases/07-transformers-deep-dive/12-kv-cache-flash-attention/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/12-kv-cache-flash-attention/code/main.py) | `forward_with_cache` optimization |
| Attention Variants | [`phases/07-transformers-deep-dive/15-attention-variants/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/15-attention-variants/code/main.py) | Linear, sparse, and RoPE variants |
| Mixture‑of‑Experts | [`phases/07-transformers-deep-dive/11-mixture-of-experts/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/11-mixture-of-experts/code/main.py) | Gated expert routing |
| Scaling Laws | [`phases/07-transformers-deep-dive/13-scaling-laws/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/13-scaling-laws/code/main.py) | Empirical scaling formulas |
| Speculative Decoding | [`phases/07-transformers-deep-dive/16-speculative-decoding/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/16-speculative-decoding/code/main.py) | Draft‑model acceleration |

## Summary

- **Phase 7** implements transformers from bare NumPy operations, exposing the `softmax(QKᵀ/√dₖ)` mechanism that replaces RNN recurrence.
- **Multi‑head attention** splits `d_model` across heads to capture diverse relational patterns, recombining via a learned output projection.
- **Positional encodings** inject sequence order into the otherwise permutation‑invariant attention operation using sinusoidal or rotary embeddings.
- **Architectural variants** reuse the same core blocks: BERT uses bidirectional encoding, GPT uses causal masking, and T5/BART adds cross‑attention.
- **Production optimizations** like **KV‑caching** and **Flash Attention** reduce inference latency from quadratic to near‑linear complexity per step.
- **Advanced techniques** including **MoE**, **linear attention**, and **speculative decoding** demonstrate how modern systems scale to trillion‑parameter models and real‑time inference.

## Frequently Asked Questions

### What is the fundamental operation that replaces recurrence in Phase 7’s transformer implementation?

The `scaled_dot_product_attention` function computes `softmax(QKᵀ/√dₖ)`, allowing every token to attend to every other token in parallel. This matrix operation eliminates the sequential dependency bottleneck found in RNNs, enabling full GPU utilization during training.

### How does the KV-cache implementation in [`12-kv-cache-flash-attention/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/12-kv-cache-flash-attention/code/main.py) improve inference speed?

The `forward_with_cache` method stores previously computed `K` and `V` matrices and concatenates new token projections only to those cached states. This avoids recomputing attention over the entire prefix for every new token, reducing per‑step complexity from `O(N²)` to `O(N)` where `N` is the sequence length.

### What distinguishes the GPT implementation in Phase 7 from the BERT implementation?

The GPT lesson in [`07-gpt-causal-language-modeling/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/07-gpt-causal-language-modeling/code/main.py) applies a **triangular causal mask** to prevent attention to future tokens, enabling autoregressive generation. The BERT lesson in [`06-bert-masked-language-modeling/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/06-bert-masked-language-modeling/code/main.py) uses no such mask, allowing bidirectional context aggregation for masked token prediction.

### How does speculative decoding achieve 3×–6× speedups without changing the model output?

[`16-speculative-decoding/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/16-speculative-decoding/code/main.py) implements a draft model that rapidly generates candidate token sequences. The full model then verifies these candidates in parallel; accepted tokens are streamed immediately while rejected positions trigger a single correction step. Because the draft and target distributions are mathematically reconciled, the final output remains identical to greedy sampling from the full model alone.