How Transformers Are Built from the Ground Up in Phase 7: A Complete Technical Deep Dive
Phase 7 of the AI Engineering from Scratch curriculum reconstructs every transformer component from first principles—beginning with naive self‑attention and culminating in speculative decoding—using pure NumPy implementations across 15 progressive lessons.
The rohitg00/ai-engineering-from-scratch repository dedicates Phase 7 to a rigorous, ground‑up construction of modern transformer architectures. Unlike tutorials that rely on pre‑built layers, this phase implements scaled‑dot‑product attention, multi‑head projections, and positional encodings from scratch, revealing the mathematical machinery behind models like BERT, GPT, and Whisper.
Foundational Attention Mechanisms
Why Transformers Replace Recurrence
The first lesson in phases/07-transformers-deep-dive/01-why-transformers/docs/en.md establishes the computational bottleneck of RNNs: their sequential hidden‑state updates prevent parallelization across sequence positions. The solution is the parallel‑attention paradigm, which computes interactions between all tokens simultaneously using matrix multiplication.
Self‑Attention from Scratch
In phases/07-transformers-deep-dive/02-self-attention-from-scratch/code/self_attention.py, the core operation is implemented as a pure NumPy function:
def scaled_dot_product_attention(Q, K, V):
dk = Q.shape[-1]
scores = Q @ K.T / np.sqrt(dk)
weights = softmax(scores)
return weights @ V, weights
This routine computes softmax(QKᵀ/√dₖ), the only operation that couples all token positions directly. By scaling with √dₖ, the implementation prevents dot‑product magnitudes from pushing the softmax into saturation zones, a critical detail for training stability.
Multi‑Head Attention
The same file introduces the MultiHeadSelfAttention class, which partitions the model dimension d_model into n_heads independent sub‑spaces. Each head learns distinct relational patterns; their outputs are concatenated and projected back via self.Wo:
class MultiHeadSelfAttention:
def __init__(self, d_model, n_heads, seed=42):
self.heads = [...]
self.Wo = rng.normal(0, scale, (n_heads * d_model//n_heads, d_model))
def forward(self, X):
head_outputs = [head.forward(X)[0] for head in self.heads]
concatenated = np.concatenate(head_outputs, axis=-1)
return concatenated @ self.Wo
Encoding Sequence and Depth
Sinusoidal Positional Encoding
Since attention is permutation‑invariant, phases/07-transformers-deep-dive/04-positional-encoding/code/main.py injects absolute position information through sinusoidal signals:
def positional_encoding(seq_len, d_model):
pos = np.arange(seq_len)[:, None]
i = np.arange(d_model)[None, :]
angle_rates = 1 / np.power(10000, (2 * (i//2)) / d_model)
angle_rads = pos * angle_rates
pe = np.empty((seq_len, d_model))
pe[:, 0::2] = np.sin(angle_rads[:, 0::2])
pe[:, 1::2] = np.cos(angle_rads[:, 1::2])
return pe
This uses wavelengths that form a geometric progression, allowing the model to learn relative positions via linear transformations of the encoding vectors.
The Full Transformer Block
phases/07-transformers-deep-dive/05-full-transformer/code/main.py assembles the complete encoder layer. The TransformerBlock class combines multi‑head attention, residual connections, layer normalization, and a position‑wise feed‑forward network:
class TransformerBlock:
def __init__(self, d_model, n_heads):
self.mha = MultiHeadSelfAttention(d_model, n_heads)
self.ffn = FeedForward(d_model)
self.ln1 = LayerNorm(d_model)
self.ln2 = LayerNorm(d_model)
def forward(self, X):
attn_out, _ = self.mha(self.ln1(X))
X = X + attn_out # residual
ffn_out = self.ffn(self.ln2(X))
return X + ffn_out # residual
Stacking L such blocks yields the full encoder stack used in BERT and the encoder‑decoder variants.
Architectural Variants: From BERT to GPT
BERT and Masked Language Modeling
phases/07-transformers-deep-dive/06-bert-masked-language-modeling/code/main.py demonstrates how to convert the encoder into a bidirectional model. Random tokens are masked during training, and the model learns to reconstruct them using context from both directions—a process that requires no causal masking.
GPT: Causal Language Modeling
Conversely, phases/07-transformers-deep-dive/07-gpt-causal-language-modeling/code/main.py implements autoregressive generation by applying a triangular mask to the attention scores. This ensures each position attends only to previous tokens, enabling next‑token prediction:
# Conceptual example from the GPT lesson
mask = np.triu(np.ones((seq_len, seq_len)), k=1) * -np.inf
scores = (Q @ K.T / np.sqrt(dk)) + mask
Encoder‑Decoder Cross‑Attention
phases/07-transformers-deep-dive/08-t5-bart-encoder-decoder/code/main.py introduces cross‑attention, where the decoder’s query matrices attend to the encoder’s final hidden states. This mechanism powers sequence‑to‑sequence tasks like translation, combining the bidirectional encoding of the source with the left‑to‑right generation of the target.
Cross‑Modal and Production Implementations
Vision Transformers (ViT)
phases/07-transformers-deep-dive/09-vision-transformers/code/main.py adapts the same blocks to image patches. An image is split into fixed‑size patches, linearly embedded, and processed by the standard transformer stack, proving the architecture is modality‑agnostic.
Audio Transformers (Whisper)
Similarly, phases/07-transformers-deep-dive/10-audio-transformers-whisper/code/main.py applies the attention stack to audio spectrograms after tokenisation, handling speech recognition with the same mathematical primitives used for text.
KV‑Cache and Flash Attention
For inference optimization, phases/07-transformers-deep-dive/12-kv-cache-flash-attention/code/main.py implements KV‑caching to eliminate redundant computation:
def forward_with_cache(self, X, cache=None):
Q = X @ self.Wq
K = X @ self.Wk if cache is None else np.concatenate([cache['K'], X @ self.Wk], axis=0)
V = X @ self.Wv if cache is None else np.concatenate([cache['V'], X @ self.Wv], axis=0)
out, _ = scaled_dot_product_attention(Q, K, V)
return out, {'K': K, 'V': V}
By storing previous K and V matrices, the decoder reuses them for each new token, reducing complexity from O(N²) per step to O(N). The lesson also covers Flash Attention, which reorders memory access patterns to keep the quadratic term compute‑bound rather than memory‑bound on GPUs.
Advanced Scaling and Inference Optimizations
Linear, Sparse, and Rotary Attention Variants
phases/07-transformers-deep-dive/15-attention-variants/code/main.py addresses the O(N²) memory bottleneck through several strategies:
- Linear attention: Computes
Q(KᵀV)instead of(QKᵀ)V, reducing memory to linear complexity. - Rotary Position Embeddings (RoPE): Encodes absolute position directly into the dot‑product via rotation matrices, eliminating the need for additive positional encodings.
Mixture‑of‑Experts (MoE)
phases/07-transformers-deep-dive/11-mixture-of-experts/code/main.py scales model capacity without linearly increasing compute. A gating network routes each token to a sparse subset of expert feed‑forward layers, allowing trillion‑parameter scale models during inference with only a fraction of the FLOPs activated per token.
Scaling Laws and Speculative Decoding
phases/07-transformers-deep-dive/13-scaling-laws/code/main.py provides empirical formulas predicting the relationship between parameters, training tokens, and final loss. Complementing this, phases/07-transformers-deep-dive/16-speculative-decoding/code/main.py implements speculative decoding, where a lightweight "draft" transformer proposes multiple candidate tokens; the full model verifies them in parallel, achieving 3×–6× speedups without altering the output distribution.
Key Source Files in Phase 7
Summary
- Phase 7 implements transformers from bare NumPy operations, exposing the
softmax(QKᵀ/√dₖ)mechanism that replaces RNN recurrence. - Multi‑head attention splits
d_modelacross heads to capture diverse relational patterns, recombining via a learned output projection. - Positional encodings inject sequence order into the otherwise permutation‑invariant attention operation using sinusoidal or rotary embeddings.
- Architectural variants reuse the same core blocks: BERT uses bidirectional encoding, GPT uses causal masking, and T5/BART adds cross‑attention.
- Production optimizations like KV‑caching and Flash Attention reduce inference latency from quadratic to near‑linear complexity per step.
- Advanced techniques including MoE, linear attention, and speculative decoding demonstrate how modern systems scale to trillion‑parameter models and real‑time inference.
Frequently Asked Questions
What is the fundamental operation that replaces recurrence in Phase 7’s transformer implementation?
The scaled_dot_product_attention function computes softmax(QKᵀ/√dₖ), allowing every token to attend to every other token in parallel. This matrix operation eliminates the sequential dependency bottleneck found in RNNs, enabling full GPU utilization during training.
How does the KV-cache implementation in 12-kv-cache-flash-attention/code/main.py improve inference speed?
The forward_with_cache method stores previously computed K and V matrices and concatenates new token projections only to those cached states. This avoids recomputing attention over the entire prefix for every new token, reducing per‑step complexity from O(N²) to O(N) where N is the sequence length.
What distinguishes the GPT implementation in Phase 7 from the BERT implementation?
The GPT lesson in 07-gpt-causal-language-modeling/code/main.py applies a triangular causal mask to prevent attention to future tokens, enabling autoregressive generation. The BERT lesson in 06-bert-masked-language-modeling/code/main.py uses no such mask, allowing bidirectional context aggregation for masked token prediction.
How does speculative decoding achieve 3×–6× speedups without changing the model output?
16-speculative-decoding/code/main.py implements a draft model that rapidly generates candidate token sequences. The full model then verifies these candidates in parallel; accepted tokens are streamed immediately while rejected positions trigger a single correction step. Because the draft and target distributions are mathematically reconciled, the final output remains identical to greedy sampling from the full model alone.
Have a question about this repo?
These articles cover the highlights, but your codebase questions are specific. Give your agent direct access to the source. Share this with your agent to get started:
curl -s "https://instagit.com/install.md" 
Maintain an open-source project? Get it listed too →