How to Build Transformers and Large Language Models (LLMs) from Scratch: Phase 7 Implementation

Phase 7 of the ai-engineering-from-scratch curriculum implements a complete GPT-style LLM using only NumPy, progressing from scaled dot-product attention to multi-head transformer blocks, positional encoding, and causal language modeling with a full training IO pipeline.

The rohitg00/ai-engineering-from-scratch repository provides a pedagogical, step-by-step construction of modern NLP architectures. By implementing Phase 7, you build a transformer-based large language model from fundamental mathematical operations without PyTorch or TensorFlow, making the underlying mechanics explicit while establishing patterns that can later be swapped for framework implementations.

The Transformer Construction Pipeline

Phase 7 breaks LLM construction into discrete, testable components. Each step adds a critical architectural element, culminating in a GPT-style causal language model.

Why Transformers: The Attention Paradigm

The foundational motivation is documented in phases/07-transformers-deep-dive/01-why-transformers/docs/en.md, which establishes why the attention-first paradigm supersedes recurrent networks. This conceptual groundwork precedes the mathematical implementation, explaining how self-attention enables parallelization and long-range dependencies that RNNs cannot capture efficiently.

Self-Attention from Scratch

The foundation begins with scaled dot-product attention. In phases/07-transformers-deep-dive/02-self-attention-from-scratch/code/self_attention.py, the scaled_dot_product_attention function computes attention weights using only NumPy operations:

import numpy as np

def softmax(x):
    shifted = x - np.max(x, axis=-1, keepdims=True)
    exp_x = np.exp(shifted)
    return exp_x / np.sum(exp_x, axis=-1, keepdims=True)

def scaled_dot_product_attention(Q, K, V):
    dk = Q.shape[-1]
    scores = Q @ K.T / np.sqrt(dk)          # Q·Kᵀ / √dₖ

    weights = softmax(scores)                # attention weights

    output = weights @ V                      # weighted sum of values

    return output, weights

This implementation calculates scaled dot-product attention using the formula softmax(Q·Kᵀ / √dₖ)·V, where dₖ represents the key dimension. The scaling factor √dₖ prevents dot-product values from growing too large in high dimensions, maintaining stable softmax gradients.

Multi-Head Attention Mechanism

The curriculum extends single attention to multi-head attention by running multiple self-attention operations in parallel. The MultiHeadSelfAttention class in self_attention.py manages independent projection matrices for each head and a final output projection:

class MultiHeadSelfAttention:
    def __init__(self, d_model, n_heads, seed=42):
        assert d_model % n_heads == 0
        self.n_heads = n_heads
        self.dk = d_model // n_heads
        self.dv = d_model // n_heads
        # one SelfAttention per head

        self.heads = [SelfAttention(d_model, self.dk, self.dv, seed=seed+i)
                      for i in range(n_heads)]
        # final linear projection back to d_model

        rng = np.random.default_rng(seed + n_heads)
        self.Wo = rng.normal(0, np.sqrt(2.0/(d_model+d_model)),
                             (n_heads * self.dv, d_model))

    def forward(self, X):
        head_outputs, all_weights = [], []
        for head in self.heads:
            out, w = head.forward(X)
            head_outputs.append(out)
            all_weights.append(w)
        concatenated = np.concatenate(head_outputs, axis=-1)
        return concatenated @ self.Wo, all_weights

Each head learns different representation subspaces. The outputs are concatenated and projected back to the model dimension using the Wo matrix, which is initialized with a Xavier-style strategy using np.sqrt(2.0/(d_model+d_model)).

Positional Encoding

Transformers lack inherent sequence order awareness. The phases/07-transformers-deep-dive/04-positional-encoding/code/main.py file implements sinusoidal positional encodings (with optional learned embeddings) to inject token position information:

def positional_encoding(seq_len, d_model):
    positions = np.arange(seq_len)[:, np.newaxis]
    angles = np.arange(0, d_model, 2) * -(np.log(10000.0) / d_model)
    encodings = np.zeros((seq_len, d_model))
    encodings[:, 0::2] = np.sin(positions * angles)
    encodings[:, 1::2] = np.cos(positions * angles)
    return encodings

These encodings are added to input embeddings, allowing the model to distinguish between token positions through unique sinusoidal patterns that the attention mechanism can learn to interpret.

Full Transformer Blocks

The complete transformer block combines attention, feed-forward networks, residual connections, and layer normalization. In phases/07-transformers-deep-dive/05-full-transformer/code/main.py, each block follows the architecture:

def transformer_block(X, attn, ff, ln1, ln2):
    # 1️⃣ Multi‑head self‑attention + residual

    attn_out, _ = attn.forward(X)
    X = ln1(X + attn_out)

    # 2️⃣ Feed‑forward + residual

    ff_out = ff(X)                 # 2‑layer MLP

    X = ln2(X + ff_out)
    return X

Residual connections (X + attn_out) and layer normalization (ln1, ln2) stabilize gradients during deep stacking. The feed-forward network typically consists of two linear transformations with a non-linear activation (GELU or ReLU).

From Blocks to GPT-Style LLMs

Stacking transformer blocks creates the base architecture, but converting this into a causal language model requires specific training objectives and masking strategies.

Causal Masking and Autoregressive Training

In phases/07-transformers-deep-dive/07-gpt-causal-language-modeling/code/main.py, the decoder implements causal masking to prevent tokens from attending to future positions. This creates the autoregressive property necessary for next-token prediction.

The attention matrix is masked so token i can only attend to tokens ≤ i. This differs from the encoder's bidirectional attention, enabling the model to generate text sequentially by predicting the next token given all previous tokens.

The IO Pipeline

The IO pipeline component handles data tokenization, batching, and the training loop. In 07-gpt-causal-language-modeling/code/main.py and the reusable helper train.py, the pipeline processes raw text into (input, target) pairs:

for epoch in range(num_epochs):
    for batch_inputs, batch_targets in data_loader:
        # forward pass through stacked decoder blocks

        logits = model(batch_inputs)                 # shape: (B, T, vocab)

        loss = cross_entropy(logits.view(-1, vocab), batch_targets.view(-1))
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()

The data loader tokenizes plain-text corpora and creates input-target pairs where each input token predicts the next token in the sequence. This next-token prediction objective trains the model as a causal language model using cross-entropy loss computed across the vocabulary dimensions.

Summary

• Phase 7 implements transformers from scratch using only NumPy, starting with scaled_dot_product_attention in self_attention.py.
• Multi-head attention parallelizes several self-attention heads and projects their concatenated outputs via the Wo matrix initialized with rng.normal(0, np.sqrt(2.0/(d_model+d_model)), ...).
• Positional encodings inject sequence order information through sinusoidal patterns in 04-positional-encoding/code/main.py.
• Full transformer blocks stack attention, feed-forward networks, and residual connections with layer normalization to enable deep stacking.
• Causal masking in 07-gpt-causal-language-modeling/code/main.py creates autoregressive GPT-style models by preventing future-token attention.
• The IO pipeline tokenizes text and manages next-token prediction training loops using cross-entropy loss between model logits and target indices.

Frequently Asked Questions

What is the difference between encoder and decoder transformers in this implementation?

The Phase 7 implementation focuses on decoder-only GPT-style architectures. While 05-full-transformer/code/main.py contains building blocks for both, the causal language model uses masked self-attention rather than bidirectional attention. The encoder processes all tokens simultaneously, while the decoder restricts each position to attend only to previous positions and itself, enabling autoregressive text generation.

Why does the curriculum use NumPy instead of PyTorch for implementing transformers?

Using NumPy exclusively forces explicit implementation of every mathematical operation including matrix multiplications, softmax calculations, and weight initializations. This removes framework abstractions and makes the mechanics of attention, backpropagation, and gradient flow visible. According to the repository structure, these same algorithms can later be swapped for PyTorch implementations without changing the high-level logic.

How does the multi-head attention mechanism improve upon single self-attention?

The MultiHeadSelfAttention class creates multiple independent attention heads (n_heads), each with separate projection matrices. This allows the model to jointly attend to information from different representation subspaces at different positions. The outputs are concatenated and linearly projected via Wo, combining diverse attention patterns into a richer representation than single-head attention could capture.

What role does the IO pipeline play in training the LLM?

The IO pipeline in 07-gpt-causal-language-modeling/code/main.py and train.py handles tokenization, creation of input-target pairs, and batching for the next-token prediction objective. It transforms raw text files into numerical tensors suitable for the NumPy transformer, iterating over minibatches to compute cross-entropy loss between model logits and target token indices during the training loop.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}