# How to Build Transformers and Large Language Models (LLMs) from Scratch: Phase 7 Implementation

> Implement a complete GPT-style LLM from scratch using NumPy. This guide covers scaled dot-product attention, multi-head transformer blocks, and causal language modeling. Build transformers and LLMs.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: how-to-guide
- Published: 2026-06-18

---

**Phase 7 of the ai-engineering-from-scratch curriculum implements a complete GPT-style LLM using only NumPy, progressing from scaled dot-product attention to multi-head transformer blocks, positional encoding, and causal language modeling with a full training IO pipeline.**

The rohitg00/ai-engineering-from-scratch repository provides a pedagogical, step-by-step construction of modern NLP architectures. By implementing Phase 7, you build a transformer-based large language model from fundamental mathematical operations without PyTorch or TensorFlow, making the underlying mechanics explicit while establishing patterns that can later be swapped for framework implementations.

## The Transformer Construction Pipeline

Phase 7 breaks LLM construction into discrete, testable components. Each step adds a critical architectural element, culminating in a GPT-style causal language model.

### Why Transformers: The Attention Paradigm

The foundational motivation is documented in [`phases/07-transformers-deep-dive/01-why-transformers/docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/01-why-transformers/docs/en.md), which establishes why the attention-first paradigm supersedes recurrent networks. This conceptual groundwork precedes the mathematical implementation, explaining how self-attention enables parallelization and long-range dependencies that RNNs cannot capture efficiently.

### Self-Attention from Scratch

The foundation begins with scaled dot-product attention. In [`phases/07-transformers-deep-dive/02-self-attention-from-scratch/code/self_attention.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/02-self-attention-from-scratch/code/self_attention.py), the `scaled_dot_product_attention` function computes attention weights using only NumPy operations:

```python
import numpy as np

def softmax(x):
    shifted = x - np.max(x, axis=-1, keepdims=True)
    exp_x = np.exp(shifted)
    return exp_x / np.sum(exp_x, axis=-1, keepdims=True)

def scaled_dot_product_attention(Q, K, V):
    dk = Q.shape[-1]
    scores = Q @ K.T / np.sqrt(dk)          # Q·Kᵀ / √dₖ

    weights = softmax(scores)                # attention weights

    output = weights @ V                      # weighted sum of values

    return output, weights

```

This implementation calculates **scaled dot-product attention** using the formula `softmax(Q·Kᵀ / √dₖ)·V`, where `dₖ` represents the key dimension. The scaling factor `√dₖ` prevents dot-product values from growing too large in high dimensions, maintaining stable softmax gradients.

### Multi-Head Attention Mechanism

The curriculum extends single attention to **multi-head attention** by running multiple self-attention operations in parallel. The `MultiHeadSelfAttention` class in [`self_attention.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/self_attention.py) manages independent projection matrices for each head and a final output projection:

```python
class MultiHeadSelfAttention:
    def __init__(self, d_model, n_heads, seed=42):
        assert d_model % n_heads == 0
        self.n_heads = n_heads
        self.dk = d_model // n_heads
        self.dv = d_model // n_heads
        # one SelfAttention per head

        self.heads = [SelfAttention(d_model, self.dk, self.dv, seed=seed+i)
                      for i in range(n_heads)]
        # final linear projection back to d_model

        rng = np.random.default_rng(seed + n_heads)
        self.Wo = rng.normal(0, np.sqrt(2.0/(d_model+d_model)),
                             (n_heads * self.dv, d_model))

    def forward(self, X):
        head_outputs, all_weights = [], []
        for head in self.heads:
            out, w = head.forward(X)
            head_outputs.append(out)
            all_weights.append(w)
        concatenated = np.concatenate(head_outputs, axis=-1)
        return concatenated @ self.Wo, all_weights

```

Each head learns different representation subspaces. The outputs are concatenated and projected back to the model dimension using the `Wo` matrix, which is initialized with a Xavier-style strategy using `np.sqrt(2.0/(d_model+d_model))`.

### Positional Encoding

Transformers lack inherent sequence order awareness. The [`phases/07-transformers-deep-dive/04-positional-encoding/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/04-positional-encoding/code/main.py) file implements **sinusoidal positional encodings** (with optional learned embeddings) to inject token position information:

```python
def positional_encoding(seq_len, d_model):
    positions = np.arange(seq_len)[:, np.newaxis]
    angles = np.arange(0, d_model, 2) * -(np.log(10000.0) / d_model)
    encodings = np.zeros((seq_len, d_model))
    encodings[:, 0::2] = np.sin(positions * angles)
    encodings[:, 1::2] = np.cos(positions * angles)
    return encodings

```

These encodings are added to input embeddings, allowing the model to distinguish between token positions through unique sinusoidal patterns that the attention mechanism can learn to interpret.

### Full Transformer Blocks

The complete transformer block combines attention, feed-forward networks, residual connections, and layer normalization. In [`phases/07-transformers-deep-dive/05-full-transformer/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/05-full-transformer/code/main.py), each block follows the architecture:

```python
def transformer_block(X, attn, ff, ln1, ln2):
    # 1️⃣ Multi‑head self‑attention + residual

    attn_out, _ = attn.forward(X)
    X = ln1(X + attn_out)

    # 2️⃣ Feed‑forward + residual

    ff_out = ff(X)                 # 2‑layer MLP

    X = ln2(X + ff_out)
    return X

```

**Residual connections** (`X + attn_out`) and **layer normalization** (`ln1`, `ln2`) stabilize gradients during deep stacking. The feed-forward network typically consists of two linear transformations with a non-linear activation (GELU or ReLU).

## From Blocks to GPT-Style LLMs

Stacking transformer blocks creates the base architecture, but converting this into a causal language model requires specific training objectives and masking strategies.

### Causal Masking and Autoregressive Training

In [`phases/07-transformers-deep-dive/07-gpt-causal-language-modeling/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/07-transformers-deep-dive/07-gpt-causal-language-modeling/code/main.py), the decoder implements **causal masking** to prevent tokens from attending to future positions. This creates the autoregressive property necessary for next-token prediction.

The attention matrix is masked so token *i* can only attend to tokens ≤ *i*. This differs from the encoder's bidirectional attention, enabling the model to generate text sequentially by predicting the next token given all previous tokens.

### The IO Pipeline

The IO pipeline component handles data tokenization, batching, and the training loop. In [`07-gpt-causal-language-modeling/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/07-gpt-causal-language-modeling/code/main.py) and the reusable helper [`train.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/train.py), the pipeline processes raw text into `(input, target)` pairs:

```python
for epoch in range(num_epochs):
    for batch_inputs, batch_targets in data_loader:
        # forward pass through stacked decoder blocks

        logits = model(batch_inputs)                 # shape: (B, T, vocab)

        loss = cross_entropy(logits.view(-1, vocab), batch_targets.view(-1))
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()

```

The data loader tokenizes plain-text corpora and creates input-target pairs where each input token predicts the next token in the sequence. This **next-token prediction** objective trains the model as a causal language model using cross-entropy loss computed across the vocabulary dimensions.

## Summary

- Phase 7 implements transformers from scratch using only NumPy, starting with `scaled_dot_product_attention` in [`self_attention.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/self_attention.py).
- **Multi-head attention** parallelizes several self-attention heads and projects their concatenated outputs via the `Wo` matrix initialized with `rng.normal(0, np.sqrt(2.0/(d_model+d_model)), ...)`.
- **Positional encodings** inject sequence order information through sinusoidal patterns in [`04-positional-encoding/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/04-positional-encoding/code/main.py).
- Full transformer blocks stack attention, feed-forward networks, and residual connections with layer normalization to enable deep stacking.
- **Causal masking** in [`07-gpt-causal-language-modeling/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/07-gpt-causal-language-modeling/code/main.py) creates autoregressive GPT-style models by preventing future-token attention.
- The IO pipeline tokenizes text and manages next-token prediction training loops using cross-entropy loss between model logits and target indices.

## Frequently Asked Questions

### What is the difference between encoder and decoder transformers in this implementation?

The Phase 7 implementation focuses on decoder-only GPT-style architectures. While [`05-full-transformer/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/05-full-transformer/code/main.py) contains building blocks for both, the causal language model uses **masked self-attention** rather than bidirectional attention. The encoder processes all tokens simultaneously, while the decoder restricts each position to attend only to previous positions and itself, enabling autoregressive text generation.

### Why does the curriculum use NumPy instead of PyTorch for implementing transformers?

Using **NumPy exclusively** forces explicit implementation of every mathematical operation including matrix multiplications, softmax calculations, and weight initializations. This removes framework abstractions and makes the mechanics of attention, backpropagation, and gradient flow visible. According to the repository structure, these same algorithms can later be swapped for PyTorch implementations without changing the high-level logic.

### How does the multi-head attention mechanism improve upon single self-attention?

The `MultiHeadSelfAttention` class creates multiple independent attention heads (`n_heads`), each with separate projection matrices. This allows the model to jointly attend to information from different representation subspaces at different positions. The outputs are concatenated and linearly projected via `Wo`, combining diverse attention patterns into a richer representation than single-head attention could capture.

### What role does the IO pipeline play in training the LLM?

The IO pipeline in [`07-gpt-causal-language-modeling/code/main.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/07-gpt-causal-language-modeling/code/main.py) and [`train.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/train.py) handles **tokenization**, creation of input-target pairs, and batching for the next-token prediction objective. It transforms raw text files into numerical tensors suitable for the NumPy transformer, iterating over minibatches to compute cross-entropy loss between model logits and target token indices during the training loop.