How to Build a Transformer Network Using TinyTorch: A Complete Implementation Guide

You can construct a full GPT-style transformer network in TinyTorch by composing modular components—LayerNorm, Multi-Head Attention, MLP blocks, and residual connections—implemented from first principles in NumPy.

TinyTorch is an educational deep learning framework found in the harvard-edge/cs249r_book repository that demonstrates transformer architecture fundamentals using pure NumPy operations. By leveraging TinyTorch's modular building blocks located in tinytorch/src/13_transformers/13_transformers.py, you can assemble a complete transformer network for language modeling without external deep learning dependencies.

Core Transformer Components in TinyTorch

TinyTorch implements every primitive of a modern transformer in tinytorch/src/13_transformers/13_transformers.py. Understanding these building blocks is essential before assembling the full network.

Layer Normalization

LayerNorm stabilizes training by normalizing each sample across its feature dimension. In TinyTorch, this operation is implemented at lines 48-80 of tinytorch/src/13_transformers/13_transformers.py.

The implementation computes the mean and variance across the last dimension, then applies learnable scale and shift parameters to maintain expressiveness:

from tinytorch.src.13_transformers.13_transformers import LayerNorm
from tinytorch.core.tensor import Tensor
import numpy as np

# Initialize LayerNorm for embedding dimension of 128

ln = LayerNorm(normalized_shape=128)

# Apply to random input

x = Tensor(np.random.randn(2, 10, 128))  # (batch, seq_len, embed_dim)

normalized = ln(x)  # shape preserved: (2, 10, 128)

Multi-Head Attention Mechanism

The attention mechanism is implemented in tinytorch/core/attention.py. This component computes scaled dot-product attention across multiple heads, allowing the model to attend to information from different representation subspaces simultaneously.

The TransformerBlock (lines 145-210 in tinytorch/src/13_transformers/13_transformers.py) integrates this attention mechanism with residual connections and layer normalization using a pre-normalization architecture.

Feed-Forward MLP Blocks

The MLP block processes each token independently after attention, applying a two-layer feed-forward network with a non-linear activation. TinyTorch implements this at lines 114-142 of tinytorch/src/13_transformers/13_transformers.py.

The MLP typically expands the embedding dimension by a factor of four, applies a GELU or ReLU activation, then projects back to the original dimension:

from tinytorch.src.13_transformers.13_transformers import MLP

# Initialize MLP with 4x expansion

mlp = MLP(embed_dim=128, mlp_ratio=4.0, dropout=0.1)

# Process attention output

mlp_output = mlp(attention_output)  # shape: (batch, seq_len, embed_dim)

Assembling the Transformer Block

The TransformerBlock class (lines 145-210 in tinytorch/src/13_transformers/13_transformers.py) represents a single layer of the transformer, implementing the pre-normalization architecture. This block wires together LayerNorm, Multi-Head Attention, and the MLP with residual connections.

The forward pass follows this computation graph:

1. Apply LayerNorm to input
2. Pass through Multi-Head Attention
3. Add residual connection
4. Apply LayerNorm to result
5. Pass through MLP
6. Add residual connection

from tinytorch.src.13_transformers.13_transformers import TransformerBlock
from tinytorch.core.tensor import Tensor
import numpy as np

# Initialize a single transformer block

block = TransformerBlock(embed_dim=128, num_heads=4, mlp_ratio=4.0, dropout=0.1)

# Forward pass

x = Tensor(np.random.randn(2, 10, 128))  # (batch, seq_len, embed_dim)

output = block(x, mask=None)  # shape: (2, 10, 128)

Building the Complete GPT Model

The GPT class (lines 1247-1310 in tinytorch/src/13_transformers/13_transformers.py) stacks multiple TransformerBlock layers to create a full GPT-style language model. This implementation includes token embeddings, positional embeddings, and a language modeling head.

Model Architecture

The GPT model consists of:

• Embedding Layer: Maps integer token IDs to vectors of size embed_dim and adds learned positional embeddings (implemented in tinytorch/core/embeddings.py)
• Causal Mask: A triangular matrix of -inf values that prevents each position from attending to future tokens (_create_causal_mask, lines 52-57)
• Stack of TransformerBlocks: num_layers blocks processing the sequence
• Final LayerNorm: ln_f normalizes the output of the last block
• Language Model Head: Projects hidden states back to vocabulary dimensions for next-token prediction

The forward pass (GPT.forward) follows the canonical transformer pipeline:

tokens → embed+pos → [block₁ … block_N] → ln_f → lm_head → logits

from tinytorch.src.13_transformers.13_transformers import GPT
from tinytorch.core.tensor import Tensor
import numpy as np

# Build a small GPT model

vocab_size = 5000
embed_dim = 128
num_layers = 4
num_heads = 4

model = GPT(vocab_size, embed_dim, num_layers, num_heads)

# Forward pass on dummy batch

batch_size, seq_len = 2, 10
tokens = Tensor(np.random.randint(0, vocab_size, (batch_size, seq_len)))
logits = model(tokens)  # shape: (batch, seq_len, vocab_size)

print(f"Output logits shape: {logits.shape}")

Autoregressive Text Generation

The GPT class implements autoregressive generation through the generate method (lines 14-31) and _sample_next_token helper (lines 78-88) in tinytorch/src/13_transformers/13_transformers.py. This process repeatedly samples new tokens and appends them to the sequence.

The generation algorithm:

1. Run forward pass on current token sequence
2. Extract logits of the last position
3. Apply temperature-scaled softmax (_sample_next_token)
4. Sample a new token from the distribution
5. Append token and repeat

# Generate text from a prompt

prompt = Tensor(np.array([[1, 23, 456]]))  # Short seed sequence

generated = model.generate(
    prompt, 
    max_new_tokens=5, 
    temperature=0.9
)
print("Generated token IDs:", generated.data)

Summary

Building a transformer network using TinyTorch involves composing several fundamental components implemented in the harvard-edge/cs249r_book repository:

• LayerNorm stabilizes training by normalizing features across the embedding dimension (lines 48-80 in 13_transformers.py)
• Multi-Head Attention enables the model to capture dependencies across the sequence (via tinytorch/core/attention.py)
• MLP blocks provide non-linear processing of individual token representations (lines 114-142)
• TransformerBlock combines these elements with residual connections and pre-normalization (lines 145-210)
• GPT stacks these blocks to create a complete autoregressive language model with embeddings and a language modeling head (lines 1247-1310)

All components rely on TinyTorch's NumPy-based tensor operations and manual autograd implementation, making the architecture transparent and extensible for educational purposes.

Frequently Asked Questions

What is TinyTorch and how does it differ from PyTorch?

TinyTorch is an educational deep learning framework implemented in the harvard-edge/cs249r_book repository that demonstrates transformer architecture fundamentals using pure NumPy operations. Unlike PyTorch, which uses optimized C++ backends and CUDA kernels, TinyTorch implements all tensor operations and automatic differentiation manually in Python, making it ideal for understanding the mathematical foundations of deep learning.

How does the causal mask prevent future token leakage in TinyTorch transformers?

The causal mask is implemented as a triangular matrix of -inf values in the _create_causal_mask method at lines 52-57 of tinytorch/src/13_transformers/13_transformers.py. When added to the attention scores before the softmax operation, this mask sets the attention weight to zero for any position attempting to attend to future tokens, ensuring the autoregressive property where each token can only depend on previous tokens in the sequence.

Can I train the TinyTorch transformer on custom datasets?

Yes, the TinyTorch transformer supports training on custom datasets through its manual autograd implementation in tinytorch/core/autograd.py. You can compute cross-entropy loss between the model's output logits and target token IDs, then call .backward() on the loss tensor to compute gradients through the entire GPT architecture. The model uses standard SGD or Adam updates on the parameters stored in the Tensor objects.

What hardware requirements are needed to run TinyTorch transformers?

TinyTorch transformers run entirely on CPU using NumPy arrays, requiring no GPU or specialized hardware acceleration. The implementation in harvard-edge/cs249r_book is designed for educational purposes with small-scale models (embedding dimensions of 128-512 and 4-12 layers), making it accessible to run on standard laptops with minimal RAM requirements compared to production frameworks like PyTorch or TensorFlow.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}