Differences Between BERT, GPT, and T5 Transformer Architectures: A Complete Technical Guide

BERT is an encoder-only model with bidirectional attention optimized for understanding tasks, GPT is a decoder-only model with causal masking designed for autoregressive generation, and T5 is an encoder-decoder architecture that combines both approaches for sequence-to-sequence transformation.

The transformer architecture revolutionized natural language processing, yet the differences between BERT, GPT, and T5 transformer architectures remain a source of confusion for practitioners. According to the rohitg00/ai-engineering-from-scratch curriculum, these models share identical core sub-layers—multi-head self-attention and feed-forward networks—but diverge critically in their masking strategies, stack compositions, and pre-training objectives. This article examines the actual implementation files to reveal how these architectures differ at the code level.

Core Architectural Patterns

BERT (Encoder-Only Bidirectional)

BERT (Bidirectional Encoder Representations from Transformers) consists exclusively of encoder blocks that allow every token to attend to all other tokens in both directions. In phases/07-transformers-deep-dive/06-bert-masked-language-modeling/code/main.py, the implementation uses full self-attention with no masking, enabling the model to capture bidirectional context simultaneously.

The architecture follows this pattern: LayerNorm → Self-Attention → Add → LayerNorm → Feed-Forward → Add. Notably absent is any cross-attention mechanism, as BERT processes input sequences in parallel without generating new tokens autoregressively.

GPT (Decoder-Only Causal)

GPT (Generative Pre-trained Transformer) employs decoder blocks with a causal (triangular) mask that restricts each position to attend only to previous positions and itself. As implemented in phases/07-transformers-deep-dive/07-gpt-causal-language-modeling/code/main.py, this masking strategy enables autoregressive text generation.

The decoder block structure includes: LayerNorm → Self-Attention (causal) → Add → LayerNorm → Cross-Attention → Add → LayerNorm → Feed-Forward → Add. While the cross-attention layer exists in the generic decoder definition, standard GPT implementations omit it since there is no encoder output to attend to.

T5 (Encoder-Decoder "Transformer-Stack")

T5 (Text-to-Text Transfer Transformer) implements the full encoder-decoder stack (sometimes called "Transformer-Stack" or TS), combining BERT's bidirectional encoder with GPT's causal decoder. The encoder processes input with full self-attention, while the decoder generates output using both causal self-attention and cross-attention to the encoder's final hidden states.

This architecture appears in phases/07-transformers-deep-dive/08-t5-bart-encoder-decoder/code/main.py, where the encoder and decoder blocks are explicitly linked to perform sequence-to-sequence mapping.

Technical Implementation Differences

Attention Masking Strategies

The primary discriminator between these architectures is the attention mask implementation:

• BERT: Uses no mask (full visibility). Tokens attend to all positions left and right simultaneously.
• GPT: Implements a causal_mask using np.triu(np.full((seq_len, seq_len), fill_value=-np.inf), k=1) to prevent attending to future positions.
• T5: Hybrid approach—the encoder applies full self-attention while the decoder applies both causal masking for self-attention and full attention for cross-attention to encoder outputs.

Pre-training Objectives

Each architecture employs a distinct training objective that dictates its optimal use case:

BERT: Masked Language Modeling (MLM)

The create_mlm_batch function in phases/07-transformers-deep-dive/06-bert-masked-language-modeling/code/main.py implements the 80/10/10 rule:

# File: phases/07-transformers-deep-dive/06-bert-masked-language-modeling/code/main.py

def create_mlm_batch(tokens, vocab_size, mask_prob=0.15, rng=None):
    """Apply BERT masking. Returns (input_ids, labels)."""
    rng = rng or random.Random()
    input_ids = list(tokens)
    labels = [IGNORE_INDEX] * len(tokens)

    for i, t in enumerate(tokens):
        # Skip special tokens such as [CLS], [SEP], [MASK]

        if t in SPECIAL_IDS:
            continue
        # Decide whether to mask this token

        if rng.random() < mask_prob:
            labels[i] = t                      # store original token as label

            r = rng.random()
            if r < 0.8:
                input_ids[i] = MASK_ID        # 80% replace with [MASK]

            elif r < 0.9:
                # 10% replace with random token (not a special token)

                rand_id = t
                while rand_id in SPECIAL_IDS or rand_id == t:
                    rand_id = rng.randrange(vocab_size)
                input_ids[i] = rand_id
            # 10% keep the original token (no change)

    return input_ids, labels

GPT: Causal Language Modeling (CLM)

GPT maximizes the probability of the next token given previous tokens, implemented through triangular masking and autoregressive generation:

# File: phases/07-transformers-deep-dive/07-gpt-causal-language-modeling/code/main.py

def causal_mask(seq_len):
    """Triangular mask – each position can attend only to earlier positions."""
    return np.triu(np.full((seq_len, seq_len), fill_value=-np.inf), k=1)

def generate(model, start_ids, max_new_tokens=20):
    """Simple autoregressive loop: generate tokens one-by-one."""
    ids = list(start_ids)
    for _ in range(max_new_tokens):
        # Forward pass through decoder-only transformer

        logits = model.forward(np.array(ids))          # shape: (len(ids), vocab)

        next_id = logits[-1].argmax()                  # greedy decode

        ids.append(int(next_id))
        if next_id == EOS_ID:
            break
    return ids

T5: Sequence-to-Sequence (Seq2Seq)

T5 combines encoder pre-training (similar to BERT) with decoder pre-training (similar to GPT), adding a seq2seq loss during fine-tuning:

# File: phases/07-transformers-deep-dive/08-t5-bart-encoder-decoder/code/main.py

def encoder_forward(x):
    """Full encoder stack – identical to BERT's block (bidirectional)."""
    for block in encoder_blocks:
        x = block(x)          # each block = self-attention + feed-forward

    return x

def decoder_forward(enc_output, y):
    """Decoder stack – causal self-attention + cross-attention to encoder."""
    for block in decoder_blocks:
        y = block(y, enc_output)  # block handles both masks internally

    return y

def seq2seq(model, src_ids, tgt_start_ids, max_len=30):
    """Encode → decode loop for a seq2seq task (e.g., translation)."""
    enc = encoder_forward(src_ids)
    out = list(tgt_start_ids)
    for _ in range(max_len):
        logits = decoder_forward(enc, out)
        next_id = logits[-1].argmax()
        out.append(int(next_id))
        if next_id == EOS_ID:
            break
    return out

Performance and Efficiency Trade-offs

• BERT: Processes all tokens in parallel, offering ~5-10× speed-up per token compared to decoder models of equivalent depth. Ideal for classification and embedding generation.
• GPT: Requires sequential token generation, making it slower for long outputs but optimal for open-ended generative tasks like chat and code synthesis.
• T5: Incurs slightly higher computational cost than BERT due to maintaining both encoder and decoder stacks, but provides superior flexibility for tasks requiring input-to-output transformation.

Summary

• BERT is encoder-only with bidirectional attention and MLM pre-training, excelling at understanding tasks like classification and named entity recognition.
• GPT is decoder-only with causal masking and CLM pre-training, optimized for autoregressive generation including text completion and dialogue.
• T5 combines encoder-decoder stacks with cross-attention mechanisms, enabling sequence-to-sequence tasks such as translation and summarization.
• The critical differences lie in masking strategies (full vs. causal vs. hybrid), presence of cross-attention, and pre-training objectives (MLM vs. CLM vs. span-to-span).
• Implementation references in rohitg00/ai-engineering-from-scratch demonstrate these differences through just a few lines of code in phases/07-transformers-deep-dive/06-bert-masked-language-modeling/code/main.py, phases/07-transformers-deep-dive/07-gpt-causal-language-modeling/code/main.py, and phases/07-transformers-deep-dive/08-t5-bart-encoder-decoder/code/main.py.

Frequently Asked Questions

Can BERT be used for text generation like GPT?

No. BERT lacks the causal masking mechanism required for autoregressive generation. While the encoder can process input bidirectionally, it has no decoder stack to generate new tokens sequentially. Attempting to use BERT for generation would result in predictions that depend on future tokens, violating the left-to-right generation constraint required for coherent text output.

Why does T5 use both encoder and decoder instead of just a large decoder like GPT?

T5's encoder-decoder architecture separates the task of understanding the input (handled by the bidirectional encoder) from generating the output (handled by the causal decoder). This separation allows the encoder to build rich contextual representations of the source text while the decoder focuses on producing the target sequence. For translation and summarization tasks, this proves more effective than forcing a single decoder to simultaneously encode source context and generate output, as the encoder can attend to all source tokens in parallel.

What is the 80/10/10 rule in BERT's masking strategy?

The 80/10/10 rule refers to how BERT creates masked language model training data: when a token is selected for masking (15% probability), 80% of the time it is replaced with the [MASK] token, 10% of the time it is replaced with a random token from the vocabulary, and 10% of the time it is left unchanged. This approach prevents the model from simply learning that [MASK] always indicates the target token while forcing it to maintain contextual representations for all positions.

Which architecture offers the best performance for document classification tasks?

BERT typically delivers superior performance for document classification because its bidirectional attention allows the model to consider the full context of the document simultaneously when making predictions. The encoder-only architecture produces a fixed-size contextualized embedding that can be fed directly into a classification head. While T5 can perform classification by treating the label as a generated sequence, BERT's parallel processing and classification-specific fine-tuning generally result in faster inference and slightly better accuracy on understanding-oriented benchmarks.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}