# Transformer Architecture and BERT Model Internals Explained: A Deep Dive into Self-Attention

> Explore the Transformer architecture and BERT model internals. Understand self-attention mechanisms and how BERT achieves state-of-the-art contextual representations for NLP.

- Repository: [Microsoft/AI-For-Beginners](https://github.com/microsoft/AI-For-Beginners)
- Tags: deep-dive
- Published: 2026-08-02

---

**Transformer architecture replaces sequential recurrence with parallel self-attention mechanisms, while BERT leverages deep bidirectional Transformer encoders pre-trained on Masked Language Modeling and Next Sentence Prediction to create state-of-the-art contextual representations.**

This comprehensive guide examines the inner workings of Transformer architecture and BERT model internals based on the educational materials in the `microsoft/AI-For-Beginners` repository. Understanding these components is essential for modern natural language processing and generative AI applications.

## How Transformer Architecture Works

The Transformer architecture overcomes the limitations of recurrent neural networks by eliminating sequential processing. Instead of processing tokens one by one, Transformers use self-attention to allow every token to attend to every other token simultaneously, enabling efficient parallel computation and superior long-range dependency modeling.

### Self-Attention: The Core Mechanism

Self-attention computes a weighted sum of all token embeddings for each position using three learned matrices: **Query (Q)**, **Key (K)**, and **Value (V)**. The attention weights determine how much each token contributes to the representation of every other token. As implemented in `lessons/5-NLP/18-Transformers/TransformersTF.ipynb` around line 13, this mechanism calculates scaled dot-product attention to capture contextual relationships regardless of positional distance.

### Multi-Head Attention and Parallel Processing

**Multi-head attention** runs several self-attention operations in parallel to capture different types of relationships across various representation subspaces. The outputs from these parallel heads are concatenated and linearly transformed. According to the notebook reference at line 34, this allows the model to jointly attend to information from different representation positions at different granularities.

### Position-wise Feed-Forward Networks

After attention, each token passes through a **position-wise feed-forward network** consisting of two linear transformations with a non-linear activation (typically GELU or ReLU). As shown at line 42 in the Transformer notebook, this applies the same feed-forward network independently to each position, adding depth and non-linearity after the attention mechanism.

### Positional Encoding and Layer Normalization

Since self-attention is order-agnostic, **positional encodings** inject sequence order information using sinusoidal functions or learned embeddings. The architecture also employs **layer normalization** and **residual connections** around each sub-layer to stabilize training and enable deep stacks. The `AI-For-Beginners` materials demonstrate this at line 48 (residual connections and normalization) and line 55 (positional encodings).

## BERT Model Internals: Bidirectional Encoder Representations

BERT (Bidirectional Encoder Representations from Transformers) utilizes only the **encoder** stack of the Transformer architecture. Unlike left-to-right language models, BERT processes context bidirectionally, attending to both left and right context simultaneously for every token.

### Architecture Specifications: BERT-Base vs BERT-Large

The `AI-For-Beginners` repository details two standard configurations:

- **BERT-Base**: 12 Transformer layers, 768 hidden dimensions, 12 attention heads, and approximately 110 million parameters
- **BERT-Large**: 24 Transformer layers, 1024 hidden dimensions, 16 attention heads, and approximately 340 million parameters

### Pre-training Objectives: MLM and NSP

BERT employs two unsupervised pre-training tasks:

1. **Masked Language Modeling (MLM)**: Randomly masks 15% of input tokens and trains the model to predict the original vocabulary IDs, forcing the network to infer context from both directions
2. **Next Sentence Prediction (NSP)**: Trains the model to predict whether two sentences appear sequentially in the original text, encouraging understanding of sentence-level coherence and discourse relationships

### The Forward Pass: From Tokenization to Output

The architectural flow of a BERT forward pass consists of four stages:

1. **Input Tokenization**: Text is split into WordPiece tokens and mapped to integer IDs
2. **Embedding Layer**: Token IDs are summed with positional embeddings and segment (sentence) embeddings
3. **Stack of Encoder Blocks**: Each block applies multi-head self-attention → add & norm → feed-forward → add & norm
4. **Output Generation**: The model returns three key tensors:
   - `encoder_outputs`: Hidden states from every layer (useful for probing and analysis)
   - `sequence_output`: Final layer hidden states for token-level tasks
   - `pooled_output`: The `[CLS]` token representation passed through a dense layer for sentence-level classification

As documented around line 361 in the TensorFlow notebook, these outputs enable different downstream architectures depending on whether the task requires token-level or sequence-level predictions.

## Implementing BERT in TensorFlow and PyTorch

The `microsoft/AI-For-Beginners` repository provides practical implementations in both frameworks.

### Loading Pre-trained BERT with TensorFlow Hub

For TensorFlow implementations, the repository demonstrates loading BERT from TensorFlow Hub:

```python
import tensorflow_hub as hub
import tensorflow_text as text

# Load BERT preprocessing and encoder modules

preprocess = hub.KerasLayer(
    "https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3")
encoder = hub.KerasLayer(
    "https://tfhub.dev/google/bert_uncased_L-12_H-768_A-12/3",
    trainable=False)  # Freeze encoder for quick experiments

# Build a simple classification model

inputs = tf.keras.layers.Input(shape=(), dtype=tf.string, name="text")
preprocessed = preprocess(inputs)
outputs = encoder(preprocessed)["pooled_output"]  # Shape (batch, 768)

logits = tf.keras.layers.Dense(num_classes, activation="softmax")(outputs)

model = tf.keras.Model(inputs, logits)
model.compile(optimizer="adam",
              loss="sparse_categorical_crossentropy",
              metrics=["accuracy"])

```

The tokenizer adaptation and model definition appear around line 324 in the TensorFlow notebook.

### Using Hugging Face Transformers in PyTorch

For PyTorch implementations, the repository uses the Hugging Face Transformers library:

```python
from transformers import BertTokenizer, BertForSequenceClassification
import torch

# Load tokenizer and model

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
model = BertForSequenceClassification.from_pretrained(
    "bert-base-uncased",
    num_labels=num_classes)

# Example inference

def predict(text):
    inputs = tokenizer(text, return_tensors="pt", 
                       truncation=True, padding=True)
    with torch.no_grad():
        logits = model(**inputs).logits
    return torch.argmax(logits, dim=-1).item()

```

This implementation is shown around line 76 in the PyTorch notebook.

### Fine-Tuning Strategies for Downstream Tasks

Effective fine-tuning requires careful learning rate selection to avoid catastrophic forgetting of pre-trained representations:

**TensorFlow Fine-Tuning:**

```python

# Unfreeze the encoder for task-specific training

encoder.trainable = True
model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=2e-5),
              loss="sparse_categorical_crossentropy",
              metrics=["accuracy"])
model.fit(train_ds.batch(32), epochs=3, 
          validation_data=val_ds.batch(32))

```

**PyTorch Fine-Tuning:**

```python

# Use AdamW optimizer with weight decay

optimizer = torch.optim.AdamW(model.parameters(), lr=2e-5)
model.train()
for epoch in range(3):
    for batch in train_loader:
        optimizer.zero_grad()
        outputs = model(**batch)
        loss = outputs.loss
        loss.backward()
        optimizer.step()

```

Both approaches follow the **freeze → adapt → unfreeze** pattern, using a low learning rate (typically 2e-5) to preserve pre-trained knowledge while adapting to specific tasks.

## Summary

- **Self-attention mechanisms** replace recurrence in Transformers, enabling parallel processing and better long-range dependency modeling than RNNs
- **Multi-head attention** captures diverse relationship types by running multiple attention operations in parallel across different representation subspaces
- **BERT architecture** consists exclusively of Transformer encoder layers, processing text bidirectionally using Masked Language Modeling and Next Sentence Prediction pre-training objectives
- **Model variants** include BERT-Base (12 layers, 768 hidden, 110M parameters) and BERT-Large (24 layers, 1024 hidden, 340M parameters)
- **Transfer learning** requires fine-tuning with low learning rates (2e-5) after pre-training on large corpora like Wikipedia and BookCorpus
- **Output tensors** (`pooled_output` for classification, `sequence_output` for token tasks) enable flexible downstream architectures in both TensorFlow and PyTorch

## Frequently Asked Questions

### What makes Transformers different from RNNs and LSTMs?

Transformers eliminate sequential processing entirely. While RNNs process tokens one at a time and suffer from vanishing gradients over long sequences, Transformers use self-attention to process all tokens simultaneously in constant time relative to layer depth. This parallelization enables training on massive datasets and capturing long-range dependencies that RNNs struggle to learn, as demonstrated in the `lessons/5-NLP/18-Transformers/TransformersTF.ipynb` notebook.

### Why does BERT use the [CLS] token for classification?

The `[CLS]` (classification) token is inserted at the beginning of every input sequence during pre-training. During the Next Sentence Prediction objective, this token learns to aggregate sentence-level information. When fine-tuning for classification tasks, the `pooled_output` associated with this token provides a fixed-dimensional representation of the entire sequence, making it ideal for classification heads without requiring complex pooling strategies over variable-length sequences.

### How does Masked Language Modeling enable bidirectional understanding?

Unlike autoregressive models that only condition on previous tokens, MLM randomly masks 15% of input tokens and requires the model to predict them based on both left and right context. This forces every layer to attend to all positions simultaneously, creating deep bidirectional representations. The model cannot rely on positional autoregression and must instead build rich contextual embeddings using the full surrounding context.

### What is the difference between sequence_output and pooled_output in BERT?

The `sequence_output` (also called `last_hidden_state`) provides contextual embeddings for every input token, making it suitable for token-level tasks like Named Entity Recognition or question answering. The `pooled_output` takes the `[CLS]` token representation from the final layer and passes it through a dense layer with tanh activation, creating a fixed-size sentence embedding ideal for classification tasks. For NER implementations, refer to `lessons/5-NLP/19-NER/NER-TF.ipynb` and `lessons/5-NLP/19-NER/NER-PyTorch.ipynb`.