# Transformers and BERT Architecture for NLP Tasks: A Complete Guide with Code Examples

> Explore Transformers and BERT architecture for NLP tasks. Learn how self-attention and positional encodings enable parallel processing. Get code examples for your AI projects.

- Repository: [Microsoft/AI-For-Beginners](https://github.com/microsoft/AI-For-Beginners)
- Tags: deep-dive
- Published: 2026-08-03

---

**Transformers are neural-network architectures that use self-attention and positional encodings to process all tokens in a sequence in parallel, replacing the sequential bottleneck of recurrent networks.**

This guide explains the core mechanisms of Transformers and the **BERT architecture** as implemented in Microsoft's AI-For-Beginners curriculum. You'll learn how to build a custom Transformer block from scratch and fine-tune pre-trained BERT models for downstream NLP tasks using TensorFlow and HuggingFace.

## What Are Transformers?

Transformers revolutionized NLP by eliminating the need for recurrent connections. Instead of processing text word-by-word, they use **self-attention** to let every token directly attend to every other token simultaneously. This enables:

- **Massive parallelism** during training and inference
- **Long-range dependency modeling** without gradient decay
- **Transfer learning** from large-scale pre-training

The canonical implementation appears in `lessons/5-NLP/18-Transformers/TransformersTF.ipynb`, which demonstrates both low-level construction and high-level BERT usage.

## Core Components of a Transformer Block

A standard Transformer encoder block, as implemented in the Keras `TransformerBlock` class, contains three essential sub-layers:

### Multi-Head Self-Attention

This mechanism projects the input into **query**, **key**, and **value** matrices, then computes scaled dot-product attention across multiple heads. Each head learns different relational patterns. In `TransformersTF.ipynb`, this is implemented via `keras.layers.MultiHeadAttention`.

### Feed-Forward Network (FFN)

A position-wise two-layer MLP with ReLU activation followed by linear projection. This applies the same transformation to every position independently, adding non-linear capacity.

### Add & Norm

Residual connections wrap both attention and FFN sub-layers, with **layer normalization** applied after each addition. This stabilizes training of deep stacks.

Here is the complete `TransformerBlock` implementation from the curriculum:

```python
class TransformerBlock(keras.layers.Layer):
    def __init__(self, embed_dim, num_heads, ff_dim, rate=0.1):
        super(TransformerBlock, self).__init__()
        self.att = keras.layers.MultiHeadAttention(
            num_heads=num_heads, key_dim=embed_dim)
        self.ffn = keras.Sequential([
            keras.layers.Dense(ff_dim, activation="relu"),
            keras.layers.Dense(embed_dim),
        ])
        self.layernorm1 = keras.layers.LayerNormalization(epsilon=1e-6)
        self.layernorm2 = keras.layers.LayerNormalization(epsilon=1e-6)
        self.dropout1 = keras.layers.Dropout(rate)
        self.dropout2 = keras.layers.Dropout(rate)

    def call(self, inputs, training):
        attn_output = self.att(inputs, inputs)
        attn_output = self.dropout1(attn_output, training=training)
        out1 = self.layernorm1(inputs + attn_output)
        ffn_output = self.fnn(out1)
        ffn_output = self.dropout2(ffn_output, training=training)
        return self.layernorm2(out1 + ffn_output)

```

The block expects inputs with shape `(batch_size, seq_len, embed_dim)` and maintains that shape throughout, allowing arbitrary stacking.

## Understanding BERT Architecture

**BERT (Bidirectional Encoder Representations from Transformers)** is a deep, multi-layer Transformer encoder pre-trained on Wikipedia and BookCorpus. Unlike left-to-right language models, BERT learns **bidirectional context** through two pre-training objectives:

- **Masked Language Modeling (MLM)**: Random tokens are masked and must be predicted from surrounding context
- **Next Sentence Prediction (NSP)**: Binary classification of whether sentence B follows sentence A

These objectives force the model to encode rich, context-aware representations that transfer effectively to downstream tasks.

### BERT Model Variants

| Variant | Layers | Hidden Size | Parameters |
|---------|--------|-------------|------------|
| **BERT-base** | 12 | 768 | 110M |
| **BERT-large** | 24 | 1024 | 340M |

The curriculum's `TransformersTF.ipynb` demonstrates both sizes via TensorFlow Hub and HuggingFace.

## Fine-Tuning BERT for NLP Tasks

Pre-training provides general language understanding; **fine-tuning** adapts this to specific tasks. The standard workflow involves three steps:

1. **Tokenization** with a BERT-compatible tokenizer
2. **Encoder forward pass** through frozen or trainable weights
3. **Classification head training** on task-specific labels

### Method 1: TensorFlow Hub Pre-trained BERT

This approach loads a lightweight BERT (4 layers, 128 hidden) as a frozen feature extractor:

```python
import tensorflow_hub as hub
import tensorflow as tf
from tensorflow import keras

# Pre-processing layer (tokenization)

vectorizer = hub.KerasLayer(
    'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3')

# Small BERT encoder (frozen)

bert = hub.KerasLayer(
    'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-4_H-128_A-2/1',
    trainable=False)

# Build classification model

inp = keras.Input(shape=(), dtype=tf.string)
x = vectorizer(inp)           # Produces input_word_ids, input_mask, input_type_ids

x = bert(x)                   # Dictionary with pooled_output, sequence_output

x = keras.layers.Dropout(0.1)(x['pooled_output'])
out = keras.layers.Dense(4, activation='softmax')(x)
model = keras.Model(inp, out)

```

The `pooled_output` tensor has shape `(batch_size, hidden_size)` and serves as a sentence-level representation. Setting `trainable=False` freezes the 4-layer BERT stack, training only the 4-class softmax head.

### Method 2: HuggingFace Transformers

For full BERT-base with 12 layers, the curriculum uses HuggingFace's `transformers` library:

```python
import transformers
from tensorflow import keras

# Load tokenizer and pre-trained model

tokenizer = transformers.BertTokenizer.from_pretrained('bert-base-uncased')
model = transformers.TFBertForSequenceClassification.from_pretrained(
    'bert-base-uncased', num_labels=4)

# Freeze encoder, train only classification head

model.layers[0].trainable = False

model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['acc'])
model.fit(train_dataset, validation_data=val_dataset, epochs=3)

```

`TFBertForSequenceClassification` automatically adds a classification layer on top of the pooled output. Freezing `model.layers[0]` (the BERT encoder) implements **feature-based transfer learning**; unfreezing enables **full fine-tuning** for higher accuracy with more data.

## Key Implementation Files

The Microsoft/AI-For-Beginners repository provides complete, runnable examples:

- **`lessons/5-NLP/18-Transformers/TransformersTF.ipynb`** — TensorFlow implementation of custom Transformer blocks, TensorFlow Hub BERT integration, and HuggingFace fine-tuning
- **`lessons/5-NLP/18-Transformers/TransformersPyTorch.ipynb`** — Equivalent PyTorch implementations for PyTorch-native workflows
- **`lessons/5-NLP/18-Transformers/images/transformer-animated-explanation.gif`** — Visual self-attention demonstration
- **`lessons/5-NLP/18-Transformers/images/jalammarBERT-language-modeling-masked-lm.png`** — BERT pre-training objective diagram

## Summary

- **Transformers** replace recurrence with self-attention, enabling parallel processing and long-range dependencies through multi-head attention, feed-forward networks, and layer normalization
- **BERT** stacks Transformer encoders and pre-trains bidirectional representations via masked language modeling, making it effective for transfer learning
- **Fine-tuning strategies** include frozen feature extraction (fast, small data) and full fine-tuning (higher accuracy, larger data), demonstrated with both TensorFlow Hub and HuggingFace
- The `TransformersTF.ipynb` notebook in Microsoft's AI-For-Beginners provides production-ready code for custom Transformer blocks and BERT adaptation

## Frequently Asked Questions

### What makes BERT different from earlier Transformer models?

BERT uses only the **Transformer encoder** stack (no decoder), enabling deep bidirectional context learning. Earlier models like GPT were left-to-right, while BERT's masked language modeling objective allows every token to attend to both previous and subsequent tokens. This produces richer representations for understanding tasks like classification and question answering.

### When should I freeze versus fine-tune BERT weights?

**Freeze the encoder** when you have limited labeled data (hundreds to low thousands of examples) or need fast training—this treats BERT as a fixed feature extractor. **Fine-tune all parameters** when you have substantial task-specific data (10,000+ examples) and compute budget, as this adapts the pre-trained representations more precisely to your domain.

### What outputs does BERT provide for downstream tasks?

BERT returns two primary tensors: `sequence_output` with shape `(batch_size, seq_len, hidden_size)` for token-level tasks like NER, and `pooled_output` with shape `(batch_size, hidden_size)` for sequence-level classification. The pooled output applies a small transformation to the `[CLS]` token representation, producing a fixed-size vector regardless of input length.