Transformers and BERT Architecture for NLP Tasks: A Complete Guide with Code Examples

Transformers are neural-network architectures that use self-attention and positional encodings to process all tokens in a sequence in parallel, replacing the sequential bottleneck of recurrent networks.

This guide explains the core mechanisms of Transformers and the BERT architecture as implemented in Microsoft's AI-For-Beginners curriculum. You'll learn how to build a custom Transformer block from scratch and fine-tune pre-trained BERT models for downstream NLP tasks using TensorFlow and HuggingFace.

What Are Transformers?

Transformers revolutionized NLP by eliminating the need for recurrent connections. Instead of processing text word-by-word, they use self-attention to let every token directly attend to every other token simultaneously. This enables:

• Massive parallelism during training and inference
• Long-range dependency modeling without gradient decay
• Transfer learning from large-scale pre-training

The canonical implementation appears in lessons/5-NLP/18-Transformers/TransformersTF.ipynb, which demonstrates both low-level construction and high-level BERT usage.

Core Components of a Transformer Block

A standard Transformer encoder block, as implemented in the Keras TransformerBlock class, contains three essential sub-layers:

Multi-Head Self-Attention

This mechanism projects the input into query, key, and value matrices, then computes scaled dot-product attention across multiple heads. Each head learns different relational patterns. In TransformersTF.ipynb, this is implemented via keras.layers.MultiHeadAttention.

Feed-Forward Network (FFN)

A position-wise two-layer MLP with ReLU activation followed by linear projection. This applies the same transformation to every position independently, adding non-linear capacity.

Add & Norm

Residual connections wrap both attention and FFN sub-layers, with layer normalization applied after each addition. This stabilizes training of deep stacks.

Here is the complete TransformerBlock implementation from the curriculum:

class TransformerBlock(keras.layers.Layer):
    def __init__(self, embed_dim, num_heads, ff_dim, rate=0.1):
        super(TransformerBlock, self).__init__()
        self.att = keras.layers.MultiHeadAttention(
            num_heads=num_heads, key_dim=embed_dim)
        self.ffn = keras.Sequential([
            keras.layers.Dense(ff_dim, activation="relu"),
            keras.layers.Dense(embed_dim),
        ])
        self.layernorm1 = keras.layers.LayerNormalization(epsilon=1e-6)
        self.layernorm2 = keras.layers.LayerNormalization(epsilon=1e-6)
        self.dropout1 = keras.layers.Dropout(rate)
        self.dropout2 = keras.layers.Dropout(rate)

    def call(self, inputs, training):
        attn_output = self.att(inputs, inputs)
        attn_output = self.dropout1(attn_output, training=training)
        out1 = self.layernorm1(inputs + attn_output)
        ffn_output = self.fnn(out1)
        ffn_output = self.dropout2(ffn_output, training=training)
        return self.layernorm2(out1 + ffn_output)

The block expects inputs with shape (batch_size, seq_len, embed_dim) and maintains that shape throughout, allowing arbitrary stacking.

Understanding BERT Architecture

BERT (Bidirectional Encoder Representations from Transformers) is a deep, multi-layer Transformer encoder pre-trained on Wikipedia and BookCorpus. Unlike left-to-right language models, BERT learns bidirectional context through two pre-training objectives:

• Masked Language Modeling (MLM): Random tokens are masked and must be predicted from surrounding context
• Next Sentence Prediction (NSP): Binary classification of whether sentence B follows sentence A

These objectives force the model to encode rich, context-aware representations that transfer effectively to downstream tasks.

BERT Model Variants

Variant	Layers	Hidden Size	Parameters
BERT-base	12	768	110M
BERT-large	24	1024	340M

The curriculum's TransformersTF.ipynb demonstrates both sizes via TensorFlow Hub and HuggingFace.

Fine-Tuning BERT for NLP Tasks

Pre-training provides general language understanding; fine-tuning adapts this to specific tasks. The standard workflow involves three steps:

1. Tokenization with a BERT-compatible tokenizer
2. Encoder forward pass through frozen or trainable weights
3. Classification head training on task-specific labels

Method 1: TensorFlow Hub Pre-trained BERT

This approach loads a lightweight BERT (4 layers, 128 hidden) as a frozen feature extractor:

import tensorflow_hub as hub
import tensorflow as tf
from tensorflow import keras

# Pre-processing layer (tokenization)

vectorizer = hub.KerasLayer(
    'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3')

# Small BERT encoder (frozen)

bert = hub.KerasLayer(
    'https://tfhub.dev/tensorflow/small_bert/bert_en_uncased_L-4_H-128_A-2/1',
    trainable=False)

# Build classification model

inp = keras.Input(shape=(), dtype=tf.string)
x = vectorizer(inp)           # Produces input_word_ids, input_mask, input_type_ids

x = bert(x)                   # Dictionary with pooled_output, sequence_output

x = keras.layers.Dropout(0.1)(x['pooled_output'])
out = keras.layers.Dense(4, activation='softmax')(x)
model = keras.Model(inp, out)

The pooled_output tensor has shape (batch_size, hidden_size) and serves as a sentence-level representation. Setting trainable=False freezes the 4-layer BERT stack, training only the 4-class softmax head.

Method 2: HuggingFace Transformers

For full BERT-base with 12 layers, the curriculum uses HuggingFace's transformers library:

import transformers
from tensorflow import keras

# Load tokenizer and pre-trained model

tokenizer = transformers.BertTokenizer.from_pretrained('bert-base-uncased')
model = transformers.TFBertForSequenceClassification.from_pretrained(
    'bert-base-uncased', num_labels=4)

# Freeze encoder, train only classification head

model.layers[0].trainable = False

model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['acc'])
model.fit(train_dataset, validation_data=val_dataset, epochs=3)

TFBertForSequenceClassification automatically adds a classification layer on top of the pooled output. Freezing model.layers[0] (the BERT encoder) implements feature-based transfer learning; unfreezing enables full fine-tuning for higher accuracy with more data.

Key Implementation Files

The Microsoft/AI-For-Beginners repository provides complete, runnable examples:

• lessons/5-NLP/18-Transformers/TransformersTF.ipynb — TensorFlow implementation of custom Transformer blocks, TensorFlow Hub BERT integration, and HuggingFace fine-tuning
• lessons/5-NLP/18-Transformers/TransformersPyTorch.ipynb — Equivalent PyTorch implementations for PyTorch-native workflows
• lessons/5-NLP/18-Transformers/images/transformer-animated-explanation.gif — Visual self-attention demonstration
• lessons/5-NLP/18-Transformers/images/jalammarBERT-language-modeling-masked-lm.png — BERT pre-training objective diagram

Summary

• Transformers replace recurrence with self-attention, enabling parallel processing and long-range dependencies through multi-head attention, feed-forward networks, and layer normalization
• BERT stacks Transformer encoders and pre-trains bidirectional representations via masked language modeling, making it effective for transfer learning
• Fine-tuning strategies include frozen feature extraction (fast, small data) and full fine-tuning (higher accuracy, larger data), demonstrated with both TensorFlow Hub and HuggingFace
• The TransformersTF.ipynb notebook in Microsoft's AI-For-Beginners provides production-ready code for custom Transformer blocks and BERT adaptation

Frequently Asked Questions

What makes BERT different from earlier Transformer models?

BERT uses only the Transformer encoder stack (no decoder), enabling deep bidirectional context learning. Earlier models like GPT were left-to-right, while BERT's masked language modeling objective allows every token to attend to both previous and subsequent tokens. This produces richer representations for understanding tasks like classification and question answering.

When should I freeze versus fine-tune BERT weights?

Freeze the encoder when you have limited labeled data (hundreds to low thousands of examples) or need fast training—this treats BERT as a fixed feature extractor. Fine-tune all parameters when you have substantial task-specific data (10,000+ examples) and compute budget, as this adapts the pre-trained representations more precisely to your domain.

What outputs does BERT provide for downstream tasks?

BERT returns two primary tensors: sequence_output with shape (batch_size, seq_len, hidden_size) for token-level tasks like NER, and pooled_output with shape (batch_size, hidden_size) for sequence-level classification. The pooled output applies a small transformation to the [CLS] token representation, producing a fixed-size vector regardless of input length.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}