# How Transformers and BERT Are Used for Language Understanding in Microsoft AI-For-Beginners

> Discover how Transformers and BERT power language understanding. Learn about self-attention, bidirectional context, and transfer learning in Microsoft AI-For-Beginners.

- Repository: [Microsoft/AI-For-Beginners](https://github.com/microsoft/AI-For-Beginners)
- Tags: deep-dive
- Published: 2026-08-06

---

**Transformers use self-attention to process text in parallel and capture long-range dependencies, while BERT leverages bidirectional context and pre-training on massive corpora to enable rapid transfer learning for NLP tasks.**

The Microsoft AI-For-Beginners curriculum demonstrates how **transformers and BERT used for language understanding** provide state-of-the-art solutions for natural language processing. The repository contains practical implementations showing how these architectures move beyond recurrent networks to process sequences efficiently and how pre-trained models can be fine-tuned for specific downstream tasks.

## What Are Transformers?

Transformers represent a family of neural architectures that eliminate recurrent connections by relying entirely on **self-attention mechanisms** to process sequences in parallel. This design allows the model to capture relationships between tokens regardless of their distance in the input sequence.

### Core Architecture Components

A standard transformer encoder block, as implemented in the curriculum, consists of five key layers:

1. **Token embeddings** – Converting words or sub-words into dense vector representations.
2. **Positional encodings** – Injecting information about token order since the architecture otherwise processes all positions simultaneously.
3. **Multi-head self-attention** – Allowing every token to attend to every other token, enabling the capture of long-range dependencies.
4. **Feed-forward network** – Applying position-wise transformations independently to each token.
5. **Layer normalization and residual connections** – Stabilizing training by normalizing inputs and preserving gradient flow through skip connections.

## How BERT Extends the Transformer Architecture

BERT (Bidirectional Encoder Representations from Transformers) is a large-scale transformer encoder that stacks 12 identical blocks in BERT-base or 24 blocks in BERT-large. Unlike earlier unidirectional language models, BERT is **bidirectional**, meaning each token's representation integrates context from both its left and right sides simultaneously.

### Pre-training Objectives

BERT learns generic language representations through two unsupervised tasks on massive corpora (Wikipedia and BooksCorpus):

- **Masked Language Modeling (MLM)** – Randomly masking approximately 15% of input tokens and training the model to predict the original vocabulary IDs.
- **Next Sentence Prediction (NSP)** – Training on sentence pairs to predict whether the second sentence logically follows the first.

## Transfer Learning with BERT for Language Understanding

The curriculum demonstrates how BERT's pre-trained hidden states capture syntactic and semantic information that can be fine-tuned for specific NLP tasks. The following table summarizes common applications:

| Task | Implementation Strategy |
|------|------------------------|
| **Text classification** | Extract the hidden state of the `[CLS]` token and feed it into a small classification head. |
| **Token classification** | Apply a token-level classifier to the hidden state of each individual token for named entity recognition or POS tagging. |
| **Question answering** | Predict start and end positions using two linear layers on top of the encoder outputs. |
| **Sentence similarity** | Combine the `[CLS]` embeddings of two sentences to classify their relationship. |

### The Transfer Learning Workflow

According to the source code in [`lessons/5-NLP/18-Transformers/README.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/18-Transformers/README.md), the standard workflow involves six steps:

1. **Load a pre-trained BERT model** from TensorFlow Hub or Hugging Face.
2. **Tokenize raw text** using the matching WordPiece tokenizer.
3. **Create input tensors** including `input_ids`, `attention_mask`, and optional `token_type_ids`.
4. **Forward pass** through BERT to obtain contextual embeddings.
5. **Add a lightweight task-specific head** (typically a linear layer with softmax).
6. **Fine-tune** the network using a small learning rate (typically 2e-5) with optional warm-up.

## Implementation Examples from the Curriculum

The repository provides hands-on notebooks demonstrating both TensorFlow and PyTorch implementations.

### TensorFlow Implementation via TF-Hub

The `TransformersTF.ipynb` notebook demonstrates loading BERT using TensorFlow Hub. The implementation extracts the `[CLS]` representation for classification tasks:

```python
import tensorflow as tf
import tensorflow_hub as hub
import tensorflow_text as text  # loads the BERT tokenizer

# Load BERT-base (uncased) from TF-Hub

bert_url = "https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/4"
bert_layer = hub.KerasLayer(bert_url, trainable=False)

# BERT tokenizer (pre-bundled with the Hub module)

vocab_file = bert_layer.resolved_object.vocab_file.asset_path.numpy()
do_lower_case = bert_layer.resolved_object.do_lower_case.numpy()
tokenizer = text.BertTokenizer(vocab_file, lower_case=do_lower_case)

def preprocess(texts):
    tokens = tokenizer.tokenize(texts)
    tokens = tokens.merge_dims(-2, -1)  # flatten batch & token dim

    input_word_ids = tokens.to_tensor()
    input_mask = tf.ones_like(input_word_ids)
    input_type_ids = tf.zeros_like(input_word_ids)
    return {"input_word_ids": input_word_ids,
            "input_mask": input_mask,
            "input_type_ids": input_type_ids}

# Simple classification head

inputs = {k: tf.keras.layers.Input(shape=(None,), dtype=tf.int32, name=k)
          for k in ["input_word_ids", "input_mask", "input_type_ids"]}

pooled_output = bert_layer(inputs)["pooled_output"]  # [CLS] representation

output = tf.keras.layers.Dense(4, activation="softmax")(pooled_output)

model = tf.keras.Model(inputs, output)
model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=2e-5),
              loss="sparse_categorical_crossentropy",
              metrics=["accuracy"])

```

### PyTorch Implementation with Hugging Face

The `TransformersPyTorch.ipynb` notebook uses the Hugging Face `BertForSequenceClassification` class to fine-tune on the AG News dataset:

```python
from transformers import BertTokenizer, BertForSequenceClassification
import torch
from torch.utils.data import DataLoader, TensorDataset

# Load tokenizer and model

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
model = BertForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=4)

# Example texts

texts = ["The new AI course is amazing!", "I dislike the lecture."]
labels = torch.tensor([1, 0])    # dummy class indices

# Tokenize

encodings = tokenizer(texts, padding=True, truncation=True, return_tensors="pt")
dataset = TensorDataset(encodings["input_ids"], encodings["attention_mask"], labels)
loader = DataLoader(dataset, batch_size=2)

# Fine-tune (single epoch for demo)

optimizer = torch.optim.AdamW(model.parameters(), lr=2e-5)
model.train()
for batch in loader:
    input_ids, attention_mask, labels = batch
    outputs = model(input_ids, attention_mask=attention_mask, labels=labels)
    loss = outputs.loss
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

```

### Named Entity Recognition Applications

The [`lessons/5-NLP/19-NER/README.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/19-NER/README.md) file explains why BERT outperforms LSTM baselines for named entity recognition. The lab materials in [`lessons/5-NLP/19-NER/lab/README.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/19-NER/lab/README.md) provide step-by-step instructions for applying domain-specific BERT variants (such as PubMedBERT) to medical NER tasks, demonstrating how to load specialized models from the Hugging Face model hub.

## Key Source Files in the Repository

The following files contain the canonical implementations and explanations:

- **[`lessons/5-NLP/18-Transformers/README.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/18-Transformers/README.md)** – Overview of the transformer architecture and introduction to BERT.
- **`lessons/5-NLP/18-Transformers/TransformersTF.ipynb`** – TensorFlow walkthrough using TF-Hub for feature extraction and classification.
- **`lessons/5-NLP/18-Transformers/TransformersPyTorch.ipynb`** – PyTorch implementation using `BertForSequenceClassification` for fine-tuning.
- **[`lessons/5-NLP/19-NER/README.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/19-NER/README.md)** – Explanation of BERT's advantages for token-level classification tasks.
- **[`lessons/5-NLP/19-NER/lab/README.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/19-NER/lab/README.md)** – Laboratory instructions for medical NER using domain-specific BERT models.

## Summary

- **Transformers** replace recurrence with self-attention, processing all tokens in parallel while capturing long-range dependencies through multi-head attention layers.
- **BERT** extends the transformer encoder with bidirectional context and pre-training on MLM and NSP objectives, learning rich language representations from Wikipedia and BooksCorpus.
- **Transfer learning** with BERT involves freezing or fine-tuning the pre-trained weights and adding task-specific heads for classification, NER, or question answering.
- The curriculum provides runnable examples in both **TensorFlow** (via TF-Hub) and **PyTorch** (via Hugging Face), demonstrating the `[CLS]` token extraction and token-level prediction patterns.
- Fine-tuning typically requires small learning rates (2e-5) and converges quickly even on modest datasets because BERT already encodes deep linguistic knowledge.

## Frequently Asked Questions

### What makes BERT different from earlier transformer models?

BERT is specifically designed as a **bidirectional** encoder, meaning it considers both left and right context when representing each token. Earlier models like GPT processed text unidirectionally (left-to-right), while BERT's MLM objective allows it to fuse context from both directions simultaneously, resulting in richer semantic representations.

### How does the `[CLS]` token work for text classification?

The `[CLS]` token is a special classification token inserted at the beginning of every input sequence. During pre-training, this token's final hidden state aggregates information from the entire sequence through the self-attention mechanism. For classification tasks, you extract this specific vector and pass it through a softmax layer to predict class labels.

### Why does BERT require a learning rate of 2e-5 during fine-tuning?

Pre-trained BERT models have already converged to a stable region of the loss landscape. Using **small learning rates** (typically between 2e-5 and 5e-5) prevents catastrophic forgetting of the pre-trained weights while allowing gentle adjustment to the target task. Higher learning rates would destabilize the fine-tuning process and overwrite the valuable linguistic knowledge learned during pre-training.

### Can BERT handle domain-specific language like medical or legal text?

Yes, though the curriculum recommends using **domain-specific variants** such as PubMedBERT for medical NLP or FinBERT for financial text. These models use the same transformer architecture but are pre-trained on specialized corpora. The [`lessons/5-NLP/19-NER/lab/README.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/19-NER/lab/README.md) file demonstrates how to load these specialized models from the Hugging Face hub instead of the general `bert-base-uncased` checkpoint.