How Transformers and BERT Are Used for Language Understanding in Microsoft AI-For-Beginners

Transformers use self-attention to process text in parallel and capture long-range dependencies, while BERT leverages bidirectional context and pre-training on massive corpora to enable rapid transfer learning for NLP tasks.

The Microsoft AI-For-Beginners curriculum demonstrates how transformers and BERT used for language understanding provide state-of-the-art solutions for natural language processing. The repository contains practical implementations showing how these architectures move beyond recurrent networks to process sequences efficiently and how pre-trained models can be fine-tuned for specific downstream tasks.

What Are Transformers?

Transformers represent a family of neural architectures that eliminate recurrent connections by relying entirely on self-attention mechanisms to process sequences in parallel. This design allows the model to capture relationships between tokens regardless of their distance in the input sequence.

Core Architecture Components

A standard transformer encoder block, as implemented in the curriculum, consists of five key layers:

1. Token embeddings – Converting words or sub-words into dense vector representations.
2. Positional encodings – Injecting information about token order since the architecture otherwise processes all positions simultaneously.
3. Multi-head self-attention – Allowing every token to attend to every other token, enabling the capture of long-range dependencies.
4. Feed-forward network – Applying position-wise transformations independently to each token.
5. Layer normalization and residual connections – Stabilizing training by normalizing inputs and preserving gradient flow through skip connections.

How BERT Extends the Transformer Architecture

BERT (Bidirectional Encoder Representations from Transformers) is a large-scale transformer encoder that stacks 12 identical blocks in BERT-base or 24 blocks in BERT-large. Unlike earlier unidirectional language models, BERT is bidirectional, meaning each token's representation integrates context from both its left and right sides simultaneously.

Pre-training Objectives

BERT learns generic language representations through two unsupervised tasks on massive corpora (Wikipedia and BooksCorpus):

• Masked Language Modeling (MLM) – Randomly masking approximately 15% of input tokens and training the model to predict the original vocabulary IDs.
• Next Sentence Prediction (NSP) – Training on sentence pairs to predict whether the second sentence logically follows the first.

Transfer Learning with BERT for Language Understanding

The curriculum demonstrates how BERT's pre-trained hidden states capture syntactic and semantic information that can be fine-tuned for specific NLP tasks. The following table summarizes common applications:

Task	Implementation Strategy
Text classification	Extract the hidden state of the [CLS] token and feed it into a small classification head.
Token classification	Apply a token-level classifier to the hidden state of each individual token for named entity recognition or POS tagging.
Question answering	Predict start and end positions using two linear layers on top of the encoder outputs.
Sentence similarity	Combine the [CLS] embeddings of two sentences to classify their relationship.

The Transfer Learning Workflow

According to the source code in lessons/5-NLP/18-Transformers/README.md, the standard workflow involves six steps:

1. Load a pre-trained BERT model from TensorFlow Hub or Hugging Face.
2. Tokenize raw text using the matching WordPiece tokenizer.
3. Create input tensors including input_ids, attention_mask, and optional token_type_ids.
4. Forward pass through BERT to obtain contextual embeddings.
5. Add a lightweight task-specific head (typically a linear layer with softmax).
6. Fine-tune the network using a small learning rate (typically 2e-5) with optional warm-up.

Implementation Examples from the Curriculum

The repository provides hands-on notebooks demonstrating both TensorFlow and PyTorch implementations.

TensorFlow Implementation via TF-Hub

The TransformersTF.ipynb notebook demonstrates loading BERT using TensorFlow Hub. The implementation extracts the [CLS] representation for classification tasks:

import tensorflow as tf
import tensorflow_hub as hub
import tensorflow_text as text  # loads the BERT tokenizer

# Load BERT-base (uncased) from TF-Hub

bert_url = "https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/4"
bert_layer = hub.KerasLayer(bert_url, trainable=False)

# BERT tokenizer (pre-bundled with the Hub module)

vocab_file = bert_layer.resolved_object.vocab_file.asset_path.numpy()
do_lower_case = bert_layer.resolved_object.do_lower_case.numpy()
tokenizer = text.BertTokenizer(vocab_file, lower_case=do_lower_case)

def preprocess(texts):
    tokens = tokenizer.tokenize(texts)
    tokens = tokens.merge_dims(-2, -1)  # flatten batch & token dim

    input_word_ids = tokens.to_tensor()
    input_mask = tf.ones_like(input_word_ids)
    input_type_ids = tf.zeros_like(input_word_ids)
    return {"input_word_ids": input_word_ids,
            "input_mask": input_mask,
            "input_type_ids": input_type_ids}

# Simple classification head

inputs = {k: tf.keras.layers.Input(shape=(None,), dtype=tf.int32, name=k)
          for k in ["input_word_ids", "input_mask", "input_type_ids"]}

pooled_output = bert_layer(inputs)["pooled_output"]  # [CLS] representation

output = tf.keras.layers.Dense(4, activation="softmax")(pooled_output)

model = tf.keras.Model(inputs, output)
model.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=2e-5),
              loss="sparse_categorical_crossentropy",
              metrics=["accuracy"])

PyTorch Implementation with Hugging Face

The TransformersPyTorch.ipynb notebook uses the Hugging Face BertForSequenceClassification class to fine-tune on the AG News dataset:

from transformers import BertTokenizer, BertForSequenceClassification
import torch
from torch.utils.data import DataLoader, TensorDataset

# Load tokenizer and model

tokenizer = BertTokenizer.from_pretrained("bert-base-uncased")
model = BertForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=4)

# Example texts

texts = ["The new AI course is amazing!", "I dislike the lecture."]
labels = torch.tensor([1, 0])    # dummy class indices

# Tokenize

encodings = tokenizer(texts, padding=True, truncation=True, return_tensors="pt")
dataset = TensorDataset(encodings["input_ids"], encodings["attention_mask"], labels)
loader = DataLoader(dataset, batch_size=2)

# Fine-tune (single epoch for demo)

optimizer = torch.optim.AdamW(model.parameters(), lr=2e-5)
model.train()
for batch in loader:
    input_ids, attention_mask, labels = batch
    outputs = model(input_ids, attention_mask=attention_mask, labels=labels)
    loss = outputs.loss
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

Named Entity Recognition Applications

The lessons/5-NLP/19-NER/README.md file explains why BERT outperforms LSTM baselines for named entity recognition. The lab materials in lessons/5-NLP/19-NER/lab/README.md provide step-by-step instructions for applying domain-specific BERT variants (such as PubMedBERT) to medical NER tasks, demonstrating how to load specialized models from the Hugging Face model hub.

Key Source Files in the Repository

The following files contain the canonical implementations and explanations:

• lessons/5-NLP/18-Transformers/README.md – Overview of the transformer architecture and introduction to BERT.
• lessons/5-NLP/18-Transformers/TransformersTF.ipynb – TensorFlow walkthrough using TF-Hub for feature extraction and classification.
• lessons/5-NLP/18-Transformers/TransformersPyTorch.ipynb – PyTorch implementation using BertForSequenceClassification for fine-tuning.
• lessons/5-NLP/19-NER/README.md – Explanation of BERT's advantages for token-level classification tasks.
• lessons/5-NLP/19-NER/lab/README.md – Laboratory instructions for medical NER using domain-specific BERT models.

Summary

• Transformers replace recurrence with self-attention, processing all tokens in parallel while capturing long-range dependencies through multi-head attention layers.
• BERT extends the transformer encoder with bidirectional context and pre-training on MLM and NSP objectives, learning rich language representations from Wikipedia and BooksCorpus.
• Transfer learning with BERT involves freezing or fine-tuning the pre-trained weights and adding task-specific heads for classification, NER, or question answering.
• The curriculum provides runnable examples in both TensorFlow (via TF-Hub) and PyTorch (via Hugging Face), demonstrating the [CLS] token extraction and token-level prediction patterns.
• Fine-tuning typically requires small learning rates (2e-5) and converges quickly even on modest datasets because BERT already encodes deep linguistic knowledge.

Frequently Asked Questions

What makes BERT different from earlier transformer models?

BERT is specifically designed as a bidirectional encoder, meaning it considers both left and right context when representing each token. Earlier models like GPT processed text unidirectionally (left-to-right), while BERT's MLM objective allows it to fuse context from both directions simultaneously, resulting in richer semantic representations.

How does the [CLS] token work for text classification?

The [CLS] token is a special classification token inserted at the beginning of every input sequence. During pre-training, this token's final hidden state aggregates information from the entire sequence through the self-attention mechanism. For classification tasks, you extract this specific vector and pass it through a softmax layer to predict class labels.

Why does BERT require a learning rate of 2e-5 during fine-tuning?

Pre-trained BERT models have already converged to a stable region of the loss landscape. Using small learning rates (typically between 2e-5 and 5e-5) prevents catastrophic forgetting of the pre-trained weights while allowing gentle adjustment to the target task. Higher learning rates would destabilize the fine-tuning process and overwrite the valuable linguistic knowledge learned during pre-training.

Can BERT handle domain-specific language like medical or legal text?

Yes, though the curriculum recommends using domain-specific variants such as PubMedBERT for medical NLP or FinBERT for financial text. These models use the same transformer architecture but are pre-trained on specialized corpora. The lessons/5-NLP/19-NER/lab/README.md file demonstrates how to load these specialized models from the Hugging Face hub instead of the general bert-base-uncased checkpoint.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}