# Working with BERT Transformers for NLP Tasks in the AI-For-Beginners Curriculum

> Learn BERT transformers for NLP with AI-For-Beginners hands-on notebooks. Master masked language modeling and fine-tuning using Hugging Face.

- Repository: [Microsoft/AI-For-Beginners](https://github.com/microsoft/AI-For-Beginners)
- Tags: tutorial
- Published: 2026-08-05

---

**The Microsoft AI-For-Beginners curriculum teaches BERT implementation through hands-on notebooks in `lessons/5-NLP/18-Transformers/`, covering masked language modeling and fine-tuning using the Hugging Face Transformers library.**

The `microsoft/AI-For-Beginners` repository provides a comprehensive introduction to working with BERT transformers for NLP tasks within its natural language processing track. Located in the `5-NLP/18-Transformers` module, the lesson explains the bidirectional encoder architecture and provides runnable code examples for both inference and fine-tuning on downstream classification tasks.

## Locating the BERT Lesson Materials

The core lesson content resides in the `lessons/5-NLP/18-Transformers/` directory, which contains explanatory markdown, Jupyter notebooks, and visual assets. The curriculum includes a detailed diagram illustrating Masked Language Modeling (MLM) at `lessons/5-NLP/18-Transformers/images/jalammarBERT-language-modeling-masked-lm.png`, sourced from the original BERT paper.

For a complete end-to-end reference, the repository provides [`examples/04-text-sentiment.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/examples/04-text-sentiment.py), which demonstrates production-ready sentiment analysis using a pre-trained BERT variant. The root [`requirements.txt`](https://github.com/microsoft/AI-For-Beginners/blob/main/requirements.txt) file lists all necessary dependencies, including `transformers` and `torch`, ensuring the notebook environment executes without configuration issues.

## Understanding BERT Architecture and Pre-Training

### Transformer Encoder Stack

BERT is built from a stack of identical transformer encoder layers. Each layer contains a **multi-head self-attention mechanism** followed by a position-wise feed-forward network. This design enables each token to attend to every other token in the sequence, producing contextualized embeddings that capture bidirectional context rather than left-to-right-only processing.

### Masked Language Modeling (MLM)

During pre-training, approximately **15% of tokens** are replaced with a `[MASK]` token. The model learns to predict the original vocabulary IDs for these masked positions, forcing it to understand context from both directions simultaneously. This objective is visualized in the curriculum's assets showing the MLM prediction task.

### Next Sentence Prediction (NSP)

BERT also employs a **binary classification auxiliary task** where the model predicts whether two input sentences are contiguous in the original text. This NSP objective enhances the model's ability to model inter-sentence relationships, a critical capability for question answering and natural language inference tasks.

## Loading Pre-trained BERT Models for Inference

The curriculum utilizes the **Hugging Face Transformers** library to abstract low-level implementation details. The following pattern demonstrates loading a distilled BERT variant for immediate sentiment classification:

```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

# Load the tokenizer and model (base-uncased fine-tuned for sentiment)

tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased-finetuned-sst-2")
model = AutoModelForSequenceClassification.from_pretrained("distilbert-base-uncased-finetuned-sst-2")

def predict_sentiment(text: str) -> str:
    # Tokenise the input text

    inputs = tokenizer(text, return_tensors="pt", truncation=True, padding=True)
    # Forward pass

    with torch.no_grad():
        logits = model(**inputs).logits
    # Convert logits to label

    predicted_id = logits.argmax().item()
    return model.config.id2label[predicted_id]

print(predict_sentiment("I love the new AI course!"))  # → POSITIVE

```

This implementation leverages `AutoTokenizer` for consistent subword tokenization and `AutoModelForSequenceClassification` for the task-specific head architecture.

## Fine-Tuning BERT on Custom NLP Tasks

For domain-specific applications, the curriculum demonstrates fine-tuning BERT by adding a linear classification head atop the `[CLS]` token representation. The entire pre-trained encoder is trained end-to-end with a small learning rate to preserve learned representations:

```python
import torch
from torch.utils.data import DataLoader, Dataset
from transformers import AutoTokenizer, AutoModelForSequenceClassification, Trainer, TrainingArguments

class SimpleSentimentDataset(Dataset):
    def __init__(self, texts, labels, tokenizer):
        self.encodings = tokenizer(texts, truncation=True, padding=True)
        self.labels = labels
    def __len__(self):
        return len(self.labels)
    def __getitem__(self, idx):
        item = {k: torch.tensor(v[idx]) for k, v in self.encodings.items()}
        item["labels"] = torch.tensor(self.labels[idx])
        return item

# Sample data

texts = ["I hate this.", "This is great!"]
labels = [0, 1]        # 0 = negative, 1 = positive

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
train_dataset = SimpleSentimentDataset(texts, labels, tokenizer)

model = AutoModelForSequenceClassification.from_pretrained(
    "bert-base-uncased", num_labels=2)

training_args = TrainingArguments(
    output_dir="./bert_sentiment",
    num_train_epochs=3,
    per_device_train_batch_size=2,
    learning_rate=2e-5,
    logging_steps=10,
    save_steps=10,
    evaluation_strategy="no",
)

trainer = Trainer(model=model, args=training_args, train_dataset=train_dataset)
trainer.train()

```

The `Trainer` API handles gradient accumulation, checkpointing, and distributed training automatically. As specified in the repository's [`requirements.txt`](https://github.com/microsoft/AI-For-Beginners/blob/main/requirements.txt), executing these examples requires installing the `transformers` and `torch` packages.

## Summary

- **Repository location**: The BERT lesson lives in `lessons/5-NLP/18-Transformers/` within the `microsoft/AI-For-Beginners` curriculum.
- **Core architecture**: BERT uses a bidirectional transformer encoder with multi-head self-attention, pre-trained via MLM (15% token masking) and NSP objectives.
- **Implementation approach**: The curriculum relies on Hugging Face's `AutoTokenizer` and `AutoModelForSequenceClassification` for both inference and fine-tuning workflows.
- **Fine-tuning parameters**: Use a low learning rate (typically **2e-5**) and the `[CLS]` token representation when adapting BERT to downstream classification tasks.
- **Reference code**: The file [`examples/04-text-sentiment.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/examples/04-text-sentiment.py) provides a complete, runnable implementation of BERT-based sentiment analysis.

## Frequently Asked Questions

### What is the difference between BERT and DistilBERT as used in the curriculum?

**DistilBERT** is a compressed version of BERT that retains 97% of BERT's language understanding capabilities while being 40% smaller and 60% faster, as implemented in the inference example using `distilbert-base-uncased-finetuned-sst-2`. Both models share the same transformer encoder architecture, but DistilBERT uses knowledge distillation during training rather than NSP pre-training.

### How does the [MASK] token work in BERT pre-training?

The `[MASK]` token replaces approximately 15% of input tokens randomly during the **Masked Language Modeling** phase. The model must predict the original vocabulary ID based solely on surrounding context, forcing deep bidirectional representation learning. This is fundamentally different from autoregressive models that only condition on previous tokens.

### Why is a low learning rate (2e-5) recommended when fine-tuning BERT?

A small learning rate prevents catastrophic forgetting of the rich semantic knowledge captured during BERT's extensive unsupervised pre-training. The `TrainingArguments` configuration in the fine-tuning example sets `learning_rate=2e-5` because higher rates would rapidly overwrite the carefully tuned attention weights and embeddings, degrading performance on the downstream task.

### Where can I find the complete BERT lesson files in the repository?

All educational materials are located in `lessons/5-NLP/18-Transformers/`, including the explanatory notebook, architectural diagrams at `images/jalammarBERT-language-modeling-masked-lm.png`, and supplementary documentation. The companion script at [`examples/04-text-sentiment.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/examples/04-text-sentiment.py) demonstrates practical implementation patterns discussed in the lesson.