# Fine-tune BERT Transformers for Text Classification: Microsoft AI for Beginners Guide

> Learn to fine-tune BERT transformers for text classification with this beginner-friendly guide from Microsoft AI. Use PyTorch or TensorFlow and train on your own data.

- Repository: [Microsoft/AI-For-Beginners](https://github.com/microsoft/AI-For-Beginners)
- Tags: tutorial
- Published: 2026-08-01

---

**You can fine-tune BERT for text classification by loading a pretrained checkpoint, adding a classification head, and training on labeled data using the PyTorch or TensorFlow notebooks in the Microsoft AI-For-Beginners repository.**

The Microsoft AI-For-Beginners curriculum introduces practical natural language processing concepts in lesson **5-NLP → 18-Transformers**, demonstrating how to adapt pretrained BERT models for specific text classification tasks. This guide walks through the complete fine-tuning workflow implemented in the repository's source code, covering architecture components, training loops, and implementation details from both framework variants.

## Understanding BERT Architecture Components

The fine-tuning pipeline consists of three core components that transform raw text into classification predictions. Each component is implemented with specific library calls in the repository's notebooks.

### Tokenization with BertTokenizer

The **BertTokenizer** converts raw text into WordPiece tokens, automatically inserting `[CLS]` and `[SEP]` markers and generating attention masks for the model. According to `lessons/5-NLP/18-Transformers/TransformersPyTorch.ipynb` (line ≈ 552), initialize the tokenizer using:

```python
tokenizer = transformers.BertTokenizer.from_pretrained('bert-base-uncased')

```

This tokenizer handles truncation to `max_length`, padding to uniform sequence lengths, and conversion to numerical input IDs required by the encoder.

### The Pre-trained Encoder

**BERT base** (12 transformer layers, 768 hidden units) provides contextualized word embeddings learned from massive corpora. Both notebooks instantiate the encoder at lines ≈ 546-550 using the checkpoint `bert_model = 'bert-base-uncased'`. Unlike left-to-right language models, BERT's bidirectional attention mechanism processes both preceding and following tokens simultaneously, yielding richer feature representations for classification tasks.

### The Classification Head

The repository implements task-specific heads using framework-specific wrapper classes:

- **PyTorch**: `BertForSequenceClassification` (see `TransformersPyTorch.ipynb`, line ≈ 186)
- **TensorFlow**: `TFBertForSequenceClassification` (see `TransformersTF.ipynb`, line ≈ 651)

These classes add a dense linear layer on top of the `[CLS]` token's final hidden state, followed by softmax activation to produce class probabilities. Instantiate the model with `num_labels` set to your target class count:

```python

# PyTorch implementation from source

model = transformers.BertForSequenceClassification.from_pretrained(
    bert_model, num_labels=4
).to(device)

```

## Step-by-Step Fine-Tuning Process

Follow these seven steps implemented in the Microsoft AI-For-Beginners notebooks to adapt BERT to your classification task:

1. **Select a pretrained checkpoint** (`bert-base-uncased` recommended for beginners).
2. **Instantiate the tokenizer** using `BertTokenizer.from_pretrained()`.
3. **Load the classification model** with `num_labels` matching your target class count.
4. **Create a DataLoader** (PyTorch) or `tf.data.Dataset` (TensorFlow) yielding token tensors and labels.
5. **Define an optimizer** using **AdamW** with a linear warm-up learning rate schedule (typically 2e-5).
6. **Run the training loop** for 2-4 epochs on your labeled dataset.
7. **Evaluate** accuracy and F1-score on a held-out validation set.

### Complete Fine-Tuning Implementation

Below is a minimal, runnable example mirroring the notebook workflow. Save this as [`fine_tune_bert.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/fine_tune_bert.py):

```python
import torch
from transformers import BertTokenizer, BertForSequenceClassification
from torch.utils.data import DataLoader, Dataset
from torch.optim import AdamW

# 1. Load pretrained tokenizer

BERT_MODEL = "bert-base-uncased"
tokenizer = BertTokenizer.from_pretrained(BERT_MODEL)

# 2. Prepare dataset

class TextDataset(Dataset):
    def __init__(self, texts, labels):
        self.encodings = tokenizer(
            texts, 
            truncation=True, 
            padding="max_length", 
            max_length=128,
            return_tensors="pt"
        )
        self.labels = torch.tensor(labels)
    
    def __len__(self):
        return len(self.labels)
    
    def __getitem__(self, idx):
        item = {k: v[idx] for k, v in self.encodings.items()}
        item["labels"] = self.labels[idx]
        return item

# Example sentiment data

texts = ["I love this movie!", "Terrible plot and bad acting."]
labels = [1, 0]  # 1 = positive, 0 = negative

dataset = TextDataset(texts, labels)
loader = DataLoader(dataset, batch_size=2, shuffle=True)

# 3. Initialize model with classification head

model = BertForSequenceClassification.from_pretrained(
    BERT_MODEL, 
    num_labels=2
)
model.train()

# 4. Configure optimizer

optimizer = AdamW(model.parameters(), lr=2e-5)

# 5. Training loop

for batch in loader:
    optimizer.zero_grad()
    outputs = model(**batch)
    loss = outputs.loss
    loss.backward()
    optimizer.step()
    print(f"Loss: {loss.item():.4f}")

# 6. Inference

model.eval()
with torch.no_grad():
    test_input = tokenizer(
        ["What a fantastic experience!"], 
        return_tensors="pt", 
        truncation=True, 
        padding=True
    )
    logits = model(**test_input).logits
    prediction = torch.argmax(logits, dim=1).item()
    print(f"Predicted class: {prediction}")

```

## PyTorch vs TensorFlow Implementation Details

The repository provides equivalent implementations in both deep learning frameworks:

- **PyTorch version**: `lessons/5-NLP/18-Transformers/TransformersPyTorch.ipynb` uses the `Trainer` API and standard PyTorch training loops with `BertForSequenceClassification`.
- **TensorFlow version**: `lessons/5-NLP/18-Transformers/TransformersTF.ipynb` implements `TFBertForSequenceClassification` with `tf.keras` training loops and `output_attentions=False` for memory efficiency.

Both versions process identical tokenized inputs (`input_ids`, `attention_mask`) and target labels, allowing seamless framework selection based on your existing infrastructure.

## Why Fine-Tuning BERT Works

**Transfer learning** enables rapid adaptation because BERT's encoder has already captured general language patterns (syntax, semantics, and contextual relationships) from massive pretraining corpora. By exposing the model to a small, task-specific dataset, you adjust these high-level representations to your specific categories while introducing only a few additional parameters through the classification head. This approach achieves state-of-the-art performance on sentiment analysis and topic categorization benchmarks with minimal training data.

For a non-transformer baseline before implementing BERT, examine [`examples/04-text-sentiment.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/examples/04-text-sentiment.py), which demonstrates classic bag-of-words sentiment analysis using traditional machine learning techniques.

## Summary

- **BertTokenizer** processes raw text into WordPiece tokens with `[CLS]` and `[SEP]` markers, implemented at line ≈ 552 in `TransformersPyTorch.ipynb`.
- Load **BertForSequenceClassification** (PyTorch, line ≈ 186) or **TFBertForSequenceClassification** (TensorFlow, line ≈ 651) with `num_labels` matching your class count.
- Use **AdamW** optimizer with a 2e-5 learning rate and train for 2-4 epochs using the implementations in `lessons/5-NLP/18-Transformers/`.
- Both PyTorch and TensorFlow notebooks reference `bert-base-uncased` at lines ≈ 546-550 for the encoder weights.
- Consider freezing early BERT layers (`model.bert.encoder.layer[-X:].requires_grad = True`) to reduce training time, or start with [`examples/04-text-sentiment.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/examples/04-text-sentiment.py) for simpler baselines.

## Frequently Asked Questions

### How do I choose between PyTorch and TensorFlow for BERT fine-tuning?

Both frameworks offer identical model capabilities through the Hugging Face transformers library. Choose PyTorch if you prefer explicit training loops and dynamic computation graphs, or TensorFlow if you favor the Keras API and built-in distribution strategies. The Microsoft AI-For-Beginners repository provides complete, functional notebooks for both approaches at `lessons/5-NLP/18-Transformers/TransformersPyTorch.ipynb` and `TransformersTF.ipynb`.

### Can I fine-tune BERT on a CPU instead of GPU?

Yes, but training will be significantly slower. The notebooks automatically detect CUDA availability with `.to(device)` (PyTorch) or default device placement (TensorFlow). For CPU training, reduce `batch_size` and `max_length` to prevent out-of-memory errors, or freeze all BERT layers and only train the classification head to speed up computation.

### What is the difference between `bert-base-uncased` and `bert-large-uncased`?

**BERT base** uses 12 transformer layers with 768 hidden units (110M parameters), while **BERT large** uses 24 layers with 1024 hidden units (340M parameters). Beginners should start with `bert-base-uncased` as shown in lines ≈ 546-550 of both notebooks, since it trains faster and requires less memory while maintaining strong performance on most text classification tasks.

### How do I handle multi-class classification with more than two labels?

Set `num_labels` to your class count when instantiating `BertForSequenceClassification`. The model automatically adjusts the output dimension of the classification head, and the loss function adapts from binary cross-entropy to categorical cross-entropy. Ensure your labels are zero-indexed integers (ranging from 0 to N-1) in your dataset preparation to match the expected format in the training loop.