# How to Work with BERT Transformers for NLP in the AI-For-Beginners Curriculum

> Learn to work with BERT transformers for NLP in the AI For Beginners curriculum using TensorFlow or PyTorch. Explore tokenization, model loading, and fine-tuning for practical AI understanding.

- Repository: [Microsoft/AI-For-Beginners](https://github.com/microsoft/AI-For-Beginners)
- Tags: tutorial
- Published: 2026-08-08

---

**The AI-For-Beginners curriculum teaches BERT transformers through two practical implementations: TensorFlow 2 with TensorFlow Hub and PyTorch with Hugging Face Transformers, both following a standard workflow of tokenization, encoder loading, classification head attachment, and fine-tuning with small learning rates.**

The Microsoft AI-For-Beginners repository provides a comprehensive introduction to BERT (Bidirectional Encoder Representations from Transformers) in lesson 5-NLP → 18-Transformers. This lesson demonstrates how to leverage pre-trained language models for downstream NLP tasks without training from scratch. Whether you prefer TensorFlow or PyTorch, the curriculum offers complete, runnable notebooks that implement the same core concepts using framework-specific tools.

## Understanding BERT Architecture in the Curriculum

The AI-For-Beginners curriculum explains BERT's key architectural innovations in the TransformersTF notebook at lines 34-38. These concepts underpin why BERT became foundational for modern NLP.

### Bidirectional Context Processing

Unlike traditional left-to-right language models, **BERT processes the entire sentence simultaneously**. Each token can attend to both its left and right context, capturing richer semantic relationships than unidirectional alternatives.

### Pre-training Objectives

BERT learns through two self-supervised tasks during pre-training:

- **Masked Language Modeling (MLM)**: 15% of input tokens are masked, and the model learns to predict them—forcing deep contextual understanding
- **Next-Sentence Prediction (NSP)**: The model learns relationships between sentence pairs, critical for entailment and question-answering tasks

### Transfer Learning Benefits

The bulk of linguistic knowledge resides in the pre-trained encoder. Fine-tuning requires **far fewer labeled examples** and converges quickly compared to training from scratch.

## TensorFlow 2 Implementation with TensorFlow Hub

The `TransformersTF.ipynb` notebook demonstrates BERT implementation using TensorFlow Hub components. This approach separates preprocessing, encoding, and classification into modular Keras layers.

### Loading the Preprocessing Layer and Encoder

```python
import tensorflow as tf
import tensorflow_hub as hub
import tensorflow_text as text  # Required for preprocessing

# Load BERT preprocessing (tokenization)

bert_preprocess = hub.KerasLayer(
    'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',
    name='bert_preprocess'
)

# Load pre-trained BERT encoder (frozen by default)

bert_encoder = hub.KerasLayer(
    'https://tfhub.dev/google/bert_uncased_L-12_H-768_A-12/3',
    trainable=False,
    name='bert_encoder'
)

```

The preprocessing layer handles word-piece tokenization automatically. The encoder specification `L-12_H-768_A-12` indicates 12 transformer layers, 768 hidden dimensions, and 12 attention heads—the standard BERT-base configuration.

### Building the Classification Model

```python
num_classes = 4  # Example: AG-News categories

inputs = tf.keras.layers.Input(shape=(), dtype=tf.string, name='text')
preprocessed = bert_preprocess(inputs)
outputs = bert_encoder(preprocessed)

# Use pooled_output for sequence classification

pooled_embedding = outputs['pooled_output']
logits = tf.keras.layers.Dense(
    num_classes,
    activation='softmax',
    name='classifier'
)(pooled_embedding)

model = tf.keras.Model(inputs, logits)

model.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate=2e-5),
    loss='sparse_categorical_crossentropy',
    metrics=['accuracy']
)

```

The `pooled_output` tensor provides a fixed-length representation of the entire input sequence, suitable for classification. Setting `trainable=False` freezes BERT's weights, training only the new classification head. For full fine-tuning, change to `trainable=True`.

### Training the Model

```python
model.fit(
    train_ds.batch(32),
    epochs=3,
    validation_data=val_ds.batch(32)
)

```

The small learning rate (2e-5) prevents catastrophic forgetting of pre-trained knowledge. This value appears consistently across BERT fine-tuning implementations in the curriculum.

## PyTorch Implementation with Hugging Face Transformers

The `TransformersPyTorch.ipynb` notebook provides equivalent functionality using the Hugging Face ecosystem. This approach offers more pre-built utilities but requires manual training loop construction.

### Loading Tokenizer and Model

```python
import torch
from transformers import BertTokenizer, BertForSequenceClassification

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForSequenceClassification.from_pretrained(
    'bert-base-uncased',
    num_labels=4
)

```

`BertForSequenceClassification` automatically appends a classification head to the pooled output. The model architecture matches the TensorFlow implementation: 12 layers, 768 hidden size, 12 attention heads.

### Preparing Optimizer with Warm-up Schedule

```python
from transformers import AdamW, get_linear_schedule_with_warmup

optimizer = AdamW(model.parameters(), lr=2e-5)

scheduler = get_linear_schedule_with_warmup(
    optimizer,
    num_warmup_steps=0,
    num_training_steps=len(train_loader) * epochs
)

```

The **AdamW optimizer** with linear learning rate scheduling is standard for transformer fine-tuning. Warm-up steps gradually increase learning rate early in training, improving stability.

### Training Loop Implementation

```python
model.train()

for epoch in range(epochs):
    for batch in train_loader:
        # Tokenize on-the-fly or use pre-tokenized data

        inputs = tokenizer(
            batch['text'],
            padding=True,
            truncation=True,
            max_length=128,
            return_tensors='pt'
        )
        labels = batch['label']
        
        outputs = model(**inputs, labels=labels)
        loss = outputs.loss
        
        loss.backward()
        optimizer.step()
        scheduler.step()
        optimizer.zero_grad()

```

Unlike the Keras API, PyTorch requires explicit gradient management. The `scheduler.step()` call adjusts learning rate according to the warm-up schedule.

## Comparing the Two Approaches

| Aspect | TensorFlow + TF-Hub | PyTorch + Hugging Face |
|--------|---------------------|------------------------|
| **Abstraction level** | Higher (Keras Model API) | Lower (manual training loop) |
| **Preprocessing** | Integrated Keras layer | `tokenizer()` function calls |
| **Classification head** | Manual `Dense` layer | Built into model class |
| **Optimizer setup** | Standard Keras optimizer | `AdamW` with custom scheduler |
| **Best for** | Quick prototyping, production deployment | Research flexibility, fine-grained control |

Both implementations achieve identical results when properly configured. The curriculum includes both to accommodate different user preferences and existing toolchains.

## Extending BERT to Named Entity Recognition

The curriculum applies BERT to token-level prediction in `lessons/5-NLP/19-NER/NER-TF.ipynb`. Instead of `pooled_output`, this task uses `sequence_output`—the contextual embedding for **each input token**.

```python

# In NER-TF.ipynb, outputs['sequence_output'] provides

# per-token embeddings for tagging each word with its entity type

```

This demonstrates BERT's flexibility: the same pre-trained encoder serves both sentence-level and token-level prediction tasks by selecting different output tensors.

## Best Practices from the Source Code

The AI-For-Beginners notebooks demonstrate several critical patterns for working with BERT transformers:

- **Learning rate**: Never exceed 5e-5; 2e-5 is the curriculum's default
- **Batch size**: 32 works well for BERT-base on consumer GPUs
- **Sequence length**: 128 tokens balances coverage and memory; increase to 512 only when necessary
- **Warm-up**: Linear scheduling with 0-10% warm-up steps stabilizes early training
- **Layer freezing**: Start with frozen encoder, unfreeze only if task performance requires it

## Summary

- The AI-For-Beginners curriculum teaches BERT in `lessons/5-NLP/18-Transformers/` through parallel TensorFlow and PyTorch implementations
- **TensorFlow + TF-Hub** uses `hub.KerasLayer` for preprocessing and encoding, with manual classification head construction
- **PyTorch + Hugging Face** provides `BertForSequenceClassification` with integrated head and requires manual training loop implementation
- Both approaches follow the same workflow: tokenization → encoder loading → classification head → fine-tuning with small learning rates
- The NER lesson extends these patterns to token-level prediction using `sequence_output` instead of `pooled_output`

## Frequently Asked Questions

### What hardware requirements does the AI-For-Beginners BERT lesson assume?

The notebooks run on CPU but benefit significantly from GPU acceleration. BERT-base with 110 million parameters requires approximately 1.5GB GPU memory for batch size 32 at sequence length 128. The curriculum code includes device-agnostic patterns that automatically use CUDA when available.

### How does the curriculum handle text preprocessing differences between frameworks?

The TensorFlow implementation uses `tensorflow_text` operations embedded in a Keras layer, making preprocessing part of the saved model. The PyTorch implementation calls `tokenizer()` explicitly in Python, requiring manual handling during inference. Both tokenizers use the same word-piece vocabulary and produce identical numerical outputs for the same input text.

### Can I use the AI-For-Beginners BERT code for languages other than English?

Yes, by substituting the model identifier or TF-Hub URL. Replace `'bert-base-uncased'` with multilingual variants like `'bert-base-multilingual-cased'` or language-specific models. The preprocessing layer URL in TensorFlow must also change to match the target language's vocabulary.

### What is the difference between `pooled_output` and `sequence_output` in BERT?

`pooled_output` provides a single 768-dimensional vector representing the entire sequence, used for classification tasks. `sequence_output` provides a vector for every input token, used for token-level tasks like NER or question answering. The AI-For-Beginners NER notebook demonstrates accessing `sequence_output` from the TF-Hub encoder's dictionary return value.