How to Work with BERT Transformers for NLP in the AI-For-Beginners Curriculum

The AI-For-Beginners curriculum teaches BERT transformers through two practical implementations: TensorFlow 2 with TensorFlow Hub and PyTorch with Hugging Face Transformers, both following a standard workflow of tokenization, encoder loading, classification head attachment, and fine-tuning with small learning rates.

The Microsoft AI-For-Beginners repository provides a comprehensive introduction to BERT (Bidirectional Encoder Representations from Transformers) in lesson 5-NLP → 18-Transformers. This lesson demonstrates how to leverage pre-trained language models for downstream NLP tasks without training from scratch. Whether you prefer TensorFlow or PyTorch, the curriculum offers complete, runnable notebooks that implement the same core concepts using framework-specific tools.

Understanding BERT Architecture in the Curriculum

The AI-For-Beginners curriculum explains BERT's key architectural innovations in the TransformersTF notebook at lines 34-38. These concepts underpin why BERT became foundational for modern NLP.

Bidirectional Context Processing

Unlike traditional left-to-right language models, BERT processes the entire sentence simultaneously. Each token can attend to both its left and right context, capturing richer semantic relationships than unidirectional alternatives.

Pre-training Objectives

BERT learns through two self-supervised tasks during pre-training:

• Masked Language Modeling (MLM): 15% of input tokens are masked, and the model learns to predict them—forcing deep contextual understanding
• Next-Sentence Prediction (NSP): The model learns relationships between sentence pairs, critical for entailment and question-answering tasks

Transfer Learning Benefits

The bulk of linguistic knowledge resides in the pre-trained encoder. Fine-tuning requires far fewer labeled examples and converges quickly compared to training from scratch.

TensorFlow 2 Implementation with TensorFlow Hub

The TransformersTF.ipynb notebook demonstrates BERT implementation using TensorFlow Hub components. This approach separates preprocessing, encoding, and classification into modular Keras layers.

Loading the Preprocessing Layer and Encoder

import tensorflow as tf
import tensorflow_hub as hub
import tensorflow_text as text  # Required for preprocessing

# Load BERT preprocessing (tokenization)

bert_preprocess = hub.KerasLayer(
    'https://tfhub.dev/tensorflow/bert_en_uncased_preprocess/3',
    name='bert_preprocess'
)

# Load pre-trained BERT encoder (frozen by default)

bert_encoder = hub.KerasLayer(
    'https://tfhub.dev/google/bert_uncased_L-12_H-768_A-12/3',
    trainable=False,
    name='bert_encoder'
)

The preprocessing layer handles word-piece tokenization automatically. The encoder specification L-12_H-768_A-12 indicates 12 transformer layers, 768 hidden dimensions, and 12 attention heads—the standard BERT-base configuration.

Building the Classification Model

num_classes = 4  # Example: AG-News categories

inputs = tf.keras.layers.Input(shape=(), dtype=tf.string, name='text')
preprocessed = bert_preprocess(inputs)
outputs = bert_encoder(preprocessed)

# Use pooled_output for sequence classification

pooled_embedding = outputs['pooled_output']
logits = tf.keras.layers.Dense(
    num_classes,
    activation='softmax',
    name='classifier'
)(pooled_embedding)

model = tf.keras.Model(inputs, logits)

model.compile(
    optimizer=tf.keras.optimizers.Adam(learning_rate=2e-5),
    loss='sparse_categorical_crossentropy',
    metrics=['accuracy']
)

The pooled_output tensor provides a fixed-length representation of the entire input sequence, suitable for classification. Setting trainable=False freezes BERT's weights, training only the new classification head. For full fine-tuning, change to trainable=True.

Training the Model

model.fit(
    train_ds.batch(32),
    epochs=3,
    validation_data=val_ds.batch(32)
)

The small learning rate (2e-5) prevents catastrophic forgetting of pre-trained knowledge. This value appears consistently across BERT fine-tuning implementations in the curriculum.

PyTorch Implementation with Hugging Face Transformers

The TransformersPyTorch.ipynb notebook provides equivalent functionality using the Hugging Face ecosystem. This approach offers more pre-built utilities but requires manual training loop construction.

Loading Tokenizer and Model

import torch
from transformers import BertTokenizer, BertForSequenceClassification

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForSequenceClassification.from_pretrained(
    'bert-base-uncased',
    num_labels=4
)

BertForSequenceClassification automatically appends a classification head to the pooled output. The model architecture matches the TensorFlow implementation: 12 layers, 768 hidden size, 12 attention heads.

Preparing Optimizer with Warm-up Schedule

from transformers import AdamW, get_linear_schedule_with_warmup

optimizer = AdamW(model.parameters(), lr=2e-5)

scheduler = get_linear_schedule_with_warmup(
    optimizer,
    num_warmup_steps=0,
    num_training_steps=len(train_loader) * epochs
)

The AdamW optimizer with linear learning rate scheduling is standard for transformer fine-tuning. Warm-up steps gradually increase learning rate early in training, improving stability.

Training Loop Implementation

model.train()

for epoch in range(epochs):
    for batch in train_loader:
        # Tokenize on-the-fly or use pre-tokenized data

        inputs = tokenizer(
            batch['text'],
            padding=True,
            truncation=True,
            max_length=128,
            return_tensors='pt'
        )
        labels = batch['label']
        
        outputs = model(**inputs, labels=labels)
        loss = outputs.loss
        
        loss.backward()
        optimizer.step()
        scheduler.step()
        optimizer.zero_grad()

Unlike the Keras API, PyTorch requires explicit gradient management. The scheduler.step() call adjusts learning rate according to the warm-up schedule.

Comparing the Two Approaches

Aspect	TensorFlow + TF-Hub	PyTorch + Hugging Face
Abstraction level	Higher (Keras Model API)	Lower (manual training loop)
Preprocessing	Integrated Keras layer	tokenizer() function calls
Classification head	Manual Dense layer	Built into model class
Optimizer setup	Standard Keras optimizer	AdamW with custom scheduler
Best for	Quick prototyping, production deployment	Research flexibility, fine-grained control

Both implementations achieve identical results when properly configured. The curriculum includes both to accommodate different user preferences and existing toolchains.

Extending BERT to Named Entity Recognition

The curriculum applies BERT to token-level prediction in lessons/5-NLP/19-NER/NER-TF.ipynb. Instead of pooled_output, this task uses sequence_output—the contextual embedding for each input token.

# In NER-TF.ipynb, outputs['sequence_output'] provides

# per-token embeddings for tagging each word with its entity type

This demonstrates BERT's flexibility: the same pre-trained encoder serves both sentence-level and token-level prediction tasks by selecting different output tensors.

Best Practices from the Source Code

The AI-For-Beginners notebooks demonstrate several critical patterns for working with BERT transformers:

• Learning rate: Never exceed 5e-5; 2e-5 is the curriculum's default
• Batch size: 32 works well for BERT-base on consumer GPUs
• Sequence length: 128 tokens balances coverage and memory; increase to 512 only when necessary
• Warm-up: Linear scheduling with 0-10% warm-up steps stabilizes early training
• Layer freezing: Start with frozen encoder, unfreeze only if task performance requires it

Summary

• The AI-For-Beginners curriculum teaches BERT in lessons/5-NLP/18-Transformers/ through parallel TensorFlow and PyTorch implementations
• TensorFlow + TF-Hub uses hub.KerasLayer for preprocessing and encoding, with manual classification head construction
• PyTorch + Hugging Face provides BertForSequenceClassification with integrated head and requires manual training loop implementation
• Both approaches follow the same workflow: tokenization → encoder loading → classification head → fine-tuning with small learning rates
• The NER lesson extends these patterns to token-level prediction using sequence_output instead of pooled_output

Frequently Asked Questions

What hardware requirements does the AI-For-Beginners BERT lesson assume?

The notebooks run on CPU but benefit significantly from GPU acceleration. BERT-base with 110 million parameters requires approximately 1.5GB GPU memory for batch size 32 at sequence length 128. The curriculum code includes device-agnostic patterns that automatically use CUDA when available.

How does the curriculum handle text preprocessing differences between frameworks?

The TensorFlow implementation uses tensorflow_text operations embedded in a Keras layer, making preprocessing part of the saved model. The PyTorch implementation calls tokenizer() explicitly in Python, requiring manual handling during inference. Both tokenizers use the same word-piece vocabulary and produce identical numerical outputs for the same input text.

Can I use the AI-For-Beginners BERT code for languages other than English?

Yes, by substituting the model identifier or TF-Hub URL. Replace 'bert-base-uncased' with multilingual variants like 'bert-base-multilingual-cased' or language-specific models. The preprocessing layer URL in TensorFlow must also change to match the target language's vocabulary.

What is the difference between pooled_output and sequence_output in BERT?

pooled_output provides a single 768-dimensional vector representing the entire sequence, used for classification tasks. sequence_output provides a vector for every input token, used for token-level tasks like NER or question answering. The AI-For-Beginners NER notebook demonstrates accessing sequence_output from the TF-Hub encoder's dictionary return value.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}