# How to Fine-Tune BERT Transformers for Named Entity Recognition: A Complete Guide

> Learn to fine-tune BERT transformers for Named Entity Recognition with this guide. Discover token classification, label alignment, and optimal training settings for superior performance.

- Repository: [Microsoft/AI-For-Beginners](https://github.com/microsoft/AI-For-Beginners)
- Tags: how-to-guide
- Published: 2026-08-09

---

**Fine-tuning BERT transformers for Named Entity Recognition involves replacing the final layer with a token-classification head, aligning BIO-encoded labels with WordPiece tokens using the tokenizer's `word_ids` method, and training with a low learning rate (2e-5) using the AdamW optimizer.**

Fine-tuning BERT transformers for Named Entity Recognition (NER) follows a classic transfer-learning workflow that leverages pre-trained language models for token-level classification tasks. In the `microsoft/AI-For-Beginners` repository, the curriculum demonstrates this process using biomedical text examples with PubMedBERT. This guide walks through the complete implementation, from architecture modifications to label alignment and evaluation metrics.

## Understanding the NER Fine-Tuning Architecture

### Transfer Learning with Domain-Specific Models

According to the source code in [`lessons/5-NLP/19-NER/lab/README.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/19-NER/lab/README.md), selecting the appropriate pre-trained model is critical for domain-specific NER tasks. For biomedical Named Entity Recognition, the curriculum recommends **PubMedBERT** (`microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract`), which has been pre-trained exclusively on PubMed abstracts. This domain alignment significantly improves entity recognition accuracy compared to general-purpose BERT models.

### The Token Classification Head

The architecture modification centers on `BertForTokenClassification`, a class that appends a linear classification layer on top of the BERT encoder. As implemented in the lab materials, this head outputs a logits vector for each input token, with dimensions matching your NER tag vocabulary (e.g., `B-DISEASE`, `I-DISEASE`, `O`). You instantiate this using the `from_pretrained` method while specifying `num_labels` to match your BIO tag set size.

## Preparing Your Dataset and Tokenizer

### BIO Encoding for Entity Labels

Named Entity Recognition datasets use **BIO encoding** (Beginning, Inside, Outside) to mark entity boundaries. Each token receives a label indicating whether it starts an entity (B-), continues an entity (I-), or lies outside any entity (O). The [`lessons/5-NLP/19-NER/lab/README.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/19-NER/lab/README.md) file references helper scripts for converting raw annotation formats (like BC5CDR) into this standardized BIO format.

### Aligning Word-Level Labels with Subword Tokens

BERT tokenizers use **WordPiece** tokenization, which splits rare words into subword units. This creates a misalignment between your word-level BIO labels and the tokenized output. The solution, as shown in [`lessons/5-NLP/19-NER/lab/README.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/19-NER/lab/README.md), uses the tokenizer's `word_ids()` method to map each subword token back to its original word index, allowing you to propagate the correct label to all subword pieces while using `-100` to mask padding and special tokens from loss computation.

## Implementing the Training Pipeline

### Model and Tokenizer Initialization

Begin by loading the pre-trained weights and ensuring tokenizer compatibility. The following code from the curriculum demonstrates proper initialization:

```python
from transformers import AutoTokenizer, BertForTokenClassification

model_name = "microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = BertForTokenClassification.from_pretrained(
    model_name,
    num_labels=NUM_LABELS,  # Length of your tag2id mapping

)

```

### Encoding Examples with Label Alignment

The critical step involves mapping word-level labels to token-level labels. The [`lessons/5-NLP/19-NER/lab/README.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/19-NER/lab/README.md) demonstrates using `is_split_into_words=True` and the `word_ids()` method:

```python
def encode_examples(examples):
    tokenized = tokenizer(
        examples["tokens"],
        is_split_into_words=True,
        truncation=True,
        padding="max_length",
        max_length=128,
    )
    
    label_ids = []
    for i, word_ids in enumerate(tokenized.word_ids(batch_index=0)):
        if word_ids is None:
            label_ids.append(-100)  # Special tokens

        else:
            label_ids.append(label2id.get(examples["labels"][word_ids], -100))
    
    tokenized["labels"] = label_ids
    return tokenized

# Apply to datasets

train_enc = train_dataset.map(encode_examples, batched=True)
val_enc = val_dataset.map(encode_examples, batched=True)

```

### Configuring the Optimizer and Learning Rate

For NER fine-tuning, the [`lessons/5-NLP/18-Transformers/README.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/18-Transformers/README.md) recommends starting with a **low learning rate** of `2e-5` to prevent catastrophic forgetting of pre-trained weights. Use the **AdamW** optimizer with weight decay and optionally freeze lower BERT layers to focus training on the classification head when working with small datasets.

```python
from transformers import TrainingArguments

training_args = TrainingArguments(
    output_dir="./ner-pubmedbert",
    learning_rate=2e-5,
    per_device_train_batch_size=16,
    per_device_eval_batch_size=32,
    num_train_epochs=3,
    weight_decay=0.01,
    evaluation_strategy="epoch",
    logging_steps=50,
)

```

## Evaluating NER Performance

### Computing Entity-Level Metrics with seqeval

Standard classification accuracy is misleading for NER because it ignores entity boundaries. Instead, use **seqeval** to calculate precision, recall, and F1-score at the entity level. The following implementation from the curriculum integrates with the Hugging Face Trainer:

```python
import numpy as np
from seqeval.metrics import f1_score

def compute_metrics(p):
    preds = np.argmax(p.predictions, axis=2)
    true_labels = p.label_ids
    
    # Remove ignored index (-100)

    preds = [[label2id_inv[p] for (p, t) in zip(pred, true_row) if t != -100]
             for pred, true_row in zip(preds, true_labels)]
    true = [[label2id_inv[t] for t in true_row if t != -100]
            for true_row in true_labels]
    
    return {"f1": f1_score(true, preds)}

```

### Running the Training Loop

Instantiate the `Trainer` class with your model, arguments, datasets, and metrics function:

```python
from transformers import Trainer

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_enc,
    eval_dataset=val_enc,
    compute_metrics=compute_metrics,
)

trainer.train()

```

## Summary

- **Model Selection**: Use domain-specific variants like PubMedBERT (`microsoft/BiomedNLP-PubMedBERT-base-uncased-abstract`) for specialized NER tasks, as recommended in [`lessons/5-NLP/19-NER/lab/README.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/19-NER/lab/README.md).
- **Architecture**: Replace the BERT pooler with `BertForTokenClassification` to enable token-level predictions.
- **Label Alignment**: Use the tokenizer's `word_ids()` method to align BIO labels with WordPiece subwords, masking special tokens with `-100`.
- **Training Strategy**: Employ AdamW with a learning rate of `2e-5` and weight decay of `0.01`; consider freezing lower layers for small datasets.
- **Evaluation**: Use **seqeval** metrics rather than token accuracy to properly assess entity boundary detection.

## Frequently Asked Questions

### What is the difference between BertForTokenClassification and standard BERT?

Standard BERT includes a pooler layer designed for sentence-level classification tasks like sentiment analysis. `BertForTokenClassification`, as used in the `microsoft/AI-For-Beginners` curriculum, replaces this with a linear layer that outputs logits for every input token, enabling **token-level classification** required for NER tasks.

### Why use -100 for padding tokens in NER training?

The value `-100` is the default ignore index for PyTorch's `CrossEntropyLoss`. By assigning `-100` to padding tokens and special tokens ([CLS], [SEP]), you ensure these positions do not contribute to the gradient calculation, preventing the model from learning to predict labels for non-content tokens.

### How do I handle subword tokenization when labeling entities?

Use the `word_ids()` method available in Hugging Face tokenizers, as demonstrated in [`lessons/5-NLP/19-NER/lab/README.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/19-NER/lab/README.md). This method returns a mapping from each subword token to its original word index, allowing you to copy the word-level BIO label to all its subword pieces while maintaining alignment.

### Should I freeze BERT layers when fine-tuning for NER?

For small NER datasets, freezing the lower BERT layers and only training the classification head can prevent overfitting and speed up training. However, the [`lessons/5-NLP/18-Transformers/README.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/18-Transformers/README.md) notes that with appropriate regularization (weight decay 0.01) and a low learning rate (2e-5), full fine-tuning often yields better generalization on entity recognition tasks.