Named Entity Recognition (NER) Implementation Examples: LSTM and BERT from Scratch

The microsoft/AI-For-Beginners curriculum provides complete, runnable Named Entity Recognition (NER) implementation examples using both a classic Bi-LSTM architecture in TensorFlow and a fine-tuned BERT transformer via HuggingFace, demonstrating token-level classification on BIO-encoded biomedical datasets.

The repository's lesson 19 located at lessons/5-NLP/19-NER treats NER as a supervised token-level classification task, where each word receives a label such as B-DATE, I-LOC, or O. These progressive examples guide learners through the full ML lifecycle—from raw text preprocessing to production-ready inference—using the BC5CDR corpus and extensible CSV formats.

Understanding the NER Pipeline Architecture

The curriculum organizes NER as a comparative study between traditional recurrent networks and modern transformers. Both approaches frame the task as token-level classification, assigning BIO (Beginning-Inside-Outside) tags to identify entity boundaries for person names, dates, locations, and chemical formulas. The README.md in lessons/5-NLP/19-NER outlines the problem definition and highlights why NER serves as a critical preprocessing step for downstream conversational agents and information extraction systems.

LSTM-Based NER Implementation with TensorFlow

Located in lessons/5-NLP/19-NER/NER-TF.ipynb, the LSTM implementation demonstrates how to build a sequence labeling model from scratch without relying on pre-trained embeddings.

Loading the Dataset

The examples use either the public BC5CDR biomedical corpus or a simplified CSV toy dataset (ner_dataset.csv) with pre-tokenized text and pipe-delimited BIO tags.

import pandas as pd

# The CSV file is shipped with the lesson (ner_dataset.csv)

df = pd.read_csv("lessons/5-NLP/19-NER/ner_dataset.csv")
sentences = df["sentence"].tolist()
tags      = df["tags"].tolist()          # BIO-encoded strings

Building the Bi-LSTM Model

The TensorFlow pipeline constructs an embedding layer initialized randomly, followed by a bidirectional LSTM and time-distributed dense layer for per-token softmax predictions.

import tensorflow as tf
from tensorflow.keras.preprocessing.sequence import pad_sequences
from tensorflow.keras.utils import to_categorical

# ---- Tokenise -------------------------------------------------

tokenizer = tf.keras.preprocessing.text.Tokenizer(lower=False, oov_token="<OOV>")
tokenizer.fit_on_texts(sentences)
X = tokenizer.texts_to_sequences(sentences)

# ---- Encode tags ------------------------------------------------

tag2idx = {t: i for i, t in enumerate(sorted(set("|".join(tags).split("|"))))}
y = [[tag2idx[t] for t in seq.split("|")] for seq in tags]

# ---- Pad -------------------------------------------------------

max_len = max(len(s) for s in X)
X = pad_sequences(X, maxlen=max_len, padding="post")
y = pad_sequences(y, maxlen=max_len, padding="post")
y = to_categorical(y, num_classes=len(tag2idx))

# ---- Model ------------------------------------------------------

model = tf.keras.Sequential([
    tf.keras.layers.Embedding(input_dim=len(tokenizer.word_index)+1,
                             output_dim=128,
                             mask_zero=True),
    tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64, return_sequences=True)),
    tf.keras.layers.TimeDistributed(tf.keras.layers.Dense(len(tag2idx), activation="softmax"))
])

model.compile(optimizer="adam", loss="categorical_crossentropy", metrics=["accuracy"])
model.fit(X, y, epochs=5, batch_size=32, validation_split=0.1)

BERT-Based NER with HuggingFace Transformers

The second implementation swaps the recurrent layer for a pre-trained transformer, leveraging AutoModelForTokenClassification from the 🤗 Transformers library to demonstrate transfer learning on token classification tasks.

Tokenization and Label Alignment

Unlike the LSTM approach, BERT uses WordPiece tokenization that splits words into subwords. The implementation aligns original word-level BIO tags with the subword tokens, assigning -100 to special tokens and subsequent subwords to ignore them during loss computation.

from transformers import AutoTokenizer, AutoModelForTokenClassification, Trainer, TrainingArguments
import torch

# ---- Load pre-trained model & tokenizer -------------------------

model_name = "bert-base-cased"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForTokenClassification.from_pretrained(model_name,
                                                        num_labels=len(tag2idx))

Fine-Tuning Configuration

The training loop uses the Trainer API with TrainingArguments configured for 3 epochs, demonstrating how transfer learning achieves higher accuracy with fewer training steps than the LSTM baseline.

# ---- Encode a single sentence ----------------------------------

def encode(sentence, tags):
    enc = tokenizer(sentence.split(),
                    is_split_into_words=True,
                    return_offsets_mapping=True,
                    padding="max_length",
                    truncation=True,
                    max_length=max_len)
    # Align label IDs with tokenised sub-words

    label_ids = []
    word_ids = enc.word_ids()
    previous_word_idx = None
    for w in word_ids:
        if w is None:
            label_ids.append(-100)          # ignore token (special tokens)

        elif w != previous_word_idx:
            label_ids.append(tag2idx[tags[w]])  # first sub-word gets the label

        else:
            label_ids.append(tag2idx[tags[w]] if label_all_tokens else -100)
        previous_word_idx = w
    enc["labels"] = label_ids
    return enc

train_encodings = [encode(s, t.split("|")) for s, t in zip(sentences, tags)]

# ---- Trainer ----------------------------------------------------

args = TrainingArguments(
    output_dir="./ner_bert",
    evaluation_strategy="epoch",
    per_device_train_batch_size=16,
    num_train_epochs=3,
    logging_dir="./logs",
)

trainer = Trainer(
    model=model,
    args=args,
    train_dataset=train_encodings,
)

trainer.train()

Inference and Evaluation

Both models support inference on raw text strings, returning token-tag pairs that identify entities like locations (B-LOC) and dates (B-DATE). The lab/README.md suggests extensions for applying these models to COVID-19 literature via the CORD-19 dataset.

def predict_lstm(text):
    seq = tokenizer.texts_to_sequences([text])
    seq = pad_sequences(seq, maxlen=max_len, padding="post")
    pred = model.predict(seq)[0]                # shape: (max_len, n_tags)

    tags_pred = [list(tag2idx.keys())[i] for i in pred.argmax(axis=-1)]
    return list(zip(text.split(), tags_pred))

def predict_bert(text):
    enc = tokenizer(text.split(),
                    is_split_into_words=True,
                    return_tensors="pt",
                    truncation=True,
                    max_length=max_len,
                    padding="max_length")
    out = model(**enc)
    preds = torch.argmax(out.logits, dim=2)[0]
    word_ids = enc.word_ids()
    result = []
    for idx, word_idx in enumerate(word_ids):
        if word_idx is None: continue
        tag = list(tag2idx.keys())[preds[idx].item()]
        result.append((text.split()[word_idx], tag))
    return result

Running either predict_lstm("I will travel to Paris tomorrow") or predict_bert(...) outputs token/tag pairs such as:

[('I', 'O'), ('will', 'O'), ('travel', 'O'), ('to', 'O'),
 ('Paris', 'B-LOC'), ('tomorrow', 'B-DATE')]

Summary

• The microsoft/AI-For-Beginners repository provides complete NER implementations in lessons/5-NLP/19-NER, comparing Bi-LSTM and BERT architectures side-by-side.
• The LSTM model uses TensorFlow layers including Bidirectional(LSTM) and TimeDistributed(Dense) for token-wise softmax predictions on the BC5CDR dataset.
• The BERT implementation leverages HuggingFace's AutoModelForTokenClassification with bert-base-cased for transfer learning on BIO-encoded data.
• Both examples process CSV-formatted data using the BIO tagging scheme, with extension suggestions for COVID-19 literature (CORD-19) in the lab materials.

Frequently Asked Questions

What is the BIO tagging scheme used in these NER examples?

BIO (Beginning-Inside-Outside) is a standard format for NER where B- prefixes mark the first token of an entity, I- prefixes mark subsequent tokens within the same entity, and O marks non-entity tokens. The implementation uses this scheme to handle multi-token entities like "New York" or "December 2023" according to the ner_dataset.csv structure.

How does the BERT model handle subword tokenization in NER?

The implementation aligns labels by using word_ids() from the tokenizer output, assigning the original word's label to the first subword and -100 (ignore index) to subsequent subwords and special tokens. This ensures the cross-entropy loss only computes gradients on relevant token predictions while maintaining the original entity boundaries.

Can I use these models on domain-specific text like medical or legal documents?

Yes. The lab/README.md specifically suggests applying the trained models to the CORD-19 dataset or other domain corpora by replacing the input CSV file. Since both the LSTM and BERT pipelines accept custom tokenized inputs with BIO tags, you can fine-tune on specialized datasets like BC5CDR for chemical-disease relations or adapt them to legal contract analysis.

What are the key differences between the LSTM and BERT approaches in these examples?

The LSTM implementation in NER-TF.ipynb trains embeddings from scratch using tensorflow.keras layers, requiring more epochs but offering an interpretable architecture for educational purposes. The BERT implementation uses pre-trained bert-base-cased weights via HuggingFace, achieving superior accuracy with fewer epochs through transfer learning but requiring careful alignment of subword tokens with BIO labels using the encode() function.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}