# Named Entity Recognition (NER) Implementation Examples: LSTM and BERT from Scratch > Learn Named Entity Recognition NER implementation with LSTM and BERT from scratch. Explore TensorFlow and HuggingFace examples for token-level classification on biomedical datasets. - Repository: [Microsoft/AI-For-Beginners](https://github.com/microsoft/AI-For-Beginners) - Tags: tutorial - Published: 2026-08-02 --- **The microsoft/AI-For-Beginners curriculum provides complete, runnable Named Entity Recognition (NER) implementation examples using both a classic Bi-LSTM architecture in TensorFlow and a fine-tuned BERT transformer via HuggingFace, demonstrating token-level classification on BIO-encoded biomedical datasets.** The repository's lesson 19 located at `lessons/5-NLP/19-NER` treats NER as a supervised token-level classification task, where each word receives a label such as `B-DATE`, `I-LOC`, or `O`. These progressive examples guide learners through the full ML lifecycle—from raw text preprocessing to production-ready inference—using the BC5CDR corpus and extensible CSV formats. ## Understanding the NER Pipeline Architecture The curriculum organizes NER as a comparative study between traditional recurrent networks and modern transformers. Both approaches frame the task as **token-level classification**, assigning BIO (Beginning-Inside-Outside) tags to identify entity boundaries for person names, dates, locations, and chemical formulas. The [`README.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/README.md) in `lessons/5-NLP/19-NER` outlines the problem definition and highlights why NER serves as a critical preprocessing step for downstream conversational agents and information extraction systems. ## LSTM-Based NER Implementation with TensorFlow Located in `lessons/5-NLP/19-NER/NER-TF.ipynb`, the LSTM implementation demonstrates how to build a sequence labeling model from scratch without relying on pre-trained embeddings. ### Loading the Dataset The examples use either the public **BC5CDR** biomedical corpus or a simplified CSV toy dataset (`ner_dataset.csv`) with pre-tokenized text and pipe-delimited BIO tags. ```python import pandas as pd # The CSV file is shipped with the lesson (ner_dataset.csv) df = pd.read_csv("lessons/5-NLP/19-NER/ner_dataset.csv") sentences = df["sentence"].tolist() tags = df["tags"].tolist() # BIO-encoded strings ``` ### Building the Bi-LSTM Model The TensorFlow pipeline constructs an embedding layer initialized randomly, followed by a bidirectional LSTM and time-distributed dense layer for per-token softmax predictions. ```python import tensorflow as tf from tensorflow.keras.preprocessing.sequence import pad_sequences from tensorflow.keras.utils import to_categorical # ---- Tokenise ------------------------------------------------- tokenizer = tf.keras.preprocessing.text.Tokenizer(lower=False, oov_token="") tokenizer.fit_on_texts(sentences) X = tokenizer.texts_to_sequences(sentences) # ---- Encode tags ------------------------------------------------ tag2idx = {t: i for i, t in enumerate(sorted(set("|".join(tags).split("|"))))} y = [[tag2idx[t] for t in seq.split("|")] for seq in tags] # ---- Pad ------------------------------------------------------- max_len = max(len(s) for s in X) X = pad_sequences(X, maxlen=max_len, padding="post") y = pad_sequences(y, maxlen=max_len, padding="post") y = to_categorical(y, num_classes=len(tag2idx)) # ---- Model ------------------------------------------------------ model = tf.keras.Sequential([ tf.keras.layers.Embedding(input_dim=len(tokenizer.word_index)+1, output_dim=128, mask_zero=True), tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(64, return_sequences=True)), tf.keras.layers.TimeDistributed(tf.keras.layers.Dense(len(tag2idx), activation="softmax")) ]) model.compile(optimizer="adam", loss="categorical_crossentropy", metrics=["accuracy"]) model.fit(X, y, epochs=5, batch_size=32, validation_split=0.1) ``` ## BERT-Based NER with HuggingFace Transformers The second implementation swaps the recurrent layer for a **pre-trained transformer**, leveraging `AutoModelForTokenClassification` from the 🤗 Transformers library to demonstrate transfer learning on token classification tasks. ### Tokenization and Label Alignment Unlike the LSTM approach, BERT uses WordPiece tokenization that splits words into subwords. The implementation aligns original word-level BIO tags with the subword tokens, assigning `-100` to special tokens and subsequent subwords to ignore them during loss computation. ```python from transformers import AutoTokenizer, AutoModelForTokenClassification, Trainer, TrainingArguments import torch # ---- Load pre-trained model & tokenizer ------------------------- model_name = "bert-base-cased" tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoModelForTokenClassification.from_pretrained(model_name, num_labels=len(tag2idx)) ``` ### Fine-Tuning Configuration The training loop uses the `Trainer` API with `TrainingArguments` configured for 3 epochs, demonstrating how transfer learning achieves higher accuracy with fewer training steps than the LSTM baseline. ```python # ---- Encode a single sentence ---------------------------------- def encode(sentence, tags): enc = tokenizer(sentence.split(), is_split_into_words=True, return_offsets_mapping=True, padding="max_length", truncation=True, max_length=max_len) # Align label IDs with tokenised sub-words label_ids = [] word_ids = enc.word_ids() previous_word_idx = None for w in word_ids: if w is None: label_ids.append(-100) # ignore token (special tokens) elif w != previous_word_idx: label_ids.append(tag2idx[tags[w]]) # first sub-word gets the label else: label_ids.append(tag2idx[tags[w]] if label_all_tokens else -100) previous_word_idx = w enc["labels"] = label_ids return enc train_encodings = [encode(s, t.split("|")) for s, t in zip(sentences, tags)] # ---- Trainer ---------------------------------------------------- args = TrainingArguments( output_dir="./ner_bert", evaluation_strategy="epoch", per_device_train_batch_size=16, num_train_epochs=3, logging_dir="./logs", ) trainer = Trainer( model=model, args=args, train_dataset=train_encodings, ) trainer.train() ``` ## Inference and Evaluation Both models support inference on raw text strings, returning token-tag pairs that identify entities like locations (`B-LOC`) and dates (`B-DATE`). The [`lab/README.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/lab/README.md) suggests extensions for applying these models to COVID-19 literature via the CORD-19 dataset. ```python def predict_lstm(text): seq = tokenizer.texts_to_sequences([text]) seq = pad_sequences(seq, maxlen=max_len, padding="post") pred = model.predict(seq)[0] # shape: (max_len, n_tags) tags_pred = [list(tag2idx.keys())[i] for i in pred.argmax(axis=-1)] return list(zip(text.split(), tags_pred)) def predict_bert(text): enc = tokenizer(text.split(), is_split_into_words=True, return_tensors="pt", truncation=True, max_length=max_len, padding="max_length") out = model(**enc) preds = torch.argmax(out.logits, dim=2)[0] word_ids = enc.word_ids() result = [] for idx, word_idx in enumerate(word_ids): if word_idx is None: continue tag = list(tag2idx.keys())[preds[idx].item()] result.append((text.split()[word_idx], tag)) return result ``` Running either `predict_lstm("I will travel to Paris tomorrow")` or `predict_bert(...)` outputs token/tag pairs such as: ```text [('I', 'O'), ('will', 'O'), ('travel', 'O'), ('to', 'O'), ('Paris', 'B-LOC'), ('tomorrow', 'B-DATE')] ``` ## Summary - The `microsoft/AI-For-Beginners` repository provides complete NER implementations in `lessons/5-NLP/19-NER`, comparing Bi-LSTM and BERT architectures side-by-side. - The **LSTM model** uses TensorFlow layers including `Bidirectional(LSTM)` and `TimeDistributed(Dense)` for token-wise softmax predictions on the BC5CDR dataset. - The **BERT implementation** leverages HuggingFace's `AutoModelForTokenClassification` with `bert-base-cased` for transfer learning on BIO-encoded data. - Both examples process CSV-formatted data using the BIO tagging scheme, with extension suggestions for COVID-19 literature (CORD-19) in the lab materials. ## Frequently Asked Questions ### What is the BIO tagging scheme used in these NER examples? BIO (Beginning-Inside-Outside) is a standard format for NER where `B-` prefixes mark the first token of an entity, `I-` prefixes mark subsequent tokens within the same entity, and `O` marks non-entity tokens. The implementation uses this scheme to handle multi-token entities like "New York" or "December 2023" according to the `ner_dataset.csv` structure. ### How does the BERT model handle subword tokenization in NER? The implementation aligns labels by using `word_ids()` from the tokenizer output, assigning the original word's label to the first subword and `-100` (ignore index) to subsequent subwords and special tokens. This ensures the cross-entropy loss only computes gradients on relevant token predictions while maintaining the original entity boundaries. ### Can I use these models on domain-specific text like medical or legal documents? Yes. The [`lab/README.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/lab/README.md) specifically suggests applying the trained models to the CORD-19 dataset or other domain corpora by replacing the input CSV file. Since both the LSTM and BERT pipelines accept custom tokenized inputs with BIO tags, you can fine-tune on specialized datasets like BC5CDR for chemical-disease relations or adapt them to legal contract analysis. ### What are the key differences between the LSTM and BERT approaches in these examples? The **LSTM implementation** in `NER-TF.ipynb` trains embeddings from scratch using `tensorflow.keras` layers, requiring more epochs but offering an interpretable architecture for educational purposes. The **BERT implementation** uses pre-trained `bert-base-cased` weights via HuggingFace, achieving superior accuracy with fewer epochs through transfer learning but requiring careful alignment of subword tokens with BIO labels using the `encode()` function.