Named Entity Recognition with TensorFlow in the AI for Beginners NLP Lessons

The AI for Beginners curriculum provides a complete TensorFlow implementation of Named Entity Recognition (NER) using bidirectional LSTM layers and BIO tagging in lesson 5-NLP/19-NER, demonstrating end-to-end token classification from raw text to entity labels.

The microsoft/AI-For-Beginners repository offers a comprehensive 12-week curriculum covering fundamental AI concepts through practical implementations. The Named Entity Recognition lesson within the NLP track presents a complete TensorFlow pipeline that treats NER as a token-level classification problem, utilizing recurrent neural networks to predict entity tags for each word in a sequence.

Understanding the NER Task Structure

Named Entity Recognition identifies and classifies key information (entities) in text into predefined categories such as person names, organizations, locations, medical terms, or chemical compounds. The AI for Beginners approach frames this as a sequence labeling task where every token receives a classification label.

The implementation resides in lessons/5-NLP/19-NER/NER-TF.ipynb, which contains a runnable Jupyter notebook that processes the ner_dataset.csv file (available from Kaggle) and builds a complete deep learning pipeline from data ingestion to prediction.

The BIO Tagging Scheme

The lesson employs the BIO tagging scheme (Begin, Inside, Outside) to handle multi-token entities. This scheme assigns specific prefixes to indicate whether a token starts an entity, continues an entity, or falls outside any entity:

• B- prefix: Marks the beginning of a named entity (e.g., B-CHEM, B-DIS)
• I- prefix: Marks tokens inside a multi-word entity (e.g., I-CHEM, I-DIS)
• O tag: Indicates the token is not part of any entity

According to the lesson documentation in lessons/5-NLP/19-NER/README.md, this scheme is essential for distinguishing adjacent entities and handling variable-length entity spans correctly.

Data Preprocessing Pipeline

The preprocessing stage converts raw CSV data into integer sequences suitable for neural network training. The workflow involves loading data with pandas, building vocabulary mappings, and vectorizing sentences.

First, load the dataset and extract unique tags:

import pandas as pd
import numpy as np
from tensorflow import keras

# Load annotated corpus

df = pd.read_csv('ner_dataset.csv', encoding='unicode-escape')

# Build tag vocabulary (BIO scheme)

tags = df.Tag.unique()
tag2id = {t: i for i, t in enumerate(tags)}
id2tag = {i: t for t, i in tag2id.items()}

Next, construct the word vocabulary with a special <UNK> token for unknown words:

# Create word-to-ID mapping (lower-cased)

vocab = set(df['Word'].apply(lambda w: w.lower()))
word2id = {w: i + 1 for i, w in enumerate(vocab)}  # Reserve 0 for <UNK>

word2id['<UNK>'] = 0
id2word = {i: w for w, i in word2id.items()}

The notebook then groups rows into sentence-level lists and applies the vectorize() and tagify() functions to convert strings to integers:

# Group rows into sentences

X, Y = [], []
sentence_words, sentence_tags = [], []

for _, row in df[['Sentence #', 'Word', 'Tag']].itertuples():
    if pd.isna(row[0]):  # Continuation of current sentence

        sentence_words.append(row[1])
        sentence_tags.append(row[2])
    else:  # New sentence begins

        if sentence_words:
            X.append(sentence_words)
            Y.append(sentence_tags)
        sentence_words = [row[1]]
        sentence_tags = [row[2]]

# Vectorization functions (lines 74-79 in NER-TF.ipynb)

def vectorize(seq):
    return [word2id.get(w.lower(), 0) for w in seq]

def tagify(seq):
    return [tag2id[t] for t in seq]

X_ids = list(map(vectorize, X))
Y_ids = list(map(tagify, Y))

Finally, the code pads all sequences to the length of the longest sentence using keras.preprocessing.sequence.pad_sequences:

# Pad sequences to uniform length

X_padded = keras.preprocessing.sequence.pad_sequences(X_ids, padding='post')
Y_padded = keras.preprocessing.sequence.pad_sequences(Y_ids, padding='post')

Building the Bidirectional LSTM Model

The architecture defined in NER-TF.ipynb (lines 303-329) utilizes a bidirectional LSTM wrapped with TimeDistributed layers to maintain token-level predictions across the entire sequence length. This design allows the model to capture context from both past and future tokens when classifying each word.

num_tags = len(tags)

model = keras.Sequential([
    # Embedding layer handles input vocabulary

    keras.layers.Embedding(input_dim=len(word2id),
                           output_dim=128,
                           mask_zero=True),
    
    # Bidirectional LSTM processes sequences in both directions

    keras.layers.Bidirectional(
        keras.layers.LSTM(64, return_sequences=True)),
    
    # TimeDistributed applies Dense layer to every time step

    keras.layers.TimeDistributed(
        keras.layers.Dense(num_tags, activation='softmax'))
])

The mask_zero=True parameter ensures the model ignores padded positions during training. The return_sequences=True setting in the LSTM layer is critical for NER tasks because every token requires a prediction, not just the final token in the sequence.

The model compiles with sparse_categorical_crossentropy loss (since tags are integer-encoded rather than one-hot) and the Adam optimizer:

model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

Training and Inference

Training proceeds with standard mini-batch gradient descent using model.fit(). The notebook configures typical hyperparameters of 5-10 epochs with a batch size of 32:

model.fit(X_padded, Y_padded,
          epochs=5,
          batch_size=32,
          validation_split=0.1)

For inference on new text, the pipeline converts words to IDs, pads the sequence, and uses np.argmax to select the highest probability tag from the softmax output:

# Sample medical text

sample = ["Tricuspid", "valve", "regurgitation", "and", 
          "lithium", "carbonate", "toxicity"]

# Vectorize and pad

sample_ids = np.array([vectorize(sample)])
sample_padded = keras.preprocessing.sequence.pad_sequences(
    sample_ids, padding='post')

# Predict and decode

pred_ids = np.argmax(model.predict(sample_padded), axis=-1)[0]
predicted_tags = [id2tag[i] for i in pred_ids[:len(sample)]]

print(predicted_tags)

This implementation outputs BIO-tagged sequences that identify medical entities (diseases and chemicals) within the input text.

Summary

• The AI for Beginners NER lesson in lessons/5-NLP/19-NER/ provides a complete TensorFlow implementation using bidirectional LSTM architectures for token classification.
• The BIO tagging scheme (Begin, Inside, Outside) handles multi-token entities by marking start and continuation positions distinctly.
• Data preprocessing involves building word2id and tag2id mappings, vectorizing sentences with custom functions, and padding sequences to uniform length using keras.preprocessing.sequence.pad_sequences.
• The model architecture combines Embedding layers, Bidirectional LSTM cells with return_sequences=True, and TimeDistributed Dense layers with softmax activation to output per-token classifications.
• The notebook uses sparse_categorical_crossentropy loss and the Adam optimizer, training for 5 epochs with a batch size of 32 to achieve token-level accuracy.

Frequently Asked Questions

What is the BIO tagging scheme used in the TensorFlow NER lesson?

The BIO scheme is a labeling convention where B- prefixes mark the beginning of an entity, I- prefixes mark tokens inside multi-word entities, and O indicates tokens outside any entity. As implemented in lessons/5-NLP/19-NER/NER-TF.ipynb, this allows the model to distinguish between adjacent entities (e.g., distinguishing where one chemical compound ends and the next begins) and properly handle entities spanning multiple words.

How does the bidirectional LSTM architecture work for named entity recognition?

The bidirectional LSTM processes input sequences in both forward and backward directions, concatenating their hidden states to create context-aware representations. In the AI for Beginners implementation, the Bidirectional wrapper around the LSTM layer with return_sequences=True ensures every time step receives contextual information from both preceding and succeeding tokens. This is essential for NER because entity boundaries often depend on future context (e.g., knowing a word is a disease name only after seeing "syndrome" or "disease" later in the phrase).

What dataset is used in the AI for Beginners NER tutorial?

The lesson utilizes the ner_dataset.csv file from Kaggle, which contains annotated medical text where each row includes a sentence identifier, word, part-of-speech tag, and NER tag. The notebook loads this via pandas.read_csv('ner_dataset.csv', encoding='unicode-escape') and processes the BIO-tagged entities to build the vocabulary and training matrices used in the TensorFlow model.

How do I adapt this TensorFlow NER model for custom text domains?

To adapt the model, replace the ner_dataset.csv with your domain-specific annotated data while maintaining the same column structure (Sentence ID, Word, Tag). Update the word2id and tag2id dictionaries to reflect your vocabulary and entity types, then adjust the input_dim of the Embedding layer and the num_tags in the final Dense layer to match your new vocabulary size and tag set. The NER-TF.ipynb notebook structure remains valid for any language or domain requiring token-level sequence classification.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}