# Named Entity Recognition with TensorFlow in the AI for Beginners NLP Lessons

> Learn Named Entity Recognition with TensorFlow using AI for Beginners NLP lessons. Explore bidirectional LSTMs and BIO tagging for token classification from raw text.

- Repository: [Microsoft/AI-For-Beginners](https://github.com/microsoft/AI-For-Beginners)
- Tags: tutorial
- Published: 2026-08-01

---

**The AI for Beginners curriculum provides a complete TensorFlow implementation of Named Entity Recognition (NER) using bidirectional LSTM layers and BIO tagging in lesson `5-NLP/19-NER`, demonstrating end-to-end token classification from raw text to entity labels.**

The microsoft/AI-For-Beginners repository offers a comprehensive 12-week curriculum covering fundamental AI concepts through practical implementations. The Named Entity Recognition lesson within the NLP track presents a complete TensorFlow pipeline that treats NER as a token-level classification problem, utilizing recurrent neural networks to predict entity tags for each word in a sequence.

## Understanding the NER Task Structure

Named Entity Recognition identifies and classifies key information (entities) in text into predefined categories such as person names, organizations, locations, medical terms, or chemical compounds. The AI for Beginners approach frames this as a **sequence labeling task** where every token receives a classification label.

The implementation resides in `lessons/5-NLP/19-NER/NER-TF.ipynb`, which contains a runnable Jupyter notebook that processes the `ner_dataset.csv` file (available from Kaggle) and builds a complete deep learning pipeline from data ingestion to prediction.

## The BIO Tagging Scheme

The lesson employs the **BIO tagging scheme** (Begin, Inside, Outside) to handle multi-token entities. This scheme assigns specific prefixes to indicate whether a token starts an entity, continues an entity, or falls outside any entity:

- **B-** prefix: Marks the beginning of a named entity (e.g., `B-CHEM`, `B-DIS`)
- **I-** prefix: Marks tokens inside a multi-word entity (e.g., `I-CHEM`, `I-DIS`)
- **O** tag: Indicates the token is not part of any entity

According to the lesson documentation in [`lessons/5-NLP/19-NER/README.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/19-NER/README.md), this scheme is essential for distinguishing adjacent entities and handling variable-length entity spans correctly.

## Data Preprocessing Pipeline

The preprocessing stage converts raw CSV data into integer sequences suitable for neural network training. The workflow involves loading data with pandas, building vocabulary mappings, and vectorizing sentences.

First, load the dataset and extract unique tags:

```python
import pandas as pd
import numpy as np
from tensorflow import keras

# Load annotated corpus

df = pd.read_csv('ner_dataset.csv', encoding='unicode-escape')

# Build tag vocabulary (BIO scheme)

tags = df.Tag.unique()
tag2id = {t: i for i, t in enumerate(tags)}
id2tag = {i: t for t, i in tag2id.items()}

```

Next, construct the word vocabulary with a special `<UNK>` token for unknown words:

```python

# Create word-to-ID mapping (lower-cased)

vocab = set(df['Word'].apply(lambda w: w.lower()))
word2id = {w: i + 1 for i, w in enumerate(vocab)}  # Reserve 0 for <UNK>

word2id['<UNK>'] = 0
id2word = {i: w for w, i in word2id.items()}

```

The notebook then groups rows into sentence-level lists and applies the `vectorize()` and `tagify()` functions to convert strings to integers:

```python

# Group rows into sentences

X, Y = [], []
sentence_words, sentence_tags = [], []

for _, row in df[['Sentence #', 'Word', 'Tag']].itertuples():
    if pd.isna(row[0]):  # Continuation of current sentence

        sentence_words.append(row[1])
        sentence_tags.append(row[2])
    else:  # New sentence begins

        if sentence_words:
            X.append(sentence_words)
            Y.append(sentence_tags)
        sentence_words = [row[1]]
        sentence_tags = [row[2]]

# Vectorization functions (lines 74-79 in NER-TF.ipynb)

def vectorize(seq):
    return [word2id.get(w.lower(), 0) for w in seq]

def tagify(seq):
    return [tag2id[t] for t in seq]

X_ids = list(map(vectorize, X))
Y_ids = list(map(tagify, Y))

```

Finally, the code pads all sequences to the length of the longest sentence using `keras.preprocessing.sequence.pad_sequences`:

```python

# Pad sequences to uniform length

X_padded = keras.preprocessing.sequence.pad_sequences(X_ids, padding='post')
Y_padded = keras.preprocessing.sequence.pad_sequences(Y_ids, padding='post')

```

## Building the Bidirectional LSTM Model

The architecture defined in `NER-TF.ipynb` (lines 303-329) utilizes a **bidirectional LSTM** wrapped with `TimeDistributed` layers to maintain token-level predictions across the entire sequence length. This design allows the model to capture context from both past and future tokens when classifying each word.

```python
num_tags = len(tags)

model = keras.Sequential([
    # Embedding layer handles input vocabulary

    keras.layers.Embedding(input_dim=len(word2id),
                           output_dim=128,
                           mask_zero=True),
    
    # Bidirectional LSTM processes sequences in both directions

    keras.layers.Bidirectional(
        keras.layers.LSTM(64, return_sequences=True)),
    
    # TimeDistributed applies Dense layer to every time step

    keras.layers.TimeDistributed(
        keras.layers.Dense(num_tags, activation='softmax'))
])

```

The `mask_zero=True` parameter ensures the model ignores padded positions during training. The `return_sequences=True` setting in the LSTM layer is critical for NER tasks because every token requires a prediction, not just the final token in the sequence.

The model compiles with `sparse_categorical_crossentropy` loss (since tags are integer-encoded rather than one-hot) and the `Adam` optimizer:

```python
model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

```

## Training and Inference

Training proceeds with standard mini-batch gradient descent using `model.fit()`. The notebook configures typical hyperparameters of 5-10 epochs with a batch size of 32:

```python
model.fit(X_padded, Y_padded,
          epochs=5,
          batch_size=32,
          validation_split=0.1)

```

For inference on new text, the pipeline converts words to IDs, pads the sequence, and uses `np.argmax` to select the highest probability tag from the softmax output:

```python

# Sample medical text

sample = ["Tricuspid", "valve", "regurgitation", "and", 
          "lithium", "carbonate", "toxicity"]

# Vectorize and pad

sample_ids = np.array([vectorize(sample)])
sample_padded = keras.preprocessing.sequence.pad_sequences(
    sample_ids, padding='post')

# Predict and decode

pred_ids = np.argmax(model.predict(sample_padded), axis=-1)[0]
predicted_tags = [id2tag[i] for i in pred_ids[:len(sample)]]

print(predicted_tags)

```

This implementation outputs BIO-tagged sequences that identify medical entities (diseases and chemicals) within the input text.

## Summary

- The **AI for Beginners** NER lesson in `lessons/5-NLP/19-NER/` provides a complete TensorFlow implementation using bidirectional LSTM architectures for token classification.
- The **BIO tagging scheme** (Begin, Inside, Outside) handles multi-token entities by marking start and continuation positions distinctly.
- Data preprocessing involves building `word2id` and `tag2id` mappings, vectorizing sentences with custom functions, and padding sequences to uniform length using `keras.preprocessing.sequence.pad_sequences`.
- The model architecture combines **Embedding** layers, **Bidirectional LSTM** cells with `return_sequences=True`, and **TimeDistributed Dense** layers with softmax activation to output per-token classifications.
- The notebook uses `sparse_categorical_crossentropy` loss and the **Adam** optimizer, training for 5 epochs with a batch size of 32 to achieve token-level accuracy.

## Frequently Asked Questions

### What is the BIO tagging scheme used in the TensorFlow NER lesson?

The BIO scheme is a labeling convention where **B-** prefixes mark the beginning of an entity, **I-** prefixes mark tokens inside multi-word entities, and **O** indicates tokens outside any entity. As implemented in `lessons/5-NLP/19-NER/NER-TF.ipynb`, this allows the model to distinguish between adjacent entities (e.g., distinguishing where one chemical compound ends and the next begins) and properly handle entities spanning multiple words.

### How does the bidirectional LSTM architecture work for named entity recognition?

The bidirectional LSTM processes input sequences in both forward and backward directions, concatenating their hidden states to create context-aware representations. In the AI for Beginners implementation, the `Bidirectional` wrapper around the LSTM layer with `return_sequences=True` ensures every time step receives contextual information from both preceding and succeeding tokens. This is essential for NER because entity boundaries often depend on future context (e.g., knowing a word is a disease name only after seeing "syndrome" or "disease" later in the phrase).

### What dataset is used in the AI for Beginners NER tutorial?

The lesson utilizes the `ner_dataset.csv` file from Kaggle, which contains annotated medical text where each row includes a sentence identifier, word, part-of-speech tag, and NER tag. The notebook loads this via `pandas.read_csv('ner_dataset.csv', encoding='unicode-escape')` and processes the BIO-tagged entities to build the vocabulary and training matrices used in the TensorFlow model.

### How do I adapt this TensorFlow NER model for custom text domains?

To adapt the model, replace the `ner_dataset.csv` with your domain-specific annotated data while maintaining the same column structure (Sentence ID, Word, Tag). Update the `word2id` and `tag2id` dictionaries to reflect your vocabulary and entity types, then adjust the `input_dim` of the Embedding layer and the `num_tags` in the final Dense layer to match your new vocabulary size and tag set. The `NER-TF.ipynb` notebook structure remains valid for any language or domain requiring token-level sequence classification.