# Text Representation in NLP: Best Practices for Tokenization, Vectorization, and Embeddings

> Learn best practices for text representation in NLP including tokenization, vectorization, and embeddings. Match techniques to task complexity for optimal results with TensorFlow.

- Repository: [Microsoft/AI-For-Beginners](https://github.com/microsoft/AI-For-Beginners)
- Tags: best-practices
- Published: 2026-08-02

---

**Effective text representation in NLP requires matching the technique to your task complexity—using Bag-of-Words or TF-IDF for simple classification, n-grams for phrase patterns, and dense embeddings for semantic similarity—while TensorFlow's `TextVectorization` layer provides a unified API for tokenization, vocabulary management, and end-to-end model training.**

Text representation transforms raw strings into numerical formats that machine learning algorithms can process, serving as the critical foundation for every NLP pipeline. The microsoft/AI-For-Beginners repository demonstrates production-ready implementations of classical sparse representations and modern dense embeddings, providing concrete guidance on when to use each approach. This article distills the best practices for text representation in NLP based on the source code in `lessons/5-NLP/13-TextRep/TextRepresentationTF.ipynb` and related curriculum modules.

## Tokenization and Vocabulary Building

Every NLP pipeline begins with **tokenization**, the process of splitting text into discrete units (tokens) and mapping them to integer indices. According to the AI-For-Beginners source code, you should always limit vocabulary size to prevent memory bloat and overfitting.

In `lessons/5-NLP/13-TextRep/TextRepresentationTF.ipynb` (lines 20-22), the vectorizer is created with a constrained vocabulary:

```python
import tensorflow as tf
from tensorflow import keras

vocab_size = 50000
vectorizer = keras.layers.experimental.preprocessing.TextVectorization(
    max_tokens=vocab_size,
    output_mode='count'
)

# Fit on a sample of the dataset to build vocabulary

vectorizer.adapt(ds_train.take(500).map(lambda x: x['title']+' '+x['description']))

```

**Key implementation details:**
- Use **`max_tokens`** (e.g., 50,000) to retain only the most frequent words
- Call **`adapt()`** on a representative sample of your corpus to build the vocabulary
- Process both title and description fields by concatenating them before vectorization

## Bag-of-Words (BoW) Representation

**Bag-of-Words** is appropriate for baseline classification tasks where word order is irrelevant and you need a simple, interpretable feature space. The curriculum demonstrates two approaches: manual implementation with one-hot encoding and automatic vectorization.

### Manual BoW with One-Hot Encoding

As shown in lines 50-52 of `TextRepresentationTF.ipynb`, you can manually create BoW vectors by summing one-hot encoded tokens:

```python

# Manual BoW: sum one-hot vectors

token_ids = vectorizer(text)
bow_vector = tf.reduce_sum(tf.one_hot(token_ids, vocab_size), axis=1)

```

### Automatic Count Vectorization

For production code, use `TextVectorization` with `output_mode='count'` (lines 93-95):

```python
vectorizer = keras.layers.experimental.preprocessing.TextVectorization(
    max_tokens=vocab_size,
    output_mode='count'  # Automatically produces BoW counts

)

```

## N-grams for Multi-Word Patterns

When phrases carry distinct meaning (e.g., "hot dog" versus "hot" and "dog" separately), **n-grams** extend your vocabulary to include token combinations. The notebook demonstrates bigram generation (lines 36-44):

```python

# Generate bigrams (n=2)

bigram_vectorizer = keras.layers.experimental.preprocessing.TextVectorization(
    ngrams=2,
    max_tokens=vocab_size,
    output_mode='count'
)
bigram_vectorizer.adapt(text_sample)
vocab = bigram_vectorizer.get_vocabulary()

```

**Best practice:** Limit n-gram range to `(1, 2)` or `(1, 3)` to avoid an explosion in feature dimensionality while capturing essential multi-word expressions.

## TF-IDF Weighting

**Term Frequency-Inverse Document Frequency (TF-IDF)** down-weights common words (like "the" and "and") while emphasizing rare, informative terms. The curriculum shows implementations in both Scikit-Learn and TensorFlow.

### Scikit-Learn Implementation

From `TextRepresentationTF.ipynb` (lines 45-51):

```python
from sklearn.feature_extraction.text import TfidfVectorizer

corpus = [
    'I like hot dogs.',
    'The dog ran fast.',
    'Its hot outside.'
]

tfidf = TfidfVectorizer(ngram_range=(1, 2))
X = tfidf.fit_transform(corpus)

# Transform new sentences

new_vec = tfidf.transform(['My dog likes hot dogs on a hot day.']).toarray()

```

### TensorFlow TF-IDF

TensorFlow's `TextVectorization` supports TF-IDF natively (lines 85-87):

```python
tfidf_vectorizer = keras.layers.experimental.preprocessing.TextVectorization(
    max_tokens=vocab_size,
    output_mode='tf-idf'
)
tfidf_vectorizer.adapt(training_data)

```

## Dense Embeddings for Semantic Tasks

While sparse representations (BoW, TF-IDF) create high-dimensional vectors with millions of features, **dense embeddings** map tokens to low-dimensional continuous vectors that capture semantic relationships. The AI-For-Beginners curriculum introduces embeddings in `lessons/5-NLP/14-Embeddings/EmbeddingsTF.ipynb` as the preferred representation for:
- Semantic similarity tasks
- Deep learning models (RNNs, Transformers)
- Memory-constrained environments

Embeddings reduce dimensionality from hundreds of thousands to a few hundred dimensions while preserving word meaning through vector arithmetic.

## End-to-End Text Vectorization

For simplified pipelines that train tokenization jointly with model parameters, embed the `TextVectorization` layer directly into your Keras model. This approach, demonstrated in lines 80-92 of `TextRepresentationTF.ipynb`, allows the model to process raw strings without pre-processing:

```python

# Define input as raw strings

input_text = keras.Input(shape=(), dtype=tf.string)

# Vectorization layer inside the model

x = vectorizer(input_text)  # Converts to token IDs

x = tf.reduce_sum(tf.one_hot(x, vocab_size), axis=1)  # BoW representation

output = keras.layers.Dense(4, activation='softmax')(x)

model = keras.Model(inputs=input_text, outputs=output)
model.compile(
    loss='sparse_categorical_crossentropy',
    optimizer='adam',
    metrics=['accuracy']
)

# Train directly on string data

model.fit(
    ds_train.map(lambda x: (x['title']+' '+x['description'], x['label'])).batch(128),
    validation_data=ds_test.map(lambda x: (x['title']+' '+x['description'], x['label'])).batch(128)
)

```

## Architectural Best Practices

Based on the source code analysis of `microsoft/AI-For-Beginners`, follow these principles when implementing text representation in NLP:

- **Start with constrained vocabularies:** Set `max_tokens` between 10,000 and 50,000 to balance coverage and memory usage
- **Match representation to task:** Use BoW/TF-IDF for shallow classifiers and logistic regression; use embeddings for neural networks requiring semantic understanding
- **Combine n-grams with embeddings:** When phrase patterns matter but memory is limited, use limited n-gram vocabularies alongside dense embeddings
- **Leverage `adapt()` sampling:** Build vocabulary on a statistically significant sample (e.g., 500+ documents) to ensure coverage without processing the entire corpus
- **Prefer end-to-end layers:** Include `TextVectorization` as a model layer for deployment simplicity and joint optimization

## Summary

- **Tokenization** with limited vocabulary size (`max_tokens`) is the essential first step in any NLP pipeline
- **Bag-of-Words** provides a simple baseline for classification tasks where word order is irrelevant
- **N-grams** capture multi-word expressions but increase dimensionality; use with `ngram_range=(1, 2)` for optimal results
- **TF-IDF** improves upon raw counts by weighting terms by their document frequency, available in both Scikit-Learn and TensorFlow
- **Dense embeddings** are required for semantic tasks and deep learning models, dramatically reducing dimensionality while preserving meaning
- **End-to-end vectorization** simplifies deployment by including the `TextVectorization` layer within the Keras model graph

## Frequently Asked Questions

### What is the difference between BoW and TF-IDF in text representation?

Bag-of-Words counts raw token frequencies, treating common words like "the" and "is" as equally important as rare, informative terms. TF-IDF multiplies term frequency by the inverse document frequency (logarithm of total documents divided by documents containing the term), thereby down-weighting ubiquitous words and highlighting distinctive vocabulary. According to the `TextRepresentationTF.ipynb` source code, TF-IDF is preferred when you want to emphasize content-specific words over stop-words.

### When should I use n-grams instead of unigrams?

Use n-grams (bigrams or trigrams) when multi-word expressions carry semantic meaning that differs from their constituent parts, such as "hot dog," "machine learning," or "New York." The microsoft/AI-For-Beginners notebook demonstrates that setting `ngram_range=(1, 2)` in Scikit-Learn or `ngrams=2` in TensorFlow captures these patterns while controlling vocabulary explosion. However, higher-order n-grams significantly increase memory usage and should be paired with aggressive `max_tokens` limits.

### How do I choose between sparse vectors and dense embeddings?

Choose **sparse representations** (BoW, TF-IDF) for simple classification tasks with smaller datasets where interpretability matters and word order is unimportant. Choose **dense embeddings** (Word2Vec, GloVe, or trainable layers) for tasks requiring semantic similarity, contextual understanding, or when feeding text into deep architectures like RNNs or Transformers. The curriculum in `lessons/5-NLP/14-Embeddings/EmbeddingsTF.ipynb` specifically recommends embeddings for downstream tasks that benefit from semantic relationships.

### Can I use TextVectorization in production models?

Yes, TensorFlow's `TextVectorization` layer is designed for production deployment. By including the layer inside your Keras model (as shown in `TextRepresentationTF.ipynb` lines 80-92), the model accepts raw strings as input and handles tokenization internally. This eliminates preprocessing discrepancies between training and inference and allows the vectorization parameters to be saved with the model weights. Ensure you call `adapt()` during training or load pre-computed vocabulary weights when deploying.