Text Representation in NLP: Best Practices for Tokenization, Vectorization, and Embeddings

Effective text representation in NLP requires matching the technique to your task complexity—using Bag-of-Words or TF-IDF for simple classification, n-grams for phrase patterns, and dense embeddings for semantic similarity—while TensorFlow's TextVectorization layer provides a unified API for tokenization, vocabulary management, and end-to-end model training.

Text representation transforms raw strings into numerical formats that machine learning algorithms can process, serving as the critical foundation for every NLP pipeline. The microsoft/AI-For-Beginners repository demonstrates production-ready implementations of classical sparse representations and modern dense embeddings, providing concrete guidance on when to use each approach. This article distills the best practices for text representation in NLP based on the source code in lessons/5-NLP/13-TextRep/TextRepresentationTF.ipynb and related curriculum modules.

Tokenization and Vocabulary Building

Every NLP pipeline begins with tokenization, the process of splitting text into discrete units (tokens) and mapping them to integer indices. According to the AI-For-Beginners source code, you should always limit vocabulary size to prevent memory bloat and overfitting.

In lessons/5-NLP/13-TextRep/TextRepresentationTF.ipynb (lines 20-22), the vectorizer is created with a constrained vocabulary:

import tensorflow as tf
from tensorflow import keras

vocab_size = 50000
vectorizer = keras.layers.experimental.preprocessing.TextVectorization(
    max_tokens=vocab_size,
    output_mode='count'
)

# Fit on a sample of the dataset to build vocabulary

vectorizer.adapt(ds_train.take(500).map(lambda x: x['title']+' '+x['description']))

Key implementation details:

• Use max_tokens (e.g., 50,000) to retain only the most frequent words
• Call adapt() on a representative sample of your corpus to build the vocabulary
• Process both title and description fields by concatenating them before vectorization

Bag-of-Words (BoW) Representation

Bag-of-Words is appropriate for baseline classification tasks where word order is irrelevant and you need a simple, interpretable feature space. The curriculum demonstrates two approaches: manual implementation with one-hot encoding and automatic vectorization.

Manual BoW with One-Hot Encoding

As shown in lines 50-52 of TextRepresentationTF.ipynb, you can manually create BoW vectors by summing one-hot encoded tokens:

# Manual BoW: sum one-hot vectors

token_ids = vectorizer(text)
bow_vector = tf.reduce_sum(tf.one_hot(token_ids, vocab_size), axis=1)

Automatic Count Vectorization

For production code, use TextVectorization with output_mode='count' (lines 93-95):

vectorizer = keras.layers.experimental.preprocessing.TextVectorization(
    max_tokens=vocab_size,
    output_mode='count'  # Automatically produces BoW counts

)

N-grams for Multi-Word Patterns

When phrases carry distinct meaning (e.g., "hot dog" versus "hot" and "dog" separately), n-grams extend your vocabulary to include token combinations. The notebook demonstrates bigram generation (lines 36-44):

# Generate bigrams (n=2)

bigram_vectorizer = keras.layers.experimental.preprocessing.TextVectorization(
    ngrams=2,
    max_tokens=vocab_size,
    output_mode='count'
)
bigram_vectorizer.adapt(text_sample)
vocab = bigram_vectorizer.get_vocabulary()

Best practice: Limit n-gram range to (1, 2) or (1, 3) to avoid an explosion in feature dimensionality while capturing essential multi-word expressions.

TF-IDF Weighting

Term Frequency-Inverse Document Frequency (TF-IDF) down-weights common words (like "the" and "and") while emphasizing rare, informative terms. The curriculum shows implementations in both Scikit-Learn and TensorFlow.

Scikit-Learn Implementation

From TextRepresentationTF.ipynb (lines 45-51):

from sklearn.feature_extraction.text import TfidfVectorizer

corpus = [
    'I like hot dogs.',
    'The dog ran fast.',
    'Its hot outside.'
]

tfidf = TfidfVectorizer(ngram_range=(1, 2))
X = tfidf.fit_transform(corpus)

# Transform new sentences

new_vec = tfidf.transform(['My dog likes hot dogs on a hot day.']).toarray()

TensorFlow TF-IDF

TensorFlow's TextVectorization supports TF-IDF natively (lines 85-87):

tfidf_vectorizer = keras.layers.experimental.preprocessing.TextVectorization(
    max_tokens=vocab_size,
    output_mode='tf-idf'
)
tfidf_vectorizer.adapt(training_data)

Dense Embeddings for Semantic Tasks

While sparse representations (BoW, TF-IDF) create high-dimensional vectors with millions of features, dense embeddings map tokens to low-dimensional continuous vectors that capture semantic relationships. The AI-For-Beginners curriculum introduces embeddings in lessons/5-NLP/14-Embeddings/EmbeddingsTF.ipynb as the preferred representation for:

• Semantic similarity tasks
• Deep learning models (RNNs, Transformers)
• Memory-constrained environments

Embeddings reduce dimensionality from hundreds of thousands to a few hundred dimensions while preserving word meaning through vector arithmetic.

End-to-End Text Vectorization

For simplified pipelines that train tokenization jointly with model parameters, embed the TextVectorization layer directly into your Keras model. This approach, demonstrated in lines 80-92 of TextRepresentationTF.ipynb, allows the model to process raw strings without pre-processing:

# Define input as raw strings

input_text = keras.Input(shape=(), dtype=tf.string)

# Vectorization layer inside the model

x = vectorizer(input_text)  # Converts to token IDs

x = tf.reduce_sum(tf.one_hot(x, vocab_size), axis=1)  # BoW representation

output = keras.layers.Dense(4, activation='softmax')(x)

model = keras.Model(inputs=input_text, outputs=output)
model.compile(
    loss='sparse_categorical_crossentropy',
    optimizer='adam',
    metrics=['accuracy']
)

# Train directly on string data

model.fit(
    ds_train.map(lambda x: (x['title']+' '+x['description'], x['label'])).batch(128),
    validation_data=ds_test.map(lambda x: (x['title']+' '+x['description'], x['label'])).batch(128)
)

Architectural Best Practices

Based on the source code analysis of microsoft/AI-For-Beginners, follow these principles when implementing text representation in NLP:

• Start with constrained vocabularies: Set max_tokens between 10,000 and 50,000 to balance coverage and memory usage
• Match representation to task: Use BoW/TF-IDF for shallow classifiers and logistic regression; use embeddings for neural networks requiring semantic understanding
• Combine n-grams with embeddings: When phrase patterns matter but memory is limited, use limited n-gram vocabularies alongside dense embeddings
• Leverage adapt() sampling: Build vocabulary on a statistically significant sample (e.g., 500+ documents) to ensure coverage without processing the entire corpus
• Prefer end-to-end layers: Include TextVectorization as a model layer for deployment simplicity and joint optimization

Summary

• Tokenization with limited vocabulary size (max_tokens) is the essential first step in any NLP pipeline
• Bag-of-Words provides a simple baseline for classification tasks where word order is irrelevant
• N-grams capture multi-word expressions but increase dimensionality; use with ngram_range=(1, 2) for optimal results
• TF-IDF improves upon raw counts by weighting terms by their document frequency, available in both Scikit-Learn and TensorFlow
• Dense embeddings are required for semantic tasks and deep learning models, dramatically reducing dimensionality while preserving meaning
• End-to-end vectorization simplifies deployment by including the TextVectorization layer within the Keras model graph

Frequently Asked Questions

What is the difference between BoW and TF-IDF in text representation?

Bag-of-Words counts raw token frequencies, treating common words like "the" and "is" as equally important as rare, informative terms. TF-IDF multiplies term frequency by the inverse document frequency (logarithm of total documents divided by documents containing the term), thereby down-weighting ubiquitous words and highlighting distinctive vocabulary. According to the TextRepresentationTF.ipynb source code, TF-IDF is preferred when you want to emphasize content-specific words over stop-words.

When should I use n-grams instead of unigrams?

Use n-grams (bigrams or trigrams) when multi-word expressions carry semantic meaning that differs from their constituent parts, such as "hot dog," "machine learning," or "New York." The microsoft/AI-For-Beginners notebook demonstrates that setting ngram_range=(1, 2) in Scikit-Learn or ngrams=2 in TensorFlow captures these patterns while controlling vocabulary explosion. However, higher-order n-grams significantly increase memory usage and should be paired with aggressive max_tokens limits.

How do I choose between sparse vectors and dense embeddings?

Choose sparse representations (BoW, TF-IDF) for simple classification tasks with smaller datasets where interpretability matters and word order is unimportant. Choose dense embeddings (Word2Vec, GloVe, or trainable layers) for tasks requiring semantic similarity, contextual understanding, or when feeding text into deep architectures like RNNs or Transformers. The curriculum in lessons/5-NLP/14-Embeddings/EmbeddingsTF.ipynb specifically recommends embeddings for downstream tasks that benefit from semantic relationships.

Can I use TextVectorization in production models?

Yes, TensorFlow's TextVectorization layer is designed for production deployment. By including the layer inside your Keras model (as shown in TextRepresentationTF.ipynb lines 80-92), the model accepts raw strings as input and handles tokenization internally. This eliminates preprocessing discrepancies between training and inference and allows the vectorization parameters to be saved with the model weights. Ensure you call adapt() during training or load pre-computed vocabulary weights when deploying.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}