Text Representation Methods: Bag-of-Words vs TF-IDF in the AI-For-Beginners Repository

Bag-of-Words counts raw word frequencies while TF-IDF weights terms by their importance across documents, with both methods implemented in the Microsoft AI-For-Beginners repository using scikit-learn's CountVectorizer and TfidfVectorizer.

The AI-For-Beginners repository provides a structured curriculum for learning foundational AI concepts, with specific text representation methods covered in the Natural Language Processing (NLP) section. These lessons demonstrate how to convert unstructured text into numerical vectors that machine learning models can process. Understanding the differences between Bag-of-Words and TF-IDF is essential for building effective NLP pipelines, as implemented in the notebooks located at lessons/5-NLP/13-TextRep/.

Bag-of-Words Implementation

The Bag-of-Words (BoW) approach creates a vocabulary of unique tokens from the corpus and represents each document as a vector of word counts. In the AI-For-Beginners curriculum, this method is demonstrated in lessons/5-NLP/13-TextRep/TextRepresentationPyTorch.ipynb using scikit-learn's CountVectorizer.

The notebook follows a standard pipeline:

1. Import CountVectorizer from sklearn.feature_extraction.text
2. Define a sample corpus of raw sentences
3. Instantiate the vectorizer and fit it to the data
4. Transform the corpus into a sparse count matrix

from sklearn.feature_extraction.text import CountVectorizer

sentences = [
    "the cat sat on the mat",
    "the dog barked loudly",
    "the cat and the dog played"
]

vectorizer = CountVectorizer()
bow_matrix = vectorizer.fit_transform(sentences)

print("Vocabulary:", vectorizer.vocabulary_)
print("BoW matrix:\n", bow_matrix.toarray())

The resulting matrix contains integer values representing how many times each word appears in each document. The vocabulary_ attribute maps each unique word to a specific column index, creating a high-dimensional but interpretable representation.

TF-IDF Implementation

The TF-IDF (Term Frequency-Inverse Document Frequency) method extends BoW by weighting terms based on their discriminative power across the corpus. Common words that appear in many documents receive lower weights, while rare, distinctive terms receive higher weights. This implementation resides in lessons/5-NLP/13-TextRep/TextRepresentationTF.ipynb using TfidfVectorizer.

from sklearn.feature_extraction.text import TfidfVectorizer

sentences = [
    "the cat sat on the mat",
    "the dog barked loudly",
    "the cat and the dog played"
]

vectorizer = TfidfVectorizer()
tfidf_matrix = vectorizer.fit_transform(sentences)

print("Vocabulary:", vectorizer.vocabulary_)
print("TF-IDF matrix:\n", tfidf_matrix.toarray())

Unlike the BoW output, the TF-IDF matrix contains floating-point values. The algorithm calculates term frequency (how often a word appears in a specific document) divided by inverse document frequency (a measure of how common the word is across all documents). This mitigates the "stop-word" problem where high-frequency words like "the" and "and" would otherwise dominate the representation.

Key Differences Between Bag-of-Words and TF-IDF

When comparing these text representation methods in the context of the AI-For-Beginners examples, several critical distinctions emerge:

• Counting vs Weighting: BoW uses raw integer counts, while TF-IDF produces normalized floating-point weights that reflect term importance
• Stop-word handling: TF-IDF automatically downweights common words across the corpus, whereas BoW treats all tokens equally regardless of their discriminative value
• Matrix sparsity: Both methods generate sparse matrices, but TF-IDF typically produces better-separated document vectors for downstream classification tasks
• Computational complexity: TF-IDF requires additional passes over the corpus to calculate document frequencies, making it slightly more computationally expensive than simple counting

Both notebooks in lessons/5-NLP/13-TextRep/ visualize these matrices using Pandas DataFrames, allowing learners to inspect exactly how each document transforms into a vector space representation.

Summary

• Bag-of-Words in TextRepresentationPyTorch.ipynb converts text to count vectors using CountVectorizer, treating every word occurrence equally regardless of corpus-wide frequency.
• TF-IDF in TextRepresentationTF.ipynb applies TfidfVectorizer to weight terms by their importance, reducing the impact of common words and highlighting distinctive vocabulary.
• Both methods require the same scikit-learn import pattern and follow an identical fit_transform workflow, making it easy to experiment with either approach.
• The AI-For-Beginners repository isolates these concepts in lessons/5-NLP/13-TextRep/ with bilingual support available in the translations/ directory.

Frequently Asked Questions

What is the main advantage of TF-IDF over Bag-of-Words?

TF-IDF weights terms by their importance across the corpus, which reduces the influence of common words like "the" and "is" while highlighting distinctive terms that better differentiate documents. This typically results in superior performance for document classification and information retrieval tasks compared to the uniform counting approach of Bag-of-Words.

How do I run the text representation notebooks from the AI-For-Beginners repository?

Clone the repository and navigate to lessons/5-NLP/13-TextRep/ where you will find TextRepresentationPyTorch.ipynb for Bag-of-Words and TextRepresentationTF.ipynb for TF-IDF. Ensure you have the dependencies listed in environment.yml installed, particularly scikit-learn, numpy, and pandas, then execute the notebooks in Jupyter or VS Code.

Why does the Bag-of-Words matrix contain only integers while TF-IDF contains decimals?

The Bag-of-Words model implemented via CountVectorizer counts raw word occurrences, resulting in integer values representing absolute frequencies. TF-IDF uses TfidfVectorizer to compute a ratio of term frequency to inverse document frequency, producing normalized floating-point weights that indicate relative importance rather than absolute counts.

Can I use these text representation methods with deep learning models?

Yes, though the AI-For-Beginners notebooks demonstrate these methods with traditional machine learning, both Bag-of-Words and TF-IDF vectors serve as valid input features for neural networks. However, modern deep learning typically uses embedding layers that learn distributed representations during training, offering more nuanced semantic capture than these static vectorization methods.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}