# Text Representation Methods: Bag-of-Words vs TF-IDF in the AI-For-Beginners Repository

> Explore Bag-of-Words vs TF-IDF text representation. Learn how these methods are implemented in the Microsoft AI-For-Beginners repository using scikit-learn for efficient text analysis.

- Repository: [Microsoft/AI-For-Beginners](https://github.com/microsoft/AI-For-Beginners)
- Tags: deep-dive
- Published: 2026-08-02

---

**Bag-of-Words counts raw word frequencies while TF-IDF weights terms by their importance across documents, with both methods implemented in the Microsoft AI-For-Beginners repository using scikit-learn's `CountVectorizer` and `TfidfVectorizer`.**

The **AI-For-Beginners** repository provides a structured curriculum for learning foundational AI concepts, with specific text representation methods covered in the Natural Language Processing (NLP) section. These lessons demonstrate how to convert unstructured text into numerical vectors that machine learning models can process. Understanding the differences between **Bag-of-Words** and **TF-IDF** is essential for building effective NLP pipelines, as implemented in the notebooks located at `lessons/5-NLP/13-TextRep/`.

## Bag-of-Words Implementation

The **Bag-of-Words (BoW)** approach creates a vocabulary of unique tokens from the corpus and represents each document as a vector of word counts. In the AI-For-Beginners curriculum, this method is demonstrated in `lessons/5-NLP/13-TextRep/TextRepresentationPyTorch.ipynb` using scikit-learn's `CountVectorizer`.

The notebook follows a standard pipeline:

1. Import `CountVectorizer` from `sklearn.feature_extraction.text`
2. Define a sample corpus of raw sentences
3. Instantiate the vectorizer and fit it to the data
4. Transform the corpus into a sparse count matrix

```python
from sklearn.feature_extraction.text import CountVectorizer

sentences = [
    "the cat sat on the mat",
    "the dog barked loudly",
    "the cat and the dog played"
]

vectorizer = CountVectorizer()
bow_matrix = vectorizer.fit_transform(sentences)

print("Vocabulary:", vectorizer.vocabulary_)
print("BoW matrix:\n", bow_matrix.toarray())

```

The resulting matrix contains integer values representing how many times each word appears in each document. The `vocabulary_` attribute maps each unique word to a specific column index, creating a high-dimensional but interpretable representation.

## TF-IDF Implementation

The **TF-IDF (Term Frequency-Inverse Document Frequency)** method extends BoW by weighting terms based on their discriminative power across the corpus. Common words that appear in many documents receive lower weights, while rare, distinctive terms receive higher weights. This implementation resides in `lessons/5-NLP/13-TextRep/TextRepresentationTF.ipynb` using `TfidfVectorizer`.

```python
from sklearn.feature_extraction.text import TfidfVectorizer

sentences = [
    "the cat sat on the mat",
    "the dog barked loudly",
    "the cat and the dog played"
]

vectorizer = TfidfVectorizer()
tfidf_matrix = vectorizer.fit_transform(sentences)

print("Vocabulary:", vectorizer.vocabulary_)
print("TF-IDF matrix:\n", tfidf_matrix.toarray())

```

Unlike the BoW output, the TF-IDF matrix contains floating-point values. The algorithm calculates **term frequency** (how often a word appears in a specific document) divided by **inverse document frequency** (a measure of how common the word is across all documents). This mitigates the "stop-word" problem where high-frequency words like "the" and "and" would otherwise dominate the representation.

## Key Differences Between Bag-of-Words and TF-IDF

When comparing these text representation methods in the context of the AI-For-Beginners examples, several critical distinctions emerge:

- **Counting vs Weighting**: BoW uses raw integer counts, while TF-IDF produces normalized floating-point weights that reflect term importance
- **Stop-word handling**: TF-IDF automatically downweights common words across the corpus, whereas BoW treats all tokens equally regardless of their discriminative value
- **Matrix sparsity**: Both methods generate sparse matrices, but TF-IDF typically produces better-separated document vectors for downstream classification tasks
- **Computational complexity**: TF-IDF requires additional passes over the corpus to calculate document frequencies, making it slightly more computationally expensive than simple counting

Both notebooks in `lessons/5-NLP/13-TextRep/` visualize these matrices using Pandas DataFrames, allowing learners to inspect exactly how each document transforms into a vector space representation.

## Summary

- **Bag-of-Words** in `TextRepresentationPyTorch.ipynb` converts text to count vectors using `CountVectorizer`, treating every word occurrence equally regardless of corpus-wide frequency.
- **TF-IDF** in `TextRepresentationTF.ipynb` applies `TfidfVectorizer` to weight terms by their importance, reducing the impact of common words and highlighting distinctive vocabulary.
- Both methods require the same scikit-learn import pattern and follow an identical `fit_transform` workflow, making it easy to experiment with either approach.
- The AI-For-Beginners repository isolates these concepts in `lessons/5-NLP/13-TextRep/` with bilingual support available in the `translations/` directory.

## Frequently Asked Questions

### What is the main advantage of TF-IDF over Bag-of-Words?

TF-IDF weights terms by their importance across the corpus, which reduces the influence of common words like "the" and "is" while highlighting distinctive terms that better differentiate documents. This typically results in superior performance for document classification and information retrieval tasks compared to the uniform counting approach of Bag-of-Words.

### How do I run the text representation notebooks from the AI-For-Beginners repository?

Clone the repository and navigate to `lessons/5-NLP/13-TextRep/` where you will find `TextRepresentationPyTorch.ipynb` for Bag-of-Words and `TextRepresentationTF.ipynb` for TF-IDF. Ensure you have the dependencies listed in [`environment.yml`](https://github.com/microsoft/AI-For-Beginners/blob/main/environment.yml) installed, particularly `scikit-learn`, `numpy`, and `pandas`, then execute the notebooks in Jupyter or VS Code.

### Why does the Bag-of-Words matrix contain only integers while TF-IDF contains decimals?

The Bag-of-Words model implemented via `CountVectorizer` counts raw word occurrences, resulting in integer values representing absolute frequencies. TF-IDF uses `TfidfVectorizer` to compute a ratio of term frequency to inverse document frequency, producing normalized floating-point weights that indicate relative importance rather than absolute counts.

### Can I use these text representation methods with deep learning models?

Yes, though the AI-For-Beginners notebooks demonstrate these methods with traditional machine learning, both Bag-of-Words and TF-IDF vectors serve as valid input features for neural networks. However, modern deep learning typically uses embedding layers that learn distributed representations during training, offering more nuanced semantic capture than these static vectorization methods.