# How to Use Word2Vec and GloVe Embeddings for Text Representation in NLP

> Learn Word2Vec GloVe embeddings for text representation in NLP with Microsoft AI For Beginners. Enhance your AI skills by understanding dense vector mappings.

- Repository: [Microsoft/AI-For-Beginners](https://github.com/microsoft/AI-For-Beginners)
- Tags: tutorial
- Published: 2026-08-01

---

**Word2Vec and GloVe embeddings map words to dense, low-dimensional vectors where semantic similarity correlates with vector proximity, allowing the Microsoft AI-For-Beginners curriculum to replace sparse bag-of-words representations with pretrained neural word embeddings via the [`torchnlp.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/torchnlp.py) helper utilities.**

The Microsoft AI-For-Beginners repository provides a comprehensive introduction to natural language processing (NLP) through hands-on lessons that demonstrate how to integrate pretrained Word2Vec and GloVe embeddings into PyTorch and TensorFlow pipelines. Understanding how to use these dense vector representations is essential for modern NLP tasks, as they capture semantic relationships that traditional TF-IDF vectors miss entirely.

## Why Embeddings Outperform Bag-of-Words

Traditional **bag-of-words (BoW)** and TF-IDF representations create high-dimensional, sparse vectors that treat each token independently. This approach fails to capture that "king" and "queen" share semantic relationships or that "Paris" and "France" are geographically related.

**Word embeddings** solve this by projecting vocabulary into a continuous vector space where distance correlates with semantic similarity. The AI-For-Beginners curriculum emphasizes two pretrained approaches:

- **Word2Vec** – Trained using continuous bag-of-words (CBoW) or skip-gram objectives to predict context words
- **GloVe** – Global Vectors for Word Representation that factorize global word-word co-occurrence statistics from corpora

## Loading Pretrained GloVe Embeddings with torchtext

The curriculum demonstrates loading pretrained vectors using `torchtext.vocab.GloVe`, which downloads 50d, 100d, 200d, or 300d vectors trained on Common Crawl, Wikipedia, or Twitter datasets.

```python
import torch
from torchtext.vocab import GloVe

# Load 300-dimensional vectors trained on Wikipedia and Gigaword (6B tokens)

glove = GloVe(name='6B', dim=300)

# Access vectors directly

vector = glove['artificial']  # Returns torch.Tensor of shape (300,)

```

For **Word2Vec**, you load binary or text format files using `torchtext.vocab.Vectors`, which exposes the same interface for vector lookup.

## The `encode` Helper Function in [`torchnlp.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/torchnlp.py)

In [`lessons/5-NLP/14-Embeddings/torchnlp.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/14-Embeddings/torchnlp.py), the curriculum defines a generic `encode` function that abstracts vocabulary handling for both GloVe objects and torchtext Vocab instances. This function handles the divergence between `get_stoi()` methods and raw `stoi` attributes while providing out-of-vocabulary (OOV) token handling.

```python

# From torchnlp.py – core encoding logic

def encode(x, voc=None, unk=0, tokenizer=tokenizer):
    v = vocab if voc is None else voc
    if v in stoi_hash:
        stoi = stoi_hash[v]
    else:
        # Polymorphic handling: works for torchtext Vocab and GloVe objects

        if hasattr(v, 'get_stoi'):
            stoi = v.get_stoi()
        else:
            stoi = v.stoi
        stoi_hash[v] = stoi
    return [stoi.get(s, unk) for s in tokenizer(x)]

```

The function caches string-to-index mappings (`stoi`) in `stoi_hash` to avoid recomputing mappings across multiple calls, significantly improving preprocessing performance for large datasets.

## Aligning Task Vocabularies with Pretrained Vectors

Real-world NLP requires mapping your task-specific vocabulary to pretrained embedding matrices. The notebooks `EmbeddingsPyTorch.ipynb` and `EmbeddingsTF.ipynb` demonstrate building a vocabulary from training data, then initializing an embedding matrix that copies pretrained vectors for known tokens and uses random initialization for OOV words.

```python
from collections import Counter
from torchtext.data.utils import get_tokenizer, ngrams_iterator
import torchtext

tokenizer = get_tokenizer('basic_english')

# Build task-specific vocabulary from training sentences

train_sentences = [
    "The stock market rallied today",
    "Scientists discovered a new particle"
]
counter = Counter()
for line in train_sentences:
    counter.update(ngrams_iterator(tokenizer(line), ngrams=1))

task_vocab = torchtext.vocab.vocab(counter, min_freq=1)

# Initialize embedding matrix (vocab_size × glove_dim)

embedding_matrix = torch.zeros(len(task_vocab), glove.dim)
for i, token in enumerate(task_vocab.get_itos()):
    if token in glove.stoi:
        embedding_matrix[i] = glove[token]
    else:
        # Gaussian initialization for OOV tokens with small variance

        embedding_matrix[i] = torch.randn(glove.dim) * 0.01

```

## Building a Neural Classifier with Pretrained Embeddings

Once aligned, you feed encoded sequences through `torch.nn.Embedding.from_pretrained()`, which accepts the matrix constructed above. Setting `freeze=False` allows fine-tuning during training, while `freeze=True` keeps the pretrained vectors static.

```python
class SimpleClassifier(torch.nn.Module):
    def __init__(self, embed_matrix, num_classes):
        super().__init__()
        num_embeddings, embed_dim = embed_matrix.shape
        # Load pretrained weights; freeze=False enables fine-tuning

        self.emb = torch.nn.Embedding.from_pretrained(embed_matrix, freeze=False)
        self.fc = torch.nn.Linear(embed_dim, num_classes)
        
    def forward(self, token_idxs):
        # token_idxs shape: (batch_size, seq_len)

        embedded = self.emb(token_idxs)        # (batch, seq_len, embed_dim)

        pooled = embedded.mean(dim=1)          # Mean pooling across sequence

        return self.fc(pooled)

# Instantiate with aligned embedding matrix

model = SimpleClassifier(embedding_matrix, num_classes=4)

```

The `encode` function from [`torchnlp.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/torchnlp.py) converts raw text into integer indices compatible with this model architecture, handling tokenization and OOV mapping automatically.

## Summary

- **Word2Vec and GloVe** provide dense semantic representations that capture linguistic relationships missed by sparse BoW vectors.
- The **`encode` function** in [`lessons/5-NLP/14-Embeddings/torchnlp.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/14-Embeddings/torchnlp.py) abstracts vocabulary handling for both embedding types, supporting `get_stoi()` methods and raw `stoi` attributes.
- **Vocabulary alignment** requires copying pretrained vectors for known tokens and initializing OOV entries randomly before training downstream classifiers.
- The curriculum provides both **PyTorch** (`EmbeddingsPyTorch.ipynb`) and **TensorFlow** (`EmbeddingsTF.ipynb`) implementations demonstrating identical workflows.

## Frequently Asked Questions

### How do I handle out-of-vocabulary words when using pretrained GloVe embeddings?

The `encode` function in [`torchnlp.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/torchnlp.py) maps unknown tokens to a default index (typically 0) that you can initialize randomly in your embedding matrix. During alignment, check if `token in glove.stoi`; if False, initialize that row with small Gaussian noise (`torch.randn(dim) * 0.01`) rather than pretrained weights.

### What is the difference between Word2Vec and GloVe embeddings?

**Word2Vec** trains locally using predictive objectives (predicting context words given a center word or vice versa), while **GloVe** trains globally using matrix factorization on word co-occurrence statistics across the entire corpus. Both produce 50-300 dimensional vectors where cosine similarity correlates with semantic relatedness, but GloVe often performs better on analogy tasks due to its global statistical approach.

### Can I fine-tune pretrained embeddings during model training?

Yes. When calling `torch.nn.Embedding.from_pretrained(embed_matrix, freeze=False)`, PyTorch allows gradients to flow through the embedding layer, updating the vectors to better suit your specific task. Setting `freeze=True` keeps the original Word2Vec or GloVe vectors static, which is preferable when training data is limited.

### Where does the Microsoft AI-For-Beginners curriculum store the embedding utilities?

The core helper functions reside in [`lessons/5-NLP/14-Embeddings/torchnlp.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/14-Embeddings/torchnlp.py), which provides `encode`, `load_dataset`, and `padify` functions used by both the PyTorch and TensorFlow notebooks in the same directory. The lesson README at [`lessons/5-NLP/14-Embeddings/README.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/14-Embeddings/README.md) explains the theoretical foundations of CBoW, skip-gram, and co-occurrence matrices.