How to Use Word2Vec and GloVe Embeddings for Text Representation in NLP

Word2Vec and GloVe embeddings map words to dense, low-dimensional vectors where semantic similarity correlates with vector proximity, allowing the Microsoft AI-For-Beginners curriculum to replace sparse bag-of-words representations with pretrained neural word embeddings via the torchnlp.py helper utilities.

The Microsoft AI-For-Beginners repository provides a comprehensive introduction to natural language processing (NLP) through hands-on lessons that demonstrate how to integrate pretrained Word2Vec and GloVe embeddings into PyTorch and TensorFlow pipelines. Understanding how to use these dense vector representations is essential for modern NLP tasks, as they capture semantic relationships that traditional TF-IDF vectors miss entirely.

Why Embeddings Outperform Bag-of-Words

Traditional bag-of-words (BoW) and TF-IDF representations create high-dimensional, sparse vectors that treat each token independently. This approach fails to capture that "king" and "queen" share semantic relationships or that "Paris" and "France" are geographically related.

Word embeddings solve this by projecting vocabulary into a continuous vector space where distance correlates with semantic similarity. The AI-For-Beginners curriculum emphasizes two pretrained approaches:

• Word2Vec – Trained using continuous bag-of-words (CBoW) or skip-gram objectives to predict context words
• GloVe – Global Vectors for Word Representation that factorize global word-word co-occurrence statistics from corpora

Loading Pretrained GloVe Embeddings with torchtext

The curriculum demonstrates loading pretrained vectors using torchtext.vocab.GloVe, which downloads 50d, 100d, 200d, or 300d vectors trained on Common Crawl, Wikipedia, or Twitter datasets.

import torch
from torchtext.vocab import GloVe

# Load 300-dimensional vectors trained on Wikipedia and Gigaword (6B tokens)

glove = GloVe(name='6B', dim=300)

# Access vectors directly

vector = glove['artificial']  # Returns torch.Tensor of shape (300,)

For Word2Vec, you load binary or text format files using torchtext.vocab.Vectors, which exposes the same interface for vector lookup.

The encode Helper Function in torchnlp.py

In lessons/5-NLP/14-Embeddings/torchnlp.py, the curriculum defines a generic encode function that abstracts vocabulary handling for both GloVe objects and torchtext Vocab instances. This function handles the divergence between get_stoi() methods and raw stoi attributes while providing out-of-vocabulary (OOV) token handling.

# From torchnlp.py – core encoding logic

def encode(x, voc=None, unk=0, tokenizer=tokenizer):
    v = vocab if voc is None else voc
    if v in stoi_hash:
        stoi = stoi_hash[v]
    else:
        # Polymorphic handling: works for torchtext Vocab and GloVe objects

        if hasattr(v, 'get_stoi'):
            stoi = v.get_stoi()
        else:
            stoi = v.stoi
        stoi_hash[v] = stoi
    return [stoi.get(s, unk) for s in tokenizer(x)]

The function caches string-to-index mappings (stoi) in stoi_hash to avoid recomputing mappings across multiple calls, significantly improving preprocessing performance for large datasets.

Aligning Task Vocabularies with Pretrained Vectors

Real-world NLP requires mapping your task-specific vocabulary to pretrained embedding matrices. The notebooks EmbeddingsPyTorch.ipynb and EmbeddingsTF.ipynb demonstrate building a vocabulary from training data, then initializing an embedding matrix that copies pretrained vectors for known tokens and uses random initialization for OOV words.

from collections import Counter
from torchtext.data.utils import get_tokenizer, ngrams_iterator
import torchtext

tokenizer = get_tokenizer('basic_english')

# Build task-specific vocabulary from training sentences

train_sentences = [
    "The stock market rallied today",
    "Scientists discovered a new particle"
]
counter = Counter()
for line in train_sentences:
    counter.update(ngrams_iterator(tokenizer(line), ngrams=1))

task_vocab = torchtext.vocab.vocab(counter, min_freq=1)

# Initialize embedding matrix (vocab_size × glove_dim)

embedding_matrix = torch.zeros(len(task_vocab), glove.dim)
for i, token in enumerate(task_vocab.get_itos()):
    if token in glove.stoi:
        embedding_matrix[i] = glove[token]
    else:
        # Gaussian initialization for OOV tokens with small variance

        embedding_matrix[i] = torch.randn(glove.dim) * 0.01

Building a Neural Classifier with Pretrained Embeddings

Once aligned, you feed encoded sequences through torch.nn.Embedding.from_pretrained(), which accepts the matrix constructed above. Setting freeze=False allows fine-tuning during training, while freeze=True keeps the pretrained vectors static.

class SimpleClassifier(torch.nn.Module):
    def __init__(self, embed_matrix, num_classes):
        super().__init__()
        num_embeddings, embed_dim = embed_matrix.shape
        # Load pretrained weights; freeze=False enables fine-tuning

        self.emb = torch.nn.Embedding.from_pretrained(embed_matrix, freeze=False)
        self.fc = torch.nn.Linear(embed_dim, num_classes)
        
    def forward(self, token_idxs):
        # token_idxs shape: (batch_size, seq_len)

        embedded = self.emb(token_idxs)        # (batch, seq_len, embed_dim)

        pooled = embedded.mean(dim=1)          # Mean pooling across sequence

        return self.fc(pooled)

# Instantiate with aligned embedding matrix

model = SimpleClassifier(embedding_matrix, num_classes=4)

The encode function from torchnlp.py converts raw text into integer indices compatible with this model architecture, handling tokenization and OOV mapping automatically.

Summary

• Word2Vec and GloVe provide dense semantic representations that capture linguistic relationships missed by sparse BoW vectors.
• The encode function in lessons/5-NLP/14-Embeddings/torchnlp.py abstracts vocabulary handling for both embedding types, supporting get_stoi() methods and raw stoi attributes.
• Vocabulary alignment requires copying pretrained vectors for known tokens and initializing OOV entries randomly before training downstream classifiers.
• The curriculum provides both PyTorch (EmbeddingsPyTorch.ipynb) and TensorFlow (EmbeddingsTF.ipynb) implementations demonstrating identical workflows.

Frequently Asked Questions

How do I handle out-of-vocabulary words when using pretrained GloVe embeddings?

The encode function in torchnlp.py maps unknown tokens to a default index (typically 0) that you can initialize randomly in your embedding matrix. During alignment, check if token in glove.stoi; if False, initialize that row with small Gaussian noise (torch.randn(dim) * 0.01) rather than pretrained weights.

What is the difference between Word2Vec and GloVe embeddings?

Word2Vec trains locally using predictive objectives (predicting context words given a center word or vice versa), while GloVe trains globally using matrix factorization on word co-occurrence statistics across the entire corpus. Both produce 50-300 dimensional vectors where cosine similarity correlates with semantic relatedness, but GloVe often performs better on analogy tasks due to its global statistical approach.

Can I fine-tune pretrained embeddings during model training?

Yes. When calling torch.nn.Embedding.from_pretrained(embed_matrix, freeze=False), PyTorch allows gradients to flow through the embedding layer, updating the vectors to better suit your specific task. Setting freeze=True keeps the original Word2Vec or GloVe vectors static, which is preferable when training data is limited.

Where does the Microsoft AI-For-Beginners curriculum store the embedding utilities?

The core helper functions reside in lessons/5-NLP/14-Embeddings/torchnlp.py, which provides encode, load_dataset, and padify functions used by both the PyTorch and TensorFlow notebooks in the same directory. The lesson README at lessons/5-NLP/14-Embeddings/README.md explains the theoretical foundations of CBoW, skip-gram, and co-occurrence matrices.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}