How to Use Word2Vec and GloVe for Semantic Word Embeddings: A Complete Guide

Word2Vec and GloVe map words to dense vectors that capture semantic meaning, with Word2Vec learning from local context windows and GloVe factoring global co-occurrence statistics—both producings lookup tables you can load via gensim or torchtext and inject into Keras or PyTorch embedding layers.

Semantic word embeddings are foundational to modern NLP. In the microsoft/AI‑For‑Beginners curriculum (Lesson 5‑NLP), these classic techniques demonstrate how to transform raw text into numerical representations that neural networks can process. Whether you need pre‑trained vectors or want to train your own from scratch, the repository provides complete TensorFlow and PyTorch implementations.

Word2Vec vs GloVe: Core Differences

Both methods generate a matrix of shape |V| × d (vocabulary size × embedding dimension), but they learn from text differently:

• Word2Vec — Uses neural prediction: CBOW predicts a target word from surrounding context, while skip‑gram predicts context from a target word. Captures local co‑occurrence patterns.
• GloVe — Factorizes a global word‑co‑occurrence matrix using weighted least squares. Preserves global statistical relationships.

According to the source code in lessons/5-NLP/14-Embeddings/README.md, GloVe often performs better on word analogy tasks due to its explicit encoding of co‑occurrence ratios, while Word2Vec scales more efficiently on very large corpora.

Loading Pre‑Trained Embeddings

The repository demonstrates two primary workflows: loading Google‑News Word2Vec via gensim and loading GloVe via torchtext.

Google‑News Word2Vec with Gensim and TensorFlow

The notebook lessons/5-NLP/14-Embeddings/EmbeddingsTF.ipynb shows how to fetch the 300‑dimensional vectors and initialize a frozen Keras embedding layer:

import gensim.downloader as api
import numpy as np
import tensorflow as tf

# Download 300‑dim Google‑News vectors (~1.5 GB)

w2v = api.load('word2vec-google-news-300')

# Build vocabulary from your dataset

vocab = ['machine', 'learning', 'AI', 'unknown_token']
vocab_size = len(vocab)
embed_dim = w2v.vector_size  # 300

# Create embedding matrix with random init for OOV words

embedding_matrix = np.zeros((vocab_size, embed_dim))
for i, word in enumerate(vocab):
    if word in w2v:
        embedding_matrix[i] = w2v[word]
    else:
        embedding_matrix[i] = np.random.normal(scale=0.6, size=(embed_dim,))

# Initialize Keras Embedding layer

embedding_layer = tf.keras.layers.Embedding(
    input_dim=vocab_size,
    output_dim=embed_dim,
    embeddings_initializer=tf.keras.initializers.Constant(embedding_matrix),
    trainable=False  # Set True to fine‑tune

)

GloVe with TorchText and PyTorch

For PyTorch workflows, lessons/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb uses torchtext.vocab.GloVe to load the 6B‑50d vectors:

import torch
import torch.nn as nn
import torchtext.vocab as vocab

# Download 6B‑50d GloVe vectors (~822 MB)

glove = vocab.GloVe(name='6B', dim=50)

# Build custom vocabulary

tokens = ['deep', 'learning', 'network', '<unk>']
stoi = {tok: i for i, tok in enumerate(tokens)}
vocab_size = len(tokens)
embed_dim = glove.dim

# Create embedding matrix; random vectors for missing tokens

embedding_matrix = torch.randn(vocab_size, embed_dim) * 0.1
for token, idx in stoi.items():
    if token in glove.stoi:
        embedding_matrix[idx] = glove.vectors[glove.stoi[token]]

# Build PyTorch Embedding layer with fine‑tuning enabled

embedding_layer = nn.Embedding.from_pretrained(embedding_matrix, freeze=False)

Training Word2Vec from Scratch

When domain‑specific text differs significantly from general corpora, training embeddings yourself yields better representations. The repository provides complete CBOW (Continuous Bag of Words) implementations in both frameworks.

CBOW in TensorFlow

From lessons/5-NLP/15-LanguageModeling/CBoW-TF.ipynb, a minimal CBOW architecture:

import tensorflow as tf
import tensorflow.keras.layers as L

vocab_size = 5000
embed_dim = 30

# Context: two surrounding words (left and right)

inputs = L.Input(shape=(2,))
embed = L.Embedding(vocab_size, embed_dim)(inputs)
avg = tf.reduce_mean(embed, axis=1)  # Average context vectors

output = L.Dense(vocab_size, activation='softmax')(avg)

model = tf.keras.Model(inputs, output)
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy')

# model.fit(context_pairs, target_word_ids, epochs=5)

After training, extract the learned embeddings via model.layers[1].get_weights() for reuse in downstream models.

CBOW in PyTorch

The equivalent PyTorch implementation in lessons/5-NLP/15-LanguageModeling/CBoW-PyTorch.ipynb uses native nn.Embedding and custom training loops.

Using Embeddings in Downstream Models

Semantic embeddings become powerful when integrated into classification or sequence models. The repository demonstrates injecting pre‑trained matrices into RNN classifiers.

Text Classifier with Learnable Embeddings

import torch
import torch.nn as nn

class TextClassifier(nn.Module):
    def __init__(self, pretrained_embeddings):
        super().__init__()
        vocab_size, embed_dim = pretrained_embeddings.shape
        self.embedding = nn.Embedding.from_pretrained(
            pretrained_embeddings, 
            freeze=False  # Enable fine‑tuning

        )
        self.rnn = nn.LSTM(embed_dim, hidden_size=64, batch_first=True)
        self.fc = nn.Linear(64, 2)  # Binary classification

    def forward(self, x):
        x = self.embedding(x)           # (batch, time, dim)

        _, (h_n, _) = self.rnn(x)       # Final hidden state

        logits = self.fc(h_n.squeeze(0))
        return logits

Set freeze=True to keep embeddings fixed during training—useful when data is scarce.

Handling Vocabulary Mismatches

A critical practical detail from lessons/5-NLP/14-Embeddings/README.md: pre‑trained vocabularies rarely align perfectly with your dataset. The recommended strategy:

1. Build your vocabulary from training data
2. Initialize embedding matrix with random values
3. Copy vectors for overlapping tokens
4. Keep random initialization for out‑of‑vocabulary (OOV) words, or use subword backoffs

Both gensim and torchtext expose stoi (string‑to‑index) dictionaries to facilitate this alignment.

Complete Pipeline Overview

Raw text → Tokenization → Vocabulary construction → Embedding matrix alignment
                                              ↓
                          [Pre‑trained via gensim/torchtext] OR [Trained via CBOW]
                                              ↓
                    Keras Embedding layer OR PyTorch nn.Embedding
                                              ↓
                         Downstream model (RNN, Transformer, classifier)

Summary

• Word2Vec learns local context patterns via predictive training (CBOW or skip‑gram); GloVe captures global co‑occurrence statistics through matrix factorization
• Load Google‑News Word2Vec via gensim.downloader in TensorFlow workflows (EmbeddingsTF.ipynb)
• Load GloVe vectors via torchtext.vocab.GloVe in PyTorch workflows (EmbeddingsPyTorch.ipynb)
• Train domain‑specific embeddings from scratch using the CBOW notebooks: CBoW-TF.ipynb and CBoW-PyTorch.ipynb
• Align pre‑trained matrices to your vocabulary by handling OOV words with random initialization
• Control fine‑tuning behavior with trainable (Keras) or freeze (PyTorch) parameters

Frequently Asked Questions

What is the difference between Word2Vec and GloVe embeddings?

Word2Vec uses a shallow neural network to predict words from their neighbors (or vice‑versa), capturing local context patterns. GloVe explicitly factorizes a global word‑co‑occurrence matrix, preserving statistical ratios that often yield better performance on word analogy tasks. Both produce dense vectors of equivalent shape and can substitute for one another in embedding layers.

How do I handle words not in the pre‑trained vocabulary?

Initialize OOV word vectors randomly—typically with small Gaussian noise (scale 0.1–0.6). In the repository examples, this happens automatically when iterating through your vocabulary and checking membership in w2v or glove.stoi. Alternatively, use subword methods like FastText (outside this curriculum) for morphologically rich languages.

Should I fine‑tune pre‑trained embeddings or freeze them?

Freeze when your downstream dataset is small; this prevents overfitting and leverages general semantic knowledge. Fine‑tune when you have substantial task‑specific data (10,000+ examples) or when domain terminology differs significantly from the pre‑training corpus. Both Keras (trainable=True/False) and PyTorch (freeze=True/False) provide this control.

Where are the embedding files stored when using gensim or torchtext?

gensim.downloader caches vectors in ~/gensim-data/ by default. torchtext.vocab.GloVe downloads to ~/.vector_cache/. Both libraries check these locations before re‑downloading, so subsequent loads are instant after the first fetch.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}