# How to Use Word2Vec and GloVe for Semantic Word Embeddings: A Complete Guide

> Master Word2Vec and GloVe for semantic word embeddings. Learn how these techniques create dense vectors from word contexts and co-occurrence, perfect for Keras or PyTorch.

- Repository: [Microsoft/AI-For-Beginners](https://github.com/microsoft/AI-For-Beginners)
- Tags: how-to-guide
- Published: 2026-08-10

---

**Word2Vec and GloVe map words to dense vectors that capture semantic meaning, with Word2Vec learning from local context windows and GloVe factoring global co-occurrence statistics—both producings lookup tables you can load via `gensim` or `torchtext` and inject into Keras or PyTorch embedding layers.**

Semantic word embeddings are foundational to modern NLP. In the **microsoft/AI‑For‑Beginners** curriculum (Lesson 5‑NLP), these classic techniques demonstrate how to transform raw text into numerical representations that neural networks can process. Whether you need pre‑trained vectors or want to train your own from scratch, the repository provides complete TensorFlow and PyTorch implementations.

## Word2Vec vs GloVe: Core Differences

Both methods generate a matrix of shape `|V| × d` (vocabulary size × embedding dimension), but they learn from text differently:

- **Word2Vec** — Uses neural prediction: CBOW predicts a target word from surrounding context, while skip‑gram predicts context from a target word. Captures **local co‑occurrence patterns**.
- **GloVe** — Factorizes a global word‑co‑occurrence matrix using weighted least squares. Preserves **global statistical relationships**.

According to the source code in [`lessons/5-NLP/14-Embeddings/README.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/14-Embeddings/README.md), GloVe often performs better on word analogy tasks due to its explicit encoding of co‑occurrence ratios, while Word2Vec scales more efficiently on very large corpora.

## Loading Pre‑Trained Embeddings

The repository demonstrates two primary workflows: loading Google‑News Word2Vec via `gensim` and loading GloVe via `torchtext`.

### Google‑News Word2Vec with Gensim and TensorFlow

The notebook `lessons/5-NLP/14-Embeddings/EmbeddingsTF.ipynb` shows how to fetch the 300‑dimensional vectors and initialize a frozen Keras embedding layer:

```python
import gensim.downloader as api
import numpy as np
import tensorflow as tf

# Download 300‑dim Google‑News vectors (~1.5 GB)

w2v = api.load('word2vec-google-news-300')

# Build vocabulary from your dataset

vocab = ['machine', 'learning', 'AI', 'unknown_token']
vocab_size = len(vocab)
embed_dim = w2v.vector_size  # 300

# Create embedding matrix with random init for OOV words

embedding_matrix = np.zeros((vocab_size, embed_dim))
for i, word in enumerate(vocab):
    if word in w2v:
        embedding_matrix[i] = w2v[word]
    else:
        embedding_matrix[i] = np.random.normal(scale=0.6, size=(embed_dim,))

# Initialize Keras Embedding layer

embedding_layer = tf.keras.layers.Embedding(
    input_dim=vocab_size,
    output_dim=embed_dim,
    embeddings_initializer=tf.keras.initializers.Constant(embedding_matrix),
    trainable=False  # Set True to fine‑tune

)

```

### GloVe with TorchText and PyTorch

For PyTorch workflows, `lessons/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb` uses `torchtext.vocab.GloVe` to load the 6B‑50d vectors:

```python
import torch
import torch.nn as nn
import torchtext.vocab as vocab

# Download 6B‑50d GloVe vectors (~822 MB)

glove = vocab.GloVe(name='6B', dim=50)

# Build custom vocabulary

tokens = ['deep', 'learning', 'network', '<unk>']
stoi = {tok: i for i, tok in enumerate(tokens)}
vocab_size = len(tokens)
embed_dim = glove.dim

# Create embedding matrix; random vectors for missing tokens

embedding_matrix = torch.randn(vocab_size, embed_dim) * 0.1
for token, idx in stoi.items():
    if token in glove.stoi:
        embedding_matrix[idx] = glove.vectors[glove.stoi[token]]

# Build PyTorch Embedding layer with fine‑tuning enabled

embedding_layer = nn.Embedding.from_pretrained(embedding_matrix, freeze=False)

```

## Training Word2Vec from Scratch

When domain‑specific text differs significantly from general corpora, training embeddings yourself yields better representations. The repository provides complete CBOW (Continuous Bag of Words) implementations in both frameworks.

### CBOW in TensorFlow

From `lessons/5-NLP/15-LanguageModeling/CBoW-TF.ipynb`, a minimal CBOW architecture:

```python
import tensorflow as tf
import tensorflow.keras.layers as L

vocab_size = 5000
embed_dim = 30

# Context: two surrounding words (left and right)

inputs = L.Input(shape=(2,))
embed = L.Embedding(vocab_size, embed_dim)(inputs)
avg = tf.reduce_mean(embed, axis=1)  # Average context vectors

output = L.Dense(vocab_size, activation='softmax')(avg)

model = tf.keras.Model(inputs, output)
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy')

# model.fit(context_pairs, target_word_ids, epochs=5)

```

After training, extract the learned embeddings via `model.layers[1].get_weights()` for reuse in downstream models.

### CBOW in PyTorch

The equivalent PyTorch implementation in `lessons/5-NLP/15-LanguageModeling/CBoW-PyTorch.ipynb` uses native `nn.Embedding` and custom training loops.

## Using Embeddings in Downstream Models

Semantic embeddings become powerful when integrated into classification or sequence models. The repository demonstrates injecting pre‑trained matrices into RNN classifiers.

### Text Classifier with Learnable Embeddings

```python
import torch
import torch.nn as nn

class TextClassifier(nn.Module):
    def __init__(self, pretrained_embeddings):
        super().__init__()
        vocab_size, embed_dim = pretrained_embeddings.shape
        self.embedding = nn.Embedding.from_pretrained(
            pretrained_embeddings, 
            freeze=False  # Enable fine‑tuning

        )
        self.rnn = nn.LSTM(embed_dim, hidden_size=64, batch_first=True)
        self.fc = nn.Linear(64, 2)  # Binary classification

    def forward(self, x):
        x = self.embedding(x)           # (batch, time, dim)

        _, (h_n, _) = self.rnn(x)       # Final hidden state

        logits = self.fc(h_n.squeeze(0))
        return logits

```

Set `freeze=True` to keep embeddings fixed during training—useful when data is scarce.

## Handling Vocabulary Mismatches

A critical practical detail from [`lessons/5-NLP/14-Embeddings/README.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/14-Embeddings/README.md): pre‑trained vocabularies rarely align perfectly with your dataset. The recommended strategy:

1. **Build your vocabulary** from training data
2. **Initialize embedding matrix** with random values
3. **Copy vectors** for overlapping tokens
4. **Keep random initialization** for out‑of‑vocabulary (OOV) words, or use subword backoffs

Both `gensim` and `torchtext` expose `stoi` (string‑to‑index) dictionaries to facilitate this alignment.

## Complete Pipeline Overview

```

Raw text → Tokenization → Vocabulary construction → Embedding matrix alignment
                                              ↓
                          [Pre‑trained via gensim/torchtext] OR [Trained via CBOW]
                                              ↓
                    Keras Embedding layer OR PyTorch nn.Embedding
                                              ↓
                         Downstream model (RNN, Transformer, classifier)

```

## Summary

- **Word2Vec** learns local context patterns via predictive training (CBOW or skip‑gram); **GloVe** captures global co‑occurrence statistics through matrix factorization
- Load Google‑News Word2Vec via `gensim.downloader` in TensorFlow workflows (`EmbeddingsTF.ipynb`)
- Load GloVe vectors via `torchtext.vocab.GloVe` in PyTorch workflows (`EmbeddingsPyTorch.ipynb`)
- Train domain‑specific embeddings from scratch using the CBOW notebooks: `CBoW-TF.ipynb` and `CBoW-PyTorch.ipynb`
- Align pre‑trained matrices to your vocabulary by handling OOV words with random initialization
- Control fine‑tuning behavior with `trainable` (Keras) or `freeze` (PyTorch) parameters

## Frequently Asked Questions

### What is the difference between Word2Vec and GloVe embeddings?

**Word2Vec** uses a shallow neural network to predict words from their neighbors (or vice‑versa), capturing local context patterns. **GloVe** explicitly factorizes a global word‑co‑occurrence matrix, preserving statistical ratios that often yield better performance on word analogy tasks. Both produce dense vectors of equivalent shape and can substitute for one another in embedding layers.

### How do I handle words not in the pre‑trained vocabulary?

Initialize OOV word vectors randomly—typically with small Gaussian noise (scale 0.1–0.6). In the repository examples, this happens automatically when iterating through your vocabulary and checking membership in `w2v` or `glove.stoi`. Alternatively, use subword methods like FastText (outside this curriculum) for morphologically rich languages.

### Should I fine‑tune pre‑trained embeddings or freeze them?

**Freeze** when your downstream dataset is small; this prevents overfitting and leverages general semantic knowledge. **Fine‑tune** when you have substantial task‑specific data (10,000+ examples) or when domain terminology differs significantly from the pre‑training corpus. Both Keras (`trainable=True/False`) and PyTorch (`freeze=True/False`) provide this control.

### Where are the embedding files stored when using gensim or torchtext?

`gensim.downloader` caches vectors in `~/gensim-data/` by default. `torchtext.vocab.GloVe` downloads to `~/.vector_cache/`. Both libraries check these locations before re‑downloading, so subsequent loads are instant after the first fetch.