# How to Create Semantic Word Embeddings Using Word2Vec and GloVe in Python

> Learn to create semantic word embeddings using Word2Vec and GloVe in Python. Explore hands-on notebooks with TensorFlow and PyTorch in the Microsoft AI-For-Beginners curriculum.

- Repository: [Microsoft/AI-For-Beginners](https://github.com/microsoft/AI-For-Beginners)
- Tags: how-to-guide
- Published: 2026-08-18

---

**Word2Vec learns dense vector representations by predicting words from their surrounding context (CBOW) or vice-versa (Skip-Gram), while GloVe factorizes global co-occurrence statistics—both techniques are demonstrated in the Microsoft AI-For-Beginners curriculum through hands-on TensorFlow and PyTorch notebooks.**

The Microsoft **AI-For-Beginners** repository provides a complete educational pathway for creating and using semantic word embeddings. This guide walks through the core techniques—training custom Word2Vec embeddings from scratch and loading pre-trained Word2Vec and GloVe vectors—based on the actual implementation files in the curriculum.

---

## Word2Vec: Training Custom Embeddings with CBOW

The repository teaches Word2Vec through the **Continuous Bag-of-Words (CBOW)** architecture, where the model predicts a target word from its surrounding context words. This forces the embedding layer to encode syntactic and semantic relationships.

### CBOW Implementation in TensorFlow

The notebook `lessons/5-NLP/15-LanguageModeling/CBoW-TF.ipynb` implements CBOW training from scratch on the AG News dataset. Here's the core pattern:

```python
import tensorflow as tf
from tensorflow.keras import layers

# Define the embedding layer that will be trained

embedder = layers.Embedding(input_dim=vocab_size,
                           output_dim=embedding_dim,
                           name='embedder')

# CBOW architecture: average context embeddings, predict target word

context_input = tf.keras.Input(shape=(context_len,), dtype='int32')
context_emb = embedder(context_input)                     # (batch, context_len, embed_dim)

context_vec = tf.reduce_mean(context_emb, axis=1)        # (batch, embed_dim)

# Output layer predicts the center word from vocabulary

output = layers.Dense(vocab_size, activation='softmax')(context_vec)

model = tf.keras.Model(inputs=context_input, outputs=output)
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy')

```

After training, extract the learned embedding matrix with `embedder.get_weights()[0]`. This matrix contains the semantic word embeddings that can be reused in downstream models.

### PyTorch Equivalent

The parallel implementation in `CBoW-PyTorch.ipynb` uses `torch.nn.Embedding` with the same CBOW logic, demonstrating framework-agnostic understanding of the word2vec principle.

---

## Loading Pre-Trained Word2Vec Vectors with Gensim

For production use, the repository emphasizes leveraging pre-trained embeddings. The TensorFlow notebook `EmbeddingsTF.ipynb` demonstrates loading Google's Word2Vec vectors via **gensim**:

```python
import gensim.downloader as api

# Load 300-dimensional vectors trained on Google News (3 billion words)

word2vec = api.load('word2vec-google-news-300')

# Query semantic relationships

similar = word2vec.most_similar('neural', topn=5)
print(similar)

# Output: [('networks', 0.72), ('computational', 0.68), ...]

```

These vectors capture rich semantic properties—for example, vector arithmetic like `king - man + woman ≈ queen`.

---

## GloVe: Global Vectors for Word Representation

**GloVe** (Global Vectors) combines the advantages of count-based matrix factorization and local context windows. It trains on global word-word co-occurrence statistics rather than prediction objectives.

### Loading GloVe with Gensim (TensorFlow)

From `EmbeddingsTF.ipynb` (around line 443):

```python
import gensim.downloader as api

# Load 100-dimensional GloVe vectors (Wikipedia + Gigaword corpus)

glove = api.load('glove-wiki-gigaword-100')

# Access individual word vectors

vector = glove['computer']
print(vector.shape)   # (100,)

```

### Loading GloVe with TorchText (PyTorch)

The PyTorch notebook `EmbeddingsPyTorch.ipynb` uses the built-in `torchtext.vocab.GloVe` class:

```python
import torchtext.vocab as vocab

# Download 6B-token, 50-dimensional GloVe vectors

glove = vocab.GloVe(name='6B', dim=50)

# Retrieve tensor directly

vector = glove['computer']      # torch.Tensor with shape (50,)

```

This approach handles downloading, caching, and vocabulary indexing automatically.

---

## Integrating Pre-Trained Embeddings into Neural Models

A critical step covered in [`lessons/5-NLP/14-Embeddings/README.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/14-Embeddings/README.md) is **vocabulary alignment**—pre-trained vocabularies rarely match your corpus exactly. The repository presents two strategies:

- **Subset-only**: Initialize only overlapping tokens from pre-trained vectors, set others to zero or random values
- **Fine-tuning**: Load pre-trained vectors then continue training on your corpus to adapt missing entries

### Injecting into Keras Embedding Layer

From `EmbeddingsTF.ipynb` (lines 454-458):

```python

# pretrained_matrix shape must match (vocab_size, embed_dim)

embedding_layer = layers.Embedding(
    input_dim=vocab_size,
    output_dim=embed_dim,
    weights=[pretrained_matrix],   # Load pre-trained GloVe/Word2Vec

    trainable=False                # Set True to fine-tune

)

```

### PyTorch Alternative

```python
import torch.nn as nn

# From pre-trained matrix tensor

embedding = nn.Embedding.from_pretrained(
    pretrained_matrix,
    freeze=False   # Allow fine-tuning

)

```

---

## Key Files and Learning Path

| File | Purpose |
|------|---------|
| [`lessons/5-NLP/14-Embeddings/README.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/lessons/5-NLP/14-Embeddings/README.md) | Conceptual foundation: Word2Vec vs. GloVe, vocabulary alignment strategies |
| `lessons/5-NLP/14-Embeddings/EmbeddingsTF.ipynb` | TensorFlow: gensim loading, Keras integration |
| `lessons/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb` | PyTorch: torchtext.vocab.GloVe, from_pretrained |
| `lessons/5-NLP/15-LanguageModeling/CBoW-TF.ipynb` | End-to-end CBOW training from scratch |
| `lessons/5-NLP/15-LanguageModeling/CBoW-PyTorch.ipynb` | PyTorch CBOW implementation |

These files progress from **theory** → **custom training** → **pre-trained usage** → **downstream integration**, forming a complete semantic word embedding workflow.

---

## Summary

- **Word2Vec CBOW** predicts target words from context; train custom embeddings via `CBoW-TF.ipynb` or `CBoW-PyTorch.ipynb`
- **GloVe** uses global co-occurrence statistics; load via `gensim.downloader` (TensorFlow) or `torchtext.vocab.GloVe` (PyTorch)
- **Pre-trained embeddings** require vocabulary alignment; handle missing tokens through subsetting or fine-tuning
- **Integration** uses `weights=` parameter in Keras or `from_pretrained()` in PyTorch to inject semantic knowledge into any neural model

---

## Frequently Asked Questions

### What is the difference between Word2Vec and GloVe?

**Word2Vec** optimizes prediction-based objectives (predict context from word, or word from context), learning local co-occurrence patterns. **GloVe** factorizes a global word-context co-occurrence matrix, directly capturing corpus-wide statistics. Both produce dense semantic vectors, but GloVe often performs better on word analogy tasks while Word2Vec trains faster on large corpora.

### Should I train my own embeddings or use pre-trained ones?

Use **pre-trained embeddings** (Word2Vec Google News or GloVe 6B/840B) for most applications—they encode semantic knowledge from billions of tokens. Train **custom embeddings** only when your domain vocabulary diverges significantly from general English (e.g., medical, legal, or highly specialized technical text).

### How do I handle words not in the pre-trained vocabulary?

The Microsoft curriculum describes two approaches in [`14-Embeddings/README.md`](https://github.com/microsoft/AI-For-Beginners/blob/main/14-Embeddings/README.md): (1) **subset-only**—restrict your vocabulary to pre-trained words and initialize unknowns randomly, or (2) **fine-tuning**—load pre-trained vectors as initialization, then continue training on your corpus to learn representations for missing tokens.

### Can I fine-tune pre-trained Word2Vec or GloVe embeddings?

Yes. In TensorFlow, set `trainable=True` in the `Embedding` layer; in PyTorch, use `nn.Embedding.from_pretrained()` with `freeze=False`. Fine-tuning adapts general semantic vectors to your specific domain while preserving the learned relationships from the original training corpus.