# How to Train Custom Word Embeddings Using Word2Vec and GloVe

> Learn to train custom word embeddings with Word2Vec and GloVe. Integrate these powerful representations into TensorFlow or PyTorch to boost your NLP task performance. Get started today!

- Repository: [Microsoft/AI-For-Beginners](https://github.com/microsoft/AI-For-Beginners)
- Tags: tutorial
- Published: 2026-08-09

---

**You can train custom word embeddings using Word2Vec (via the gensim library or from-scratch CBOW implementations) and utilize pre-trained GloVe vectors via torchtext, then integrate these dense representations into TensorFlow or PyTorch embedding layers to improve downstream NLP task performance.**

The microsoft/AI-For-Beginners repository provides a comprehensive, hands-on curriculum for training custom word embeddings using Word2Vec and GloVe across multiple frameworks. The course covers both high-level implementations using established libraries and low-level neural architectures that reveal the underlying mechanics of vector space models.

## Word2Vec vs. GloVe: Architectural Differences

Understanding the distinction between these two approaches guides implementation choices.

### Word2Vec: Predictive Context Modeling

Word2Vec learns embeddings through predictive tasks using two architectures:

- **CBOW (Continuous Bag of Words)**: Predicts a target word from its surrounding context window
- **Skip-gram**: Predicts context words given a specific target word

As implemented in `lessons/5-NLP/14-Embeddings/EmbeddingsTF.ipynb`, these models capture local semantic relationships by adjusting vectors to minimize prediction loss.

### GloVe: Global Co-occurrence Statistics

**GloVe** (Global Vectors) leverages global corpus statistics by factorizing a word-word co-occurrence matrix. Unlike Word2Vec's local context window approach, GloVe captures broader statistical relationships by examining how frequently words appear together across the entire corpus.

## Training Word2Vec with Gensim

The repository demonstrates training custom Word2Vec embeddings using the **gensim** library, which handles corpus preprocessing and skip-gram/CBOW optimization internally.

```python
from gensim.models import Word2Vec

# Prepare tokenized corpus

sentences = [
    "the quick brown fox jumps over the lazy dog".split(),
    "I love natural language processing".split(),
    "word embeddings capture semantic similarity".split(),
]

# Train CBOW model (default)

model = Word2Vec(
    sentences, 
    vector_size=30,  # Starting with smaller dims for faster training

    window=5, 
    min_count=1, 
    workers=2
)

# Query learned representations

print(model.wv.most_similar("language"))

```

In `lessons/5-NLP/14-Embeddings/EmbeddingsTF.ipynb`, this approach provides immediate access to dense vectors via `model.wv[word]` and similarity queries through `model.wv.most_similar()`.

## Loading Pre-Trained GloVe Vectors

For GloVe implementations, the curriculum uses **torchtext** to access pre-trained vectors and integrate them into PyTorch workflows.

```python
import torch
import torchtext.vocab as vocab

# Load 6B token, 50-dimensional GloVe vectors

glove = vocab.GloVe(name='6B', dim=50)  # Downloads automatically

# Create embedding layer from pre-trained weights

embedding = torch.nn.Embedding.from_pretrained(glove.vectors)

# Tokenize and encode sample text

tokenizer = torchtext.data.utils.get_tokenizer('basic_english')
indices = [glove.stoi.get(tok, glove.stoi['unk']) 
           for tok in tokenizer("hello world")]
indices = torch.tensor(indices)

# Retrieve embeddings

embedded = embedding(indices)
print(embedded.shape)  # Output: (seq_len, 50)

```

This pattern from `lessons/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb` demonstrates direct matrix loading via `glove.vectors` and vocabulary mapping through `stoi` (string-to-index).

## Implementing CBOW from Scratch

To understand the underlying mechanics, the repository includes from-scratch CBOW implementations in both TensorFlow and PyTorch.

### TensorFlow CBOW Model

As shown in `lessons/5-NLP/15-LanguageModeling/CBoW-TF.ipynb`, you can construct the CBOW architecture using Keras layers:

```python
import tensorflow as tf
from tensorflow.keras import layers

# Context windows (batch_size=2, context_size=2)

contexts = tf.constant([[1, 2], [3, 4]], dtype='int32')
targets = tf.constant([5, 6], dtype='int32')

vocab_sz = 10
embed_sz = 30

# Architecture: Input -> Embedding -> Average -> Dense

input_ctx = layers.Input(shape=(2,), dtype='int32')
embed = layers.Embedding(vocab_sz, embed_sz)(input_ctx)
avg_embed = tf.reduce_mean(embed, axis=1)
output = layers.Dense(vocab_sz, activation='softmax')(avg_embed)

model = tf.keras.Model(inputs=input_ctx, outputs=output)
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy')
model.fit(contexts, targets, epochs=100, verbose=0)

# Extract trained embeddings

trained_embeddings = model.layers[1].get_weights()[0]

```

The PyTorch variant in `lessons/5-NLP/15-LanguageModeling/CBoW-PyTorch.ipynb` follows identical logic using `torch.nn.Embedding` and custom training loops.

## Integrating Custom Embeddings into Downstream Models

Once trained or loaded, embedding matrices must transfer into neural network architectures. The repository demonstrates this in `lessons/5-NLP/16-RNN/RNNPyTorch.ipynb`.

### Initializing Keras Layers with Word2Vec

```python
import numpy as np
from tensorflow.keras.layers import Embedding

# Assume 'model' is the trained Word2Vec from gensim

vocab_size = len(model.wv.key_to_index) + 1  # +1 for padding

embed_dim = model.wv.vector_size

# Build matrix with index 0 reserved for padding (zeros)

embedding_matrix = np.zeros((vocab_size, embed_dim))
for word, i in model.wv.key_to_index.items():
    embedding_matrix[i + 1] = model.wv[word]

# Create frozen embedding layer

embedding_layer = Embedding(
    input_dim=vocab_size,
    output_dim=embed_dim,
    weights=[embedding_matrix],
    trainable=False,  # Keep vectors fixed

)

```

This technique applies equally to GloVe vectors by substituting `glove.vectors` for the Word2Vec matrix.

## Practical Considerations for Custom Embeddings

### Dimensionality Selection

While production Word2Vec models typically use **300 dimensions**, the AI-For-Beginners curriculum recommends starting with **30-50 dimensions** to reduce training time during experimentation. You can scale up complexity after validating your pipeline.

### Out-of-Vocabulary Handling

When integrating pre-trained embeddings, words absent from the original vocabulary require initialization strategies:

- Initialize OOV tokens with zeros or small random values
- Expand vocabulary by re-training Word2Vec on your target corpus
- Use subword tokenization (Byte Pair Encoding) to minimize OOV instances

### Evaluation Methods

Validate embedding quality using intrinsic evaluation:

```python

# Similarity queries from EmbeddingsTF.ipynb

model.wv.most_similar(positive=['king', 'woman'], negative=['man'])

```

Semantic clustering indicates successful training—related words should appear in nearest neighbor searches.

## Summary

- **Word2Vec** trains via predictive objectives (CBOW or Skip-gram) using `gensim.models.Word2Vec` or custom neural architectures as shown in `CBoW-TF.ipynb`.
- **GloVe** provides pre-trained global co-occurrence vectors accessible through `torchtext.vocab.GloVe` in `EmbeddingsPyTorch.ipynb`.
- Initialize Keras `Embedding` layers or PyTorch `nn.Embedding` using extracted weight matrices via `Embedding(weights=[embedding_matrix], trainable=False)`.
- Start with smaller vector dimensions (30-50) for rapid prototyping, then scale to 300D for production models.
- Handle OOV words by reserving index 0 for padding and initializing missing tokens with appropriate default values.

## Frequently Asked Questions

### What is the difference between Word2Vec and GloVe?

Word2Vec learns embeddings by predicting words from local context windows (CBOW) or predicting context from words (Skip-gram), making it a predictive model. GloVe constructs embeddings by factorizing a global word-word co-occurrence matrix, making it a count-based model that captures corpus-wide statistical relationships. Both produce dense vector representations, but GloVe often performs better on word analogy tasks due to its global statistical perspective.

### How do I handle words not in my pre-trained embedding vocabulary?

Out-of-vocabulary (OOV) words can be handled by mapping them to a special `<unk>` token index, initializing them with random vectors or zeros, or expanding your vocabulary by re-training Word2Vec on your specific domain corpus. In the repository notebooks, the `glove.stoi.get(tok, glove.stoi['unk'])` pattern provides fallback handling for missing tokens.

### Can I fine-tune pre-trained Word2Vec or GloVe embeddings during model training?

Yes, by setting `trainable=True` in the Keras `Embedding` layer or setting `weight.requires_grad = True` in PyTorch, the pre-trained vectors will update during backpropagation. However, the curriculum recommends starting with `trainable=False` to preserve semantic relationships, then fine-tuning only if your downstream task has sufficient data to improve rather than degrade the representations.

### What vector dimension should I choose for custom word embeddings?

The repository suggests starting with **30 to 50 dimensions** for educational purposes and rapid iteration, then scaling to the standard **300 dimensions** (or 100-300 for GloVe) for production systems. Smaller dimensions reduce computational overhead and training time while still capturing sufficient semantic information for many NLP tasks.