How to Train Custom Word Embeddings Using Word2Vec and GloVe

You can train custom word embeddings using Word2Vec (via the gensim library or from-scratch CBOW implementations) and utilize pre-trained GloVe vectors via torchtext, then integrate these dense representations into TensorFlow or PyTorch embedding layers to improve downstream NLP task performance.

The microsoft/AI-For-Beginners repository provides a comprehensive, hands-on curriculum for training custom word embeddings using Word2Vec and GloVe across multiple frameworks. The course covers both high-level implementations using established libraries and low-level neural architectures that reveal the underlying mechanics of vector space models.

Word2Vec vs. GloVe: Architectural Differences

Understanding the distinction between these two approaches guides implementation choices.

Word2Vec: Predictive Context Modeling

Word2Vec learns embeddings through predictive tasks using two architectures:

• CBOW (Continuous Bag of Words): Predicts a target word from its surrounding context window
• Skip-gram: Predicts context words given a specific target word

As implemented in lessons/5-NLP/14-Embeddings/EmbeddingsTF.ipynb, these models capture local semantic relationships by adjusting vectors to minimize prediction loss.

GloVe: Global Co-occurrence Statistics

GloVe (Global Vectors) leverages global corpus statistics by factorizing a word-word co-occurrence matrix. Unlike Word2Vec's local context window approach, GloVe captures broader statistical relationships by examining how frequently words appear together across the entire corpus.

Training Word2Vec with Gensim

The repository demonstrates training custom Word2Vec embeddings using the gensim library, which handles corpus preprocessing and skip-gram/CBOW optimization internally.

from gensim.models import Word2Vec

# Prepare tokenized corpus

sentences = [
    "the quick brown fox jumps over the lazy dog".split(),
    "I love natural language processing".split(),
    "word embeddings capture semantic similarity".split(),
]

# Train CBOW model (default)

model = Word2Vec(
    sentences, 
    vector_size=30,  # Starting with smaller dims for faster training

    window=5, 
    min_count=1, 
    workers=2
)

# Query learned representations

print(model.wv.most_similar("language"))

In lessons/5-NLP/14-Embeddings/EmbeddingsTF.ipynb, this approach provides immediate access to dense vectors via model.wv[word] and similarity queries through model.wv.most_similar().

Loading Pre-Trained GloVe Vectors

For GloVe implementations, the curriculum uses torchtext to access pre-trained vectors and integrate them into PyTorch workflows.

import torch
import torchtext.vocab as vocab

# Load 6B token, 50-dimensional GloVe vectors

glove = vocab.GloVe(name='6B', dim=50)  # Downloads automatically

# Create embedding layer from pre-trained weights

embedding = torch.nn.Embedding.from_pretrained(glove.vectors)

# Tokenize and encode sample text

tokenizer = torchtext.data.utils.get_tokenizer('basic_english')
indices = [glove.stoi.get(tok, glove.stoi['unk']) 
           for tok in tokenizer("hello world")]
indices = torch.tensor(indices)

# Retrieve embeddings

embedded = embedding(indices)
print(embedded.shape)  # Output: (seq_len, 50)

This pattern from lessons/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb demonstrates direct matrix loading via glove.vectors and vocabulary mapping through stoi (string-to-index).

Implementing CBOW from Scratch

To understand the underlying mechanics, the repository includes from-scratch CBOW implementations in both TensorFlow and PyTorch.

TensorFlow CBOW Model

As shown in lessons/5-NLP/15-LanguageModeling/CBoW-TF.ipynb, you can construct the CBOW architecture using Keras layers:

import tensorflow as tf
from tensorflow.keras import layers

# Context windows (batch_size=2, context_size=2)

contexts = tf.constant([[1, 2], [3, 4]], dtype='int32')
targets = tf.constant([5, 6], dtype='int32')

vocab_sz = 10
embed_sz = 30

# Architecture: Input -> Embedding -> Average -> Dense

input_ctx = layers.Input(shape=(2,), dtype='int32')
embed = layers.Embedding(vocab_sz, embed_sz)(input_ctx)
avg_embed = tf.reduce_mean(embed, axis=1)
output = layers.Dense(vocab_sz, activation='softmax')(avg_embed)

model = tf.keras.Model(inputs=input_ctx, outputs=output)
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy')
model.fit(contexts, targets, epochs=100, verbose=0)

# Extract trained embeddings

trained_embeddings = model.layers[1].get_weights()[0]

The PyTorch variant in lessons/5-NLP/15-LanguageModeling/CBoW-PyTorch.ipynb follows identical logic using torch.nn.Embedding and custom training loops.

Integrating Custom Embeddings into Downstream Models

Once trained or loaded, embedding matrices must transfer into neural network architectures. The repository demonstrates this in lessons/5-NLP/16-RNN/RNNPyTorch.ipynb.

Initializing Keras Layers with Word2Vec

import numpy as np
from tensorflow.keras.layers import Embedding

# Assume 'model' is the trained Word2Vec from gensim

vocab_size = len(model.wv.key_to_index) + 1  # +1 for padding

embed_dim = model.wv.vector_size

# Build matrix with index 0 reserved for padding (zeros)

embedding_matrix = np.zeros((vocab_size, embed_dim))
for word, i in model.wv.key_to_index.items():
    embedding_matrix[i + 1] = model.wv[word]

# Create frozen embedding layer

embedding_layer = Embedding(
    input_dim=vocab_size,
    output_dim=embed_dim,
    weights=[embedding_matrix],
    trainable=False,  # Keep vectors fixed

)

This technique applies equally to GloVe vectors by substituting glove.vectors for the Word2Vec matrix.

Practical Considerations for Custom Embeddings

Dimensionality Selection

While production Word2Vec models typically use 300 dimensions, the AI-For-Beginners curriculum recommends starting with 30-50 dimensions to reduce training time during experimentation. You can scale up complexity after validating your pipeline.

Out-of-Vocabulary Handling

When integrating pre-trained embeddings, words absent from the original vocabulary require initialization strategies:

• Initialize OOV tokens with zeros or small random values
• Expand vocabulary by re-training Word2Vec on your target corpus
• Use subword tokenization (Byte Pair Encoding) to minimize OOV instances

Evaluation Methods

Validate embedding quality using intrinsic evaluation:

# Similarity queries from EmbeddingsTF.ipynb

model.wv.most_similar(positive=['king', 'woman'], negative=['man'])

Semantic clustering indicates successful training—related words should appear in nearest neighbor searches.

Summary

• Word2Vec trains via predictive objectives (CBOW or Skip-gram) using gensim.models.Word2Vec or custom neural architectures as shown in CBoW-TF.ipynb.
• GloVe provides pre-trained global co-occurrence vectors accessible through torchtext.vocab.GloVe in EmbeddingsPyTorch.ipynb.
• Initialize Keras Embedding layers or PyTorch nn.Embedding using extracted weight matrices via Embedding(weights=[embedding_matrix], trainable=False).
• Start with smaller vector dimensions (30-50) for rapid prototyping, then scale to 300D for production models.
• Handle OOV words by reserving index 0 for padding and initializing missing tokens with appropriate default values.

Frequently Asked Questions

What is the difference between Word2Vec and GloVe?

Word2Vec learns embeddings by predicting words from local context windows (CBOW) or predicting context from words (Skip-gram), making it a predictive model. GloVe constructs embeddings by factorizing a global word-word co-occurrence matrix, making it a count-based model that captures corpus-wide statistical relationships. Both produce dense vector representations, but GloVe often performs better on word analogy tasks due to its global statistical perspective.

How do I handle words not in my pre-trained embedding vocabulary?

Out-of-vocabulary (OOV) words can be handled by mapping them to a special <unk> token index, initializing them with random vectors or zeros, or expanding your vocabulary by re-training Word2Vec on your specific domain corpus. In the repository notebooks, the glove.stoi.get(tok, glove.stoi['unk']) pattern provides fallback handling for missing tokens.

Can I fine-tune pre-trained Word2Vec or GloVe embeddings during model training?

Yes, by setting trainable=True in the Keras Embedding layer or setting weight.requires_grad = True in PyTorch, the pre-trained vectors will update during backpropagation. However, the curriculum recommends starting with trainable=False to preserve semantic relationships, then fine-tuning only if your downstream task has sufficient data to improve rather than degrade the representations.

What vector dimension should I choose for custom word embeddings?

The repository suggests starting with 30 to 50 dimensions for educational purposes and rapid iteration, then scaling to the standard 300 dimensions (or 100-300 for GloVe) for production systems. Smaller dimensions reduce computational overhead and training time while still capturing sufficient semantic information for many NLP tasks.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}