How to Train Word Embeddings Using Word2Vec and GloVe: A Complete Guide for Beginners

You can train Word2Vec embeddings using the gensim library on a custom text corpus or load pre-trained GloVe vectors via torchtext to initialize neural network layers, as demonstrated in Microsoft's AI-For-Beginners curriculum.

Word embeddings transform sparse, high-dimensional text representations into dense, low-dimensional vectors that capture semantic meaning. This article examines how the microsoft/AI-For-Beginners repository implements both prediction-based and count-based approaches to train word embeddings using Word2Vec and GloVe, providing practical code examples from the actual course materials.

Understanding Word Embedding Architectures

The repository teaches two complementary approaches to learning vector representations.

Word2Vec uses shallow neural networks to learn embeddings by predicting words from context. The implementation covers two architectures:

• CBOW (Continuous Bag of Words): Predicts a target word from its surrounding context window
• Skip-gram: Predicts surrounding context words from a target word

GloVe (Global Vectors) takes a matrix factorization approach, decomposing a word co-occurrence matrix to capture global statistical information about word relationships across the entire corpus.

Both methods follow a shared pipeline in lessons/5-NLP/14-Embeddings/: corpus tokenization, embedding matrix creation, and integration with downstream neural models.

Training Word2Vec Embeddings with Gensim

The TensorFlow notebook lessons/5-NLP/14-Embeddings/EmbeddingsTF.ipynb demonstrates training custom Word2Vec models using the gensim library.

First, prepare your tokenized corpus:

from gensim.models import Word2Vec

sentences = [
    "the quick brown fox jumps over the lazy dog".split(),
    "I love natural language processing".split(),
    "word embeddings capture semantic similarity".split(),
]

Train the model with your desired dimensionality and context window:

model = Word2Vec(
    sentences, 
    vector_size=30,  # Dimensionality of vectors

    window=5,        # Context window size

    min_count=1,     # Minimum word frequency

    workers=2
)

Access trained vectors and query similarity:

vector = model.wv["language"]
similar_words = model.wv.most_similar("language")

The model.wv object serves as a key-to-index mapping that allows you to extract the embedding matrix for integration with neural networks.

Loading Pre-Trained GloVe Vectors with Torchtext

For GloVe, the repository in lessons/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb uses torchtext to load pre-trained global vectors instead of training from scratch.

Load the pre-trained vectors:

import torch
import torchtext.vocab as vocab

glove = vocab.GloVe(name='6B', dim=50)  # Downloads 50-dimensional vectors

Convert the GloVe vectors into a PyTorch embedding layer:

embedding = torch.nn.Embedding.from_pretrained(glove.vectors)

Tokenize and encode new text:

tokenizer = torchtext.data.utils.get_tokenizer('basic_english')
indices = [glove.stoi.get(tok, glove.stoi['unk']) for tok in tokenizer("hello world")]
indices = torch.tensor(indices)
embedded = embedding(indices)

This approach leverages billions of pre-computed co-occurrence statistics from large corpora, providing robust representations without requiring extensive training data.

Integrating Embeddings into Neural Networks

Once you have Word2Vec or GloVe vectors, you must integrate them into downstream models. The repository shows two primary patterns.

For Word2Vec in Keras, build an embedding matrix from the trained gensim model:

import numpy as np
import tensorflow as tf
from tensorflow.keras.layers import Embedding

vocab_size = len(model.wv.key_to_index) + 1  # +1 for padding token

embed_dim = model.wv.vector_size

embedding_matrix = np.zeros((vocab_size, embed_dim))
for word, i in model.wv.key_to_index.items():
    embedding_matrix[i + 1] = model.wv[word]

embedding_layer = Embedding(
    input_dim=vocab_size,
    output_dim=embed_dim,
    weights=[embedding_matrix],
    trainable=False,  # Set True to fine-tune

)

For GloVe in PyTorch, use torch.nn.Embedding.from_pretrained() as shown in lessons/5-NLP/16-RNN/RNNPyTorch.ipynb, which demonstrates replacing randomly initialized layers with pre-trained vectors to improve RNN performance on sentiment analysis tasks.

Building CBOW from Scratch in TensorFlow

To understand the internal mechanics, lessons/5-NLP/15-LanguageModeling/CBoW-TF.ipynb implements CBOW training without external libraries.

Define the architecture using Keras layers:

import tensorflow as tf
from tensorflow.keras import layers

vocab_sz = 10
embed_sz = 30

input_ctx = layers.Input(shape=(2,), dtype='int32')
embed = layers.Embedding(vocab_sz, embed_sz, input_length=2)(input_ctx)
avg_embed = tf.reduce_mean(embed, axis=1)
output = layers.Dense(vocab_sz, activation='softmax')(avg_embed)

cbow_model = tf.keras.Model(inputs=input_ctx, outputs=output)
cbow_model.compile(optimizer='adam', loss='sparse_categorical_crossentropy')

Train on context-target pairs:

contexts = tf.constant([[1, 2], [3, 4]], dtype=tf.int32)
targets = tf.constant([5, 6], dtype=tf.int32)

cbow_model.fit(contexts, targets, epochs=100, verbose=0)

Extract learned embeddings from the weights:

trained_embeddings = cbow_model.layers[1].get_weights()[0]

This explicit implementation reveals how the averaging of context embeddings and prediction of target words drives the learning process.

Practical Considerations for Training Embeddings

The curriculum highlights several implementation details that affect model performance.

Dimensionality: While production Word2Vec models typically use 300 dimensions, the notebooks start with 30-50 dimensions to accelerate training during experimentation.

Out-of-Vocabulary (OOV) Handling: Words not present in the training corpus require special handling. Options include initializing with zeros, random values, or expanding vocabulary by retraining on domain-specific text.

Evaluation: Semantic quality is verified through model.wv.most_similar() queries, which should return semantically related words (e.g., "king" and "queen" clustering near each other).

Summary

• Word2Vec trains embeddings by predicting words from context using CBOW or Skip-gram architectures, implemented via gensim or custom TensorFlow models in CBoW-TF.ipynb.
• GloVe provides pre-trained global vectors via torchtext, offering immediate semantic richness without training from scratch.
• Both approaches require tokenization and vocabulary mapping before creating embedding matrices compatible with Embedding layers in Keras or PyTorch.
• Source files lessons/5-NLP/14-Embeddings/EmbeddingsTF.ipynb and EmbeddingsPyTorch.ipynb contain the primary implementations for TensorFlow and PyTorch respectively.

Frequently Asked Questions

What is the difference between Word2Vec and GloVe embeddings?

Word2Vec is a predictive model that learns embeddings by training neural networks to predict words from their neighbors (CBOW) or neighbors from words (Skip-gram). GloVe is a count-based model that factorizes a global word co-occurrence matrix to produce vectors. Word2Vec focuses on local context windows, while GloVe captures global corpus statistics.

How long does it take to train Word2Vec embeddings from scratch?

Training time depends on corpus size, vector dimensionality, and window size. For small corpora (thousands of sentences), training completes in seconds to minutes on CPU. The AI-For-Beginners notebooks use small dimensions (30-50) and limited epochs to ensure the training completes interactively during lessons.

How do you handle words that are not in the Word2Vec vocabulary?

Out-of-vocabulary (OOV) words can be handled by initializing them with random vectors or zeros, adding a special <UNK> token to the vocabulary, or expanding the vocabulary by retraining the model on your specific domain corpus. The torchtext GloVe implementation provides a built-in 'unk' token index for unknown words.

Can you use Word2Vec and GloVe vectors together in the same model?

While technically possible by concatenating or averaging vectors, it is not standard practice because the two methods produce vectors in different semantic spaces with different scales. Typically, you choose one pre-trained embedding source and use it consistently throughout your model architecture.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}