# How to Train Word Embeddings Using Word2Vec and GloVe: A Complete Guide for Beginners

> Learn to train Word2Vec embeddings with gensim or load GloVe vectors using torchtext. This guide from Microsoft's AI-For-Beginners simplifies word embedding training for newcomers.

- Repository: [Microsoft/AI-For-Beginners](https://github.com/microsoft/AI-For-Beginners)
- Tags: how-to-guide
- Published: 2026-08-06

---

**You can train Word2Vec embeddings using the `gensim` library on a custom text corpus or load pre-trained GloVe vectors via `torchtext` to initialize neural network layers, as demonstrated in Microsoft's AI-For-Beginners curriculum.**

Word embeddings transform sparse, high-dimensional text representations into dense, low-dimensional vectors that capture semantic meaning. This article examines how the **microsoft/AI-For-Beginners** repository implements both prediction-based and count-based approaches to train word embeddings using Word2Vec and GloVe, providing practical code examples from the actual course materials.

## Understanding Word Embedding Architectures

The repository teaches two complementary approaches to learning vector representations.

**Word2Vec** uses shallow neural networks to learn embeddings by predicting words from context. The implementation covers two architectures:
- **CBOW (Continuous Bag of Words)**: Predicts a target word from its surrounding context window
- **Skip-gram**: Predicts surrounding context words from a target word

**GloVe (Global Vectors)** takes a matrix factorization approach, decomposing a word co-occurrence matrix to capture global statistical information about word relationships across the entire corpus.

Both methods follow a shared pipeline in `lessons/5-NLP/14-Embeddings/`: corpus tokenization, embedding matrix creation, and integration with downstream neural models.

## Training Word2Vec Embeddings with Gensim

The TensorFlow notebook `lessons/5-NLP/14-Embeddings/EmbeddingsTF.ipynb` demonstrates training custom Word2Vec models using the `gensim` library.

First, prepare your tokenized corpus:

```python
from gensim.models import Word2Vec

sentences = [
    "the quick brown fox jumps over the lazy dog".split(),
    "I love natural language processing".split(),
    "word embeddings capture semantic similarity".split(),
]

```

Train the model with your desired dimensionality and context window:

```python
model = Word2Vec(
    sentences, 
    vector_size=30,  # Dimensionality of vectors

    window=5,        # Context window size

    min_count=1,     # Minimum word frequency

    workers=2
)

```

Access trained vectors and query similarity:

```python
vector = model.wv["language"]
similar_words = model.wv.most_similar("language")

```

The `model.wv` object serves as a key-to-index mapping that allows you to extract the embedding matrix for integration with neural networks.

## Loading Pre-Trained GloVe Vectors with Torchtext

For GloVe, the repository in `lessons/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb` uses `torchtext` to load pre-trained global vectors instead of training from scratch.

Load the pre-trained vectors:

```python
import torch
import torchtext.vocab as vocab

glove = vocab.GloVe(name='6B', dim=50)  # Downloads 50-dimensional vectors

```

Convert the GloVe vectors into a PyTorch embedding layer:

```python
embedding = torch.nn.Embedding.from_pretrained(glove.vectors)

```

Tokenize and encode new text:

```python
tokenizer = torchtext.data.utils.get_tokenizer('basic_english')
indices = [glove.stoi.get(tok, glove.stoi['unk']) for tok in tokenizer("hello world")]
indices = torch.tensor(indices)
embedded = embedding(indices)

```

This approach leverages billions of pre-computed co-occurrence statistics from large corpora, providing robust representations without requiring extensive training data.

## Integrating Embeddings into Neural Networks

Once you have Word2Vec or GloVe vectors, you must integrate them into downstream models. The repository shows two primary patterns.

**For Word2Vec in Keras**, build an embedding matrix from the trained `gensim` model:

```python
import numpy as np
import tensorflow as tf
from tensorflow.keras.layers import Embedding

vocab_size = len(model.wv.key_to_index) + 1  # +1 for padding token

embed_dim = model.wv.vector_size

embedding_matrix = np.zeros((vocab_size, embed_dim))
for word, i in model.wv.key_to_index.items():
    embedding_matrix[i + 1] = model.wv[word]

embedding_layer = Embedding(
    input_dim=vocab_size,
    output_dim=embed_dim,
    weights=[embedding_matrix],
    trainable=False,  # Set True to fine-tune

)

```

**For GloVe in PyTorch**, use `torch.nn.Embedding.from_pretrained()` as shown in `lessons/5-NLP/16-RNN/RNNPyTorch.ipynb`, which demonstrates replacing randomly initialized layers with pre-trained vectors to improve RNN performance on sentiment analysis tasks.

## Building CBOW from Scratch in TensorFlow

To understand the internal mechanics, `lessons/5-NLP/15-LanguageModeling/CBoW-TF.ipynb` implements CBOW training without external libraries.

Define the architecture using Keras layers:

```python
import tensorflow as tf
from tensorflow.keras import layers

vocab_sz = 10
embed_sz = 30

input_ctx = layers.Input(shape=(2,), dtype='int32')
embed = layers.Embedding(vocab_sz, embed_sz, input_length=2)(input_ctx)
avg_embed = tf.reduce_mean(embed, axis=1)
output = layers.Dense(vocab_sz, activation='softmax')(avg_embed)

cbow_model = tf.keras.Model(inputs=input_ctx, outputs=output)
cbow_model.compile(optimizer='adam', loss='sparse_categorical_crossentropy')

```

Train on context-target pairs:

```python
contexts = tf.constant([[1, 2], [3, 4]], dtype=tf.int32)
targets = tf.constant([5, 6], dtype=tf.int32)

cbow_model.fit(contexts, targets, epochs=100, verbose=0)

```

Extract learned embeddings from the weights:

```python
trained_embeddings = cbow_model.layers[1].get_weights()[0]

```

This explicit implementation reveals how the averaging of context embeddings and prediction of target words drives the learning process.

## Practical Considerations for Training Embeddings

The curriculum highlights several implementation details that affect model performance.

**Dimensionality**: While production Word2Vec models typically use 300 dimensions, the notebooks start with 30-50 dimensions to accelerate training during experimentation.

**Out-of-Vocabulary (OOV) Handling**: Words not present in the training corpus require special handling. Options include initializing with zeros, random values, or expanding vocabulary by retraining on domain-specific text.

**Evaluation**: Semantic quality is verified through `model.wv.most_similar()` queries, which should return semantically related words (e.g., "king" and "queen" clustering near each other).

## Summary

- **Word2Vec** trains embeddings by predicting words from context using CBOW or Skip-gram architectures, implemented via `gensim` or custom TensorFlow models in `CBoW-TF.ipynb`.
- **GloVe** provides pre-trained global vectors via `torchtext`, offering immediate semantic richness without training from scratch.
- Both approaches require tokenization and vocabulary mapping before creating embedding matrices compatible with `Embedding` layers in Keras or PyTorch.
- Source files `lessons/5-NLP/14-Embeddings/EmbeddingsTF.ipynb` and `EmbeddingsPyTorch.ipynb` contain the primary implementations for TensorFlow and PyTorch respectively.

## Frequently Asked Questions

### What is the difference between Word2Vec and GloVe embeddings?

Word2Vec is a predictive model that learns embeddings by training neural networks to predict words from their neighbors (CBOW) or neighbors from words (Skip-gram). GloVe is a count-based model that factorizes a global word co-occurrence matrix to produce vectors. Word2Vec focuses on local context windows, while GloVe captures global corpus statistics.

### How long does it take to train Word2Vec embeddings from scratch?

Training time depends on corpus size, vector dimensionality, and window size. For small corpora (thousands of sentences), training completes in seconds to minutes on CPU. The AI-For-Beginners notebooks use small dimensions (30-50) and limited epochs to ensure the training completes interactively during lessons.

### How do you handle words that are not in the Word2Vec vocabulary?

Out-of-vocabulary (OOV) words can be handled by initializing them with random vectors or zeros, adding a special `<UNK>` token to the vocabulary, or expanding the vocabulary by retraining the model on your specific domain corpus. The `torchtext` GloVe implementation provides a built-in 'unk' token index for unknown words.

### Can you use Word2Vec and GloVe vectors together in the same model?

While technically possible by concatenating or averaging vectors, it is not standard practice because the two methods produce vectors in different semantic spaces with different scales. Typically, you choose one pre-trained embedding source and use it consistently throughout your model architecture.