How to Create Semantic Word Embeddings Using Word2Vec and GloVe in Python

Word2Vec learns dense vector representations by predicting words from their surrounding context (CBOW) or vice-versa (Skip-Gram), while GloVe factorizes global co-occurrence statistics—both techniques are demonstrated in the Microsoft AI-For-Beginners curriculum through hands-on TensorFlow and PyTorch notebooks.

The Microsoft AI-For-Beginners repository provides a complete educational pathway for creating and using semantic word embeddings. This guide walks through the core techniques—training custom Word2Vec embeddings from scratch and loading pre-trained Word2Vec and GloVe vectors—based on the actual implementation files in the curriculum.

---

Word2Vec: Training Custom Embeddings with CBOW

The repository teaches Word2Vec through the Continuous Bag-of-Words (CBOW) architecture, where the model predicts a target word from its surrounding context words. This forces the embedding layer to encode syntactic and semantic relationships.

CBOW Implementation in TensorFlow

The notebook lessons/5-NLP/15-LanguageModeling/CBoW-TF.ipynb implements CBOW training from scratch on the AG News dataset. Here's the core pattern:

import tensorflow as tf
from tensorflow.keras import layers

# Define the embedding layer that will be trained

embedder = layers.Embedding(input_dim=vocab_size,
                           output_dim=embedding_dim,
                           name='embedder')

# CBOW architecture: average context embeddings, predict target word

context_input = tf.keras.Input(shape=(context_len,), dtype='int32')
context_emb = embedder(context_input)                     # (batch, context_len, embed_dim)

context_vec = tf.reduce_mean(context_emb, axis=1)        # (batch, embed_dim)

# Output layer predicts the center word from vocabulary

output = layers.Dense(vocab_size, activation='softmax')(context_vec)

model = tf.keras.Model(inputs=context_input, outputs=output)
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy')

After training, extract the learned embedding matrix with embedder.get_weights()[0]. This matrix contains the semantic word embeddings that can be reused in downstream models.

PyTorch Equivalent

The parallel implementation in CBoW-PyTorch.ipynb uses torch.nn.Embedding with the same CBOW logic, demonstrating framework-agnostic understanding of the word2vec principle.

---

Loading Pre-Trained Word2Vec Vectors with Gensim

For production use, the repository emphasizes leveraging pre-trained embeddings. The TensorFlow notebook EmbeddingsTF.ipynb demonstrates loading Google's Word2Vec vectors via gensim:

import gensim.downloader as api

# Load 300-dimensional vectors trained on Google News (3 billion words)

word2vec = api.load('word2vec-google-news-300')

# Query semantic relationships

similar = word2vec.most_similar('neural', topn=5)
print(similar)

# Output: [('networks', 0.72), ('computational', 0.68), ...]

These vectors capture rich semantic properties—for example, vector arithmetic like king - man + woman ≈ queen.

---

GloVe: Global Vectors for Word Representation

GloVe (Global Vectors) combines the advantages of count-based matrix factorization and local context windows. It trains on global word-word co-occurrence statistics rather than prediction objectives.

Loading GloVe with Gensim (TensorFlow)

From EmbeddingsTF.ipynb (around line 443):

import gensim.downloader as api

# Load 100-dimensional GloVe vectors (Wikipedia + Gigaword corpus)

glove = api.load('glove-wiki-gigaword-100')

# Access individual word vectors

vector = glove['computer']
print(vector.shape)   # (100,)

Loading GloVe with TorchText (PyTorch)

The PyTorch notebook EmbeddingsPyTorch.ipynb uses the built-in torchtext.vocab.GloVe class:

import torchtext.vocab as vocab

# Download 6B-token, 50-dimensional GloVe vectors

glove = vocab.GloVe(name='6B', dim=50)

# Retrieve tensor directly

vector = glove['computer']      # torch.Tensor with shape (50,)

This approach handles downloading, caching, and vocabulary indexing automatically.

---

Integrating Pre-Trained Embeddings into Neural Models

A critical step covered in lessons/5-NLP/14-Embeddings/README.md is vocabulary alignment—pre-trained vocabularies rarely match your corpus exactly. The repository presents two strategies:

• Subset-only: Initialize only overlapping tokens from pre-trained vectors, set others to zero or random values
• Fine-tuning: Load pre-trained vectors then continue training on your corpus to adapt missing entries

Injecting into Keras Embedding Layer

From EmbeddingsTF.ipynb (lines 454-458):

# pretrained_matrix shape must match (vocab_size, embed_dim)

embedding_layer = layers.Embedding(
    input_dim=vocab_size,
    output_dim=embed_dim,
    weights=[pretrained_matrix],   # Load pre-trained GloVe/Word2Vec

    trainable=False                # Set True to fine-tune

)

PyTorch Alternative

import torch.nn as nn

# From pre-trained matrix tensor

embedding = nn.Embedding.from_pretrained(
    pretrained_matrix,
    freeze=False   # Allow fine-tuning

)

---

Key Files and Learning Path

File	Purpose
lessons/5-NLP/14-Embeddings/README.md	Conceptual foundation: Word2Vec vs. GloVe, vocabulary alignment strategies
lessons/5-NLP/14-Embeddings/EmbeddingsTF.ipynb	TensorFlow: gensim loading, Keras integration
lessons/5-NLP/14-Embeddings/EmbeddingsPyTorch.ipynb	PyTorch: torchtext.vocab.GloVe, from_pretrained
lessons/5-NLP/15-LanguageModeling/CBoW-TF.ipynb	End-to-end CBOW training from scratch
lessons/5-NLP/15-LanguageModeling/CBoW-PyTorch.ipynb	PyTorch CBOW implementation

These files progress from theory → custom training → pre-trained usage → downstream integration, forming a complete semantic word embedding workflow.

---

Summary

• Word2Vec CBOW predicts target words from context; train custom embeddings via CBoW-TF.ipynb or CBoW-PyTorch.ipynb
• GloVe uses global co-occurrence statistics; load via gensim.downloader (TensorFlow) or torchtext.vocab.GloVe (PyTorch)
• Pre-trained embeddings require vocabulary alignment; handle missing tokens through subsetting or fine-tuning
• Integration uses weights= parameter in Keras or from_pretrained() in PyTorch to inject semantic knowledge into any neural model

---

Frequently Asked Questions

What is the difference between Word2Vec and GloVe?

Word2Vec optimizes prediction-based objectives (predict context from word, or word from context), learning local co-occurrence patterns. GloVe factorizes a global word-context co-occurrence matrix, directly capturing corpus-wide statistics. Both produce dense semantic vectors, but GloVe often performs better on word analogy tasks while Word2Vec trains faster on large corpora.

Should I train my own embeddings or use pre-trained ones?

Use pre-trained embeddings (Word2Vec Google News or GloVe 6B/840B) for most applications—they encode semantic knowledge from billions of tokens. Train custom embeddings only when your domain vocabulary diverges significantly from general English (e.g., medical, legal, or highly specialized technical text).

How do I handle words not in the pre-trained vocabulary?

The Microsoft curriculum describes two approaches in 14-Embeddings/README.md: (1) subset-only—restrict your vocabulary to pre-trained words and initialize unknowns randomly, or (2) fine-tuning—load pre-trained vectors as initialization, then continue training on your corpus to learn representations for missing tokens.

Can I fine-tune pre-trained Word2Vec or GloVe embeddings?

Yes. In TensorFlow, set trainable=True in the Embedding layer; in PyTorch, use nn.Embedding.from_pretrained() with freeze=False. Fine-tuning adapts general semantic vectors to your specific domain while preserving the learned relationships from the original training corpus.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}