# How to Implement Language Modeling and Text Generation Using CBoW-PyTorch

> Learn to implement CBoW PyTorch for language modeling and text generation. Build a model that predicts words using context and generates new text.

- Repository: [Microsoft/AI-For-Beginners](https://github.com/microsoft/AI-For-Beginners)
- Tags: how-to-guide
- Published: 2026-08-08

---

**The CBoW-PyTorch.ipynb notebook demonstrates how to build a Continuous Bag-of-Words language model in PyTorch that learns word embeddings from context windows and generates text by predicting target words from averaged surrounding vectors.**

The microsoft/AI-For-Beginners repository provides a hands-on introduction to natural language processing through practical Jupyter notebooks. In `lessons/5-NLP/15-LanguageModeling/CBoW-PyTorch.ipynb`, you will implement a foundational neural language model that predicts a target word from its surrounding context, serving as an accessible entry point into language modeling and text generation using CBoW-PyTorch architectures.

## Loading the AG News Corpus and Building Vocabulary

The notebook begins by ingesting the AG News classification dataset using `torchtext.datasets.AG_NEWS`. This corpus provides news articles that serve as training data for the embedding layer.

The data pipeline uses `torchtext.data.utils.get_tokenizer('basic_english')` to split text into lower-cased tokens. To maintain computational efficiency, the vocabulary is restricted to the 5,000 most frequent tokens using `torchtext.vocab.Vocab`, preventing the embedding matrix from growing unwieldy.

```python
train_dataset, test_dataset, _, vocab, tokenizer = load_dataset()

```

The `load_dataset()` helper function handles the download, tokenization, and vocabulary construction, returning ready-to-use dataset objects and a mapping between tokens and integer indices.

## Generating CBoW Training Pairs with Sliding Windows

The CBoW architecture requires (context, target) pairs created via a sliding window approach. The notebook implements this in the `to_cbow()` function, which iterates over tokenized sentences with a configurable window size (defaulting to 2).

For each position in a sentence, the function collects neighboring words as context indices and identifies the central word as the target label. The `encode()` function converts token strings to integer indices using the vocabulary mapping.

```python
for w1, w2 in to_cbow(encode(sentence, vocab), window_size=5):
    X.append(w1)   # context word indices

    Y.append(w2)   # target word index

```

This process transforms the raw text into supervised learning examples where the model must predict the center word given its surrounding context.

## Building the CBoW Neural Architecture in PyTorch

The model architecture in `CBoW-PyTorch.ipynb` consists of two core layers implemented using `torch.nn` modules:

- **Embedding Layer** (`nn.Embedding`): Maps each vocabulary token to a dense 30-dimensional vector representation. This layer learns the Word2Vec-style embeddings during training.
- **Linear Projection** (`nn.Linear`): Transforms the averaged context embedding into a `vocab_size` dimensional vector containing logits for each possible target word.

```python
embedder = torch.nn.Embedding(num_embeddings=vocab_size,
                              embedding_dim=30)
model = torch.nn.Sequential(
    embedder,
    torch.nn.Linear(in_features=30, out_features=vocab_size)
)

```

The loss function uses `torch.nn.CrossEntropyLoss`, which accepts raw logits and integer class labels without requiring one-hot encoding. This architecture enforces the CBoW assumption that context words contribute equally to the target prediction through simple averaging.

## Training the Language Model

Training utilizes a custom `SimpleIterableDataset` class that stores (label, feature) tuples and supports shuffling. The `torch.utils.data.DataLoader` streams batches of size 256 through the network.

The `train_epoch()` helper function manages the optimization loop, applying stochastic gradient descent or Adam to minimize the cross-entropy loss across multiple epochs.

```python
train_epoch(net=model,
            dataloader=dl,
            optimizer=torch.optim.SGD(model.parameters(), lr=0.1),
            loss_fn=torch.nn.CrossEntropyLoss(),
            epochs=10)

```

After approximately 10 epochs, the loss typically converges to around 5.5, indicating the model has learned to predict high-frequency words based on their contextual patterns.

## Extracting Word2Vec Embeddings

Once training completes, the embedding matrix contains learned vector representations for each vocabulary token. You can extract these vectors by indexing into the `embedder` layer with vocabulary tokens:

```python
vectors = torch.stack([embedder(torch.tensor(vocab[s]))
                      for s in vocab.itos], dim=0)

```

Each row in the resulting tensor corresponds to a word vector suitable for similarity queries, analogical reasoning, or visualization techniques like t-SNE. These embeddings capture semantic relationships learned from the co-occurrence statistics in the AG News corpus.

## Implementing Text Generation

While the notebook primarily focuses on embedding learning, the trained CBoW model supports basic text generation through iterative prediction. The generation process works by:

1. **Seeding the context** with an initial sequence of word indices
2. **Averaging embeddings** of the current context window using `embedder(context).mean(0)`
3. **Projecting through the linear layer** to obtain logits
4. **Sampling from the distribution** using `torch.nn.functional.softmax` and `torch.multinomial`

```python
def generate_text(seed, length=20):
    generated = list(seed)
    for _ in range(length):
        ctx = torch.tensor(generated[-window_size:]).to(device)
        logits = model(ctx).mean(0)          # average over context embeddings

        probs = torch.nn.functional.softmax(logits, dim=0)
        next_idx = torch.multinomial(probs, 1).item()
        generated.append(next_idx)
    return " ".join(vocab.itos[i] for i in generated)

```

Calling `generate_text([vocab['i'], vocab['like']])` produces a sequence of plausible continuation words based on the distributional patterns learned during training. While less sophisticated than transformer-based generators, this approach demonstrates how CBoW embeddings encode contextual coherence.

## Summary

- **Data Preparation**: The notebook loads AG News via `torchtext`, tokenizes with `basic_english`, and constrains vocabulary to 5,000 tokens for efficiency.
- **Pair Generation**: The `to_cbow()` function creates training examples using sliding windows of configurable size, treating neighboring words as context and center words as targets.
- **Architecture**: A two-layer network combines `nn.Embedding` (30-dimensional vectors) with a linear projection to `vocab_size` logits, optimized with `CrossEntropyLoss`.
- **Training**: The `train_epoch()` function processes batches of 256 examples through SGD, typically achieving convergence within 10 epochs.
- **Embeddings**: Post-training, the embedding layer provides Word2Vec-style vectors accessible via `vocab.itos` indexing.
- **Generation**: Text generation averages context embeddings and samples from the softmax distribution over vocabulary items.

## Frequently Asked Questions

### What is the difference between CBoW and Skip-gram models?

CBoW (Continuous Bag-of-Words) predicts a target word from its surrounding context by averaging the embedding vectors of context words. Skip-gram operates in reverse, predicting context words from a target word. According to the AI-For-Beginners curriculum, CBoW trains faster and works better with frequent words, while Skip-gram excels with rare words and captures more nuanced semantic relationships.

### How does the window size affect CBoW training in PyTorch?

The window size determines how many surrounding words contribute to each prediction. In `CBoW-PyTorch.ipynb`, the default window size of 2 captures immediate neighbors, while increasing to 5 incorporates broader context. Larger windows provide more training data per sentence but dilute the specificity of local context, potentially requiring more epochs or larger embedding dimensions to converge effectively.

### Can I use the trained CBoW embeddings for downstream NLP tasks?

Yes. The 30-dimensional vectors learned by the `nn.Embedding` layer in this notebook can initialize embeddings for classification, named entity recognition, or sentiment analysis models. For production applications, increasing `embedding_dim` to 100-300 dimensions and training on the full AG News corpus (removing the `lines_cnt` limit) typically yields higher-quality representations suitable for transfer learning.

### Why does the CBoW model use averaging instead of concatenation for context words?

CBoW assumes that context words contribute equally to the meaning of the target word regardless of their position. Averaging the embedding vectors of context words (as implemented in the `generate_text()` function via `mean(0)`) maintains permutation invariance and reduces the input dimensionality to match the embedding size. This architectural choice distinguishes CBoW from position-sensitive models like transformers or recurrent neural networks.