How to Implement Recurrent Neural Networks for Text Generation with RNNPyTorch.ipynb

The RNNPyTorch.ipynb notebook in Microsoft's AI-For-Beginners repository provides a complete PyTorch implementation of recurrent neural networks that can be adapted from sequence classification to autoregressive text generation by replacing the output head and modifying the training objective.

This tutorial walks through the end-to-end workflow found in lessons/5-NLP/16-RNN/RNNPyTorch.ipynb from the microsoft/AI-For-Beginners curriculum. You will learn how to leverage the notebook's LSTMClassifier architecture, train_epoch loop, and packed sequence utilities to build models that generate text token-by-token.

Understanding the RNN Architecture in RNNPyTorch.ipynb

The notebook demonstrates three variants of recurrent networks for processing sequential text data. Each implementation follows a consistent pattern: embedding layer → recurrent cell → aggregation → classification head.

Data Loading and Vocabulary Construction

The notebook utilizes the torchnlp utility module (torchnlp.py) to load a news-category dataset and construct a token-to-index vocabulary. The padify function handles variable-length sequences by padding them to uniform tensors suitable for batching.

# Conceptual flow from the notebook's data preparation

vocab = build_vocab(texts)  # Maps tokens to indices

padded_batch = padify(raw_texts, vocab)  # Returns (batch, seq_len) tensor

Embedding and Recurrent Layers

Every model begins with torch.nn.Embedding to convert token indices into dense vectors, followed by a recurrent cell. The notebook compares three PyTorch implementations:

• Simple RNN: torch.nn.RNN processes sequences step-by-step but suffers from vanishing gradients on long texts.
• LSTM: torch.nn.LSTM introduces forget, input, and output gates to stabilize gradient flow across longer sequences.
• GRU: A gated alternative mentioned in the notebook's markdown cells.

All cells share weights across time steps, propagating a hidden state Sᵢ that captures temporal dependencies.

Classification Head vs. Generation Head

The notebook's default implementations (RNNClassifier and LSTMClassifier) aggregate hidden states via mean pooling or final-state extraction, then project to class logits using torch.nn.Linear. For text generation, you replace this classification head with a vocabulary projection layer that predicts the next token probability distribution.

Building Text Generation Models from the Classification Base

While RNNPyTorch.ipynb focuses on sequence classification, the same architectural components support autoregressive text generation with minimal modifications.

Converting the Classifier to a Generator

Replace the class-projection layer with a vocabulary projection and adjust the forward pass to return per-token logits:

import torch
import torch.nn as nn

class LSTMGenerator(nn.Module):
    def __init__(self, vocab_size, embed_dim, hidden_dim):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim)
        self.rnn = nn.LSTM(embed_dim, hidden_dim, batch_first=True)
        self.fc = nn.Linear(hidden_dim, vocab_size)  # next-token logits

    def forward(self, x, lengths=None):
        x = self.embedding(x)
        if lengths is not None:
            x = nn.utils.rnn.pack_padded_sequence(
                x, lengths, 
                batch_first=True, 
                enforce_sorted=False
            )
            x, (h, c) = self.rnn(x)
            x, _ = nn.utils.rnn.pad_packed_sequence(x, batch_first=True)
        else:
            x, (h, c) = self.rnn(x)
        return self.fc(x)  # shape: (batch, seq_len, vocab_size)

Autoregressive Inference Loop

During generation, feed the model's output back as the next input, sampling from the probability distribution until an end-of-sequence token or length limit is reached:

def generate(model, start_token, vocab, max_len=50, temperature=1.0):
    model.eval()
    tokens = [start_token]
    hidden = None
    
    for _ in range(max_len):
        input_tensor = torch.tensor([tokens[-1]]).unsqueeze(0)  # (1, 1)

        logits = model(input_tensor)
        logits = logits[:, -1, :] / temperature
        probs = torch.softmax(logits, dim=-1)
        next_token = torch.multinomial(probs, 1).item()
        tokens.append(next_token)
        
        if next_token == vocab['<eos>']:
            break
            
    return tokens

Training this generator requires shifting the target sequence by one position (predicting x[:, 1:] from x[:, :-1]) and using cross-entropy loss, following the same train_epoch pattern defined in the notebook.

Core Implementation Patterns from the Notebook

The following excerpts from lessons/5-NLP/16-RNN/RNNPyTorch.ipynb demonstrate the fundamental building blocks used in both classification and generation tasks.

Simple RNN Classifier

This implementation averages hidden states across the time dimension for classification:

class RNNClassifier(torch.nn.Module):
    def __init__(self, vocab_size, embed_dim, hidden_dim, num_class):
        super().__init__()
        self.embedding = torch.nn.Embedding(vocab_size, embed_dim)
        self.rnn = torch.nn.RNN(embed_dim, hidden_dim, batch_first=True)
        self.fc = torch.nn.Linear(hidden_dim, num_class)

    def forward(self, x):
        # x shape: (batch, seq_len)

        x = self.embedding(x)               # (batch, seq_len, embed_dim)

        x, _ = self.rnn(x)                   # (batch, seq_len, hidden_dim)

        return self.fc(x.mean(dim=1))        # average pooling over time

Source: Lines 63-71 in RNNPyTorch.ipynb

LSTM Classifier for Longer Sequences

Using the final hidden state rather than mean pooling often yields better performance for LSTM-based models:

class LSTMClassifier(torch.nn.Module):
    def __init__(self, vocab_size, embed_dim, hidden_dim, num_class):
        super().__init__()
        self.embedding = torch.nn.Embedding(vocab_size, embed_dim)
        self.rnn = torch.nn.LSTM(embed_dim, hidden_dim, batch_first=True)
        self.fc = torch.nn.Linear(hidden_dim, num_class)

    def forward(self, x):
        x = self.embedding(x)
        x, (h, _) = self.rnn(x)          # h shape: (1, batch, hidden_dim)

        return self.fc(h.squeeze(0))      # use final hidden state

Source: Lines 75-84 in RNNPyTorch.ipynb

Packed Sequence Optimization

For variable-length batches, the notebook implements LSTMPackClassifier using torch.nn.utils.rnn.pack_padded_sequence to skip padding tokens in the computation graph:

class LSTMPackClassifier(torch.nn.Module):
    def __init__(self, vocab_size, embed_dim, hidden_dim, num_class):
        super().__init__()
        self.embedding = torch.nn.Embedding(vocab_size, embed_dim)
        self.rnn = torch.nn.LSTM(embed_dim, hidden_dim,
                                 batch_first=True)
        self.fc = torch.nn.Linear(hidden_dim, num_class)

    def forward(self, x, lengths):
        x = self.embedding(x)
        packed = torch.nn.utils.rnn.pack_padded_sequence(
                    x, lengths, batch_first=True, enforce_sorted=False)
        packed, (h, _) = self.rnn(packed)
        return self.fc(h.squeeze(0))

Source: Lines 30-37 in RNNPyTorch.ipynb

Training Loop Implementation

The train_epoch function handles both standard and packed-sequence models:

def train_epoch(net, loader, lr=0.001):
    optimizer = torch.optim.Adam(net.parameters(), lr=lr)
    criterion = torch.nn.CrossEntropyLoss()
    net.train()
    for step, (labels, batch, *rest) in enumerate(loader):
        optimizer.zero_grad()
        # For packed seq models, `rest` contains lengths

        if rest:
            outputs = net(batch.to(device), rest[0].to('cpu'))
        else:
            outputs = net(batch.to(device))
        loss = criterion(outputs, labels.to(device))
        loss.backward()
        optimizer.step()
        if step % 100 == 0:
            acc = (outputs.argmax(dim=1) == labels.to(device)).float().mean()
            print(f"{step*len(batch)}: acc={acc.item():.4f}")

Source: Training cell L42-L45 in RNNPyTorch.ipynb

Summary

• RNNPyTorch.ipynb provides complete implementations of RNNClassifier, LSTMClassifier, and LSTMPackClassifier using PyTorch's nn.RNN and nn.LSTM modules.
• The notebook's architecture separates embedding layers, recurrent cells, and output heads, making it straightforward to adapt classification models for text generation.
• Packed sequences via torch.nn.utils.rnn.pack_padded_sequence eliminate unnecessary computation on padding tokens, significantly improving training efficiency for variable-length text.
• To convert the provided classifiers into generators, replace the classification nn.Linear layer with a vocabulary-sized output layer and implement autoregressive sampling using torch.multinomial.
• The train_epoch function and data loading utilities in torchnlp.py support both classification and generation training loops with minimal modification.

Frequently Asked Questions

What is the difference between RNN and LSTM in the notebook?

The RNNClassifier uses torch.nn.RNN, which implements a simple recurrent cell that struggles with long-term dependencies due to vanishing gradients. The LSTMClassifier uses torch.nn.LSTM, which adds forget, input, and output gates to control information flow, allowing the model to retain context over longer sequences. According to the notebook's markdown cells, LSTM is preferred for text tasks exceeding 10-20 tokens.

How does packed padding improve RNN performance?

The LSTMPackClassifier implementation uses torch.nn.utils.rnn.pack_padded_sequence to remove padding tokens from the computation graph before feeding data to the LSTM. This prevents the recurrent cell from processing meaningless zero-vectors, reducing both computation time and memory usage during training. The notebook demonstrates this pattern in lines 30-37.

Can I use the same model for both classification and generation?

Yes. The architectural backbone—embedding layer plus LSTM—is identical for both tasks. To switch from classification to generation, replace the final nn.Linear(hidden_dim, num_class) with nn.Linear(hidden_dim, vocab_size) and modify the forward pass to return sequences rather than aggregated vectors. The training objective changes from cross-entropy over classes to next-token prediction.

What dataset does the notebook use for training?

The notebook utilizes a news-category dataset loaded through the torchnlp.py utility module, which provides load_dataset and padify functions. This dataset contains news headlines labeled by category (such as sports, politics, and entertainment), making it suitable for training sequence classification models that can later be repurposed for text generation tasks.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}