How to Implement Sequence-to-Sequence Models with RNNs for Text Generation

Sequence-to-sequence models use an encoder to compress input text into a context vector and a decoder to generate output tokens autoregressively, making them ideal for machine translation and text generation tasks.

The microsoft/AI-For-Beginners curriculum provides foundational building blocks for implementing sequence-to-sequence (seq2seq) architectures using recurrent neural networks. This guide walks through a complete PyTorch implementation based on the RNN and generative network lessons from the repository, showing you how to implement sequence-to-sequence models with RNNs for text generation from scratch.

What Is a Sequence-to-Sequence Model?

A sequence-to-sequence architecture consists of two recurrent networks working in tandem. The encoder reads the input sequence $\langle x_1, \ldots, x_T \rangle$ and compresses it into fixed-size hidden and cell states $(h_T, c_T)$. The decoder then uses these states as its initial hidden state to generate the output sequence one token at a time, feeding each prediction back as the next input until an end-of-sequence token is produced.

This architecture forms the backbone of neural machine translation, text summarization, and conversational AI systems.

Core Architecture Components

The Encoder

The encoder is typically a stack of LSTM or GRU layers that processes the source sequence and returns the final hidden states. In lessons/5-NLP/16-RNN/RNNPyTorch.ipynb, the curriculum demonstrates how LSTM cells handle sequential data by maintaining hidden states across timesteps.

The encoder's final hidden state $h_T$ and cell state $c_T$ capture the semantic meaning of the entire input sequence, serving as the "memory" passed to the decoder.

The Decoder

The decoder is another LSTM or GRU stack initialized with the encoder's final states. At each timestep $t$, it receives the previously generated token $y_{t-1}$ (or a start-of-sequence token for $t=1$) and outputs a probability distribution over the target vocabulary.

As implemented in lessons/5-NLP/17-GenerativeNetworks/GenerativePyTorch.ipynb, the decoder uses softmax outputs to generate text by sampling from the probability distribution.

The Context Vector

The context vector refers to the final hidden states passed from encoder to decoder. This fixed-size representation must capture all relevant information from the variable-length input sequence, creating an information bottleneck that attention mechanisms (covered in advanced lessons) later address.

Implementing Seq2Seq with PyTorch

Vocabulary and Dataset Preparation

First, define special tokens for padding, sequence start, sequence end, and unknown characters. The curriculum uses character-level tokenization for simplicity, as shown in the generative networks notebook.

import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader, Dataset

PAD_TOKEN = "<PAD>"
SOS_TOKEN = "<SOS>"
EOS_TOKEN = "<EOS>"
UNK_TOKEN = "<UNK>"

class Vocab:
    def __init__(self, tokens):
        self.itos = [PAD_TOKEN, SOS_TOKEN, EOS_TOKEN, UNK_TOKEN] + sorted(set(tokens))
        self.stoi = {tok: i for i, tok in enumerate(self.itos)}
    
    def encode(self, seq):
        return [self.stoi.get(tok, self.stoi[UNK_TOKEN]) for tok in seq]
    
    def decode(self, ids):
        return [self.itos[i] for i in ids]

class TextDataset(Dataset):
    def __init__(self, text, seq_len, vocab):
        self.seq_len = seq_len
        self.vocab = vocab
        self.pairs = []
        for i in range(0, len(text) - seq_len):
            inp = text[i : i + seq_len]
            tgt = text[i + 1 : i + seq_len + 1]
            self.pairs.append((vocab.encode(inp), vocab.encode(tgt)))
    
    def __len__(self):
        return len(self.pairs)
    
    def __getitem__(self, idx):
        src, tgt = self.pairs[idx]
        return torch.tensor(src), torch.tensor(tgt)

Building the Encoder

The encoder embeds input tokens and processes them through an LSTM layer, returning the final hidden and cell states:

class Encoder(nn.Module):
    def __init__(self, vocab_size, embed_dim, hidden_dim, num_layers=1):
        super().__init__()
        self.emb = nn.Embedding(vocab_size, embed_dim, padding_idx=0)
        self.rnn = nn.LSTM(embed_dim, hidden_dim, num_layers, batch_first=True)
    
    def forward(self, x):
        emb = self.emb(x)
        outputs, (h, c) = self.rnn(emb)
        return h, c

Building the Decoder

The decoder mirrors the encoder structure but projects the hidden state to vocabulary logits:

class Decoder(nn.Module):
    def __init__(self, vocab_size, embed_dim, hidden_dim, num_layers=1):
        super().__init__()
        self.emb = nn.Embedding(vocab_size, embed_dim, padding_idx=0)
        self.rnn = nn.LSTM(embed_dim, hidden_dim, num_layers, batch_first=True)
        self.fc = nn.Linear(hidden_dim, vocab_size)
    
    def forward(self, x, hidden, cell):
        emb = self.emb(x)
        out, (h, c) = self.rnn(emb, (hidden, cell))
        logits = self.fc(out.squeeze(1))
        return logits, h, c

Combining Encoder and Decoder

The Seq2Seq class orchestrates the forward pass, handling teacher forcing during training:

import random

class Seq2Seq(nn.Module):
    def __init__(self, encoder, decoder, device):
        super().__init__()
        self.encoder = encoder
        self.decoder = decoder
        self.device = device
    
    def forward(self, src, tgt, teacher_forcing_ratio=0.5):
        batch_size, tgt_len = tgt.shape
        vocab_size = self.decoder.fc.out_features
        outputs = torch.zeros(batch_size, tgt_len, vocab_size).to(self.device)
        
        hidden, cell = self.encoder(src)
        input_tok = tgt[:, 0].unsqueeze(1)
        
        for t in range(1, tgt_len):
            logits, hidden, cell = self.decoder(input_tok, hidden, cell)
            outputs[:, t] = logits
            teacher_force = random.random() < teacher_forcing_ratio
            top1 = logits.argmax(1).unsqueeze(1)
            input_tok = tgt[:, t].unsqueeze(1) if teacher_force else top1
        
        return outputs

Training with Teacher Forcing

Teacher forcing accelerates convergence by feeding the true previous token instead of the model's prediction during training. The loss function sums categorical cross-entropy across all decoder timesteps, ignoring padding tokens:

def train(model, dataloader, criterion, optimizer, device, epochs=5):
    model.train()
    for epoch in range(1, epochs + 1):
        epoch_loss = 0
        for src, tgt in dataloader:
            src, tgt = src.to(device), tgt.to(device)
            optimizer.zero_grad()
            output = model(src, tgt)
            loss = criterion(output[:, 1:].reshape(-1, output.size(-1)),
                           tgt[:, 1:].reshape(-1))
            loss.backward()
            optimizer.step()
            epoch_loss += loss.item()
        print(f"Epoch {epoch} – loss: {epoch_loss/len(dataloader):.4f}")

Inference and Text Generation

During inference, the model generates text autoregressively starting with a <SOS> token:

def generate(model, vocab, start_seq, max_len=100):
    model.eval()
    device = next(model.parameters()).device
    src = torch.tensor(vocab.encode(start_seq)).unsqueeze(0).to(device)
    hidden, cell = model.encoder(src)
    input_tok = torch.tensor([vocab.stoi[SOS_TOKEN]]).unsqueeze(0).to(device)
    
    generated = []
    for _ in range(max_len):
        logits, hidden, cell = model.decoder(input_tok, hidden, cell)
        next_id = logits.argmax(1)
        token = vocab.itos[next_id.item()]
        if token == EOS_TOKEN:
            break
        generated.append(token)
        input_tok = next_id.unsqueeze(1)
    
    return "".join(generated)

Complete End-to-End Code Example

Here is the full implementation combining all components for character-level text generation:

if __name__ == "__main__":
    raw = open("data/shakespeare.txt", "r", encoding="utf-8").read()
    chars = list(set(raw))
    vocab = Vocab(chars)
    
    seq_len = 100
    dataset = TextDataset(raw, seq_len, vocab)
    loader = DataLoader(dataset, batch_size=64, shuffle=True)
    
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    enc = Encoder(len(vocab.itos), embed_dim=128, hidden_dim=256).to(device)
    dec = Decoder(len(vocab.itos), embed_dim=128, hidden_dim=256).to(device)
    model = Seq2Seq(enc, dec, device).to(device)
    
    criterion = nn.CrossEntropyLoss(ignore_index=vocab.stoi[PAD_TOKEN])
    optimizer = optim.Adam(model.parameters(), lr=0.001)
    
    train(model, loader, criterion, optimizer, device, epochs=8)
    print(generate(model, vocab, "To be", max_len=200))

Key Files in the AI-For-Beginners Repository

The implementation above draws directly from these curriculum files:

• lessons/5-NLP/16-RNN/RNNPyTorch.ipynb – Introduces LSTM cells, hidden state handling, and recurrent processing fundamentals that power the encoder.
• lessons/5-NLP/17-GenerativeNetworks/GenerativePyTorch.ipynb – Demonstrates character-level text generation and softmax sampling techniques used in the decoder.
• lessons/5-NLP/20-LangModels/GPT-PyTorch.ipynb – Covers beam search strategies that can replace greedy decoding for higher-quality generation.

Summary

• Encoder-decoder architecture separates input processing from output generation, using the encoder's final hidden states to initialize the decoder.
• Teacher forcing during training uses ground-truth tokens instead of model predictions to stabilize and accelerate learning.
• Greedy decoding selects the highest probability token at each step, while beam search (discussed in GPT-PyTorch.ipynb) maintains multiple candidate sequences for better results.
• Character-level modeling provides a simple vocabulary but can be extended to word-level or subword tokenization for complex tasks.
• The microsoft/AI-For-Beginners notebooks provide foundational concepts for LSTM handling and generative text sampling that apply directly to seq2seq implementation.

Frequently Asked Questions

What is teacher forcing in seq2seq models?

Teacher forcing is a training technique where the decoder receives the actual ground-truth token from the previous timestep rather than its own prediction. This reduces error accumulation during early training stages and speeds convergence. In the code example, the teacher_forcing_ratio parameter controls how often this ground-truth feeding occurs versus using the model's own predictions.

How does the context vector work in an encoder-decoder architecture?

The context vector consists of the final hidden state $h_T$ and cell state $c_T$ produced by the encoder LSTM after processing the entire input sequence. These tensors encode the semantic meaning of the source text into a fixed-size representation that serves as the decoder's initial state. The decoder uses this vector as its "memory" when generating the output sequence, though basic seq2seq models struggle with long sequences due to this compression bottleneck.

What is the difference between greedy decoding and beam search?

Greedy decoding selects the single most probable token at each generation step, which is fast but can lead to suboptimal sequences because early mistakes cannot be corrected. Beam search maintains $k$ candidate sequences (beams) at each step and expands all possibilities, keeping only the top $k$ most likely partial sequences. As noted in lessons/5-NLP/20-LangModels/GPT-PyTorch.ipynb, beam search generally produces higher quality text but requires more computation.

Can I use GRU instead of LSTM for sequence-to-sequence models?

Yes, you can substitute GRU (Gated Recurrent Unit) layers for LSTM layers in both the encoder and decoder. GRUs have fewer parameters than LSTMs because they lack a separate cell state, making them faster to train while often achieving similar performance. To use GRU, replace nn.LSTM with nn.GRU in the encoder and decoder classes, and adjust the state handling to manage only the hidden state $h_T$ without the cell state $c_T$.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}