# How to Implement Sequence-to-Sequence Models with RNNs for Text Generation > Learn to implement sequence-to-sequence models with RNNs for text generation. Understand encoder decoder architecture for machine translation and AI content creation. - Repository: [Microsoft/AI-For-Beginners](https://github.com/microsoft/AI-For-Beginners) - Tags: how-to-guide - Published: 2026-08-09 --- **Sequence-to-sequence models use an encoder to compress input text into a context vector and a decoder to generate output tokens autoregressively, making them ideal for machine translation and text generation tasks.** The microsoft/AI-For-Beginners curriculum provides foundational building blocks for implementing sequence-to-sequence (seq2seq) architectures using recurrent neural networks. This guide walks through a complete PyTorch implementation based on the RNN and generative network lessons from the repository, showing you how to implement sequence-to-sequence models with RNNs for text generation from scratch. ## What Is a Sequence-to-Sequence Model? A sequence-to-sequence architecture consists of two recurrent networks working in tandem. The **encoder** reads the input sequence $\langle x_1, \ldots, x_T \rangle$ and compresses it into fixed-size hidden and cell states $(h_T, c_T)$. The **decoder** then uses these states as its initial hidden state to generate the output sequence one token at a time, feeding each prediction back as the next input until an end-of-sequence token is produced. This architecture forms the backbone of neural machine translation, text summarization, and conversational AI systems. ## Core Architecture Components ### The Encoder The encoder is typically a stack of LSTM or GRU layers that processes the source sequence and returns the final hidden states. In `lessons/5-NLP/16-RNN/RNNPyTorch.ipynb`, the curriculum demonstrates how LSTM cells handle sequential data by maintaining hidden states across timesteps. The encoder's final hidden state $h_T$ and cell state $c_T$ capture the semantic meaning of the entire input sequence, serving as the "memory" passed to the decoder. ### The Decoder The decoder is another LSTM or GRU stack initialized with the encoder's final states. At each timestep $t$, it receives the previously generated token $y_{t-1}$ (or a start-of-sequence token for $t=1$) and outputs a probability distribution over the target vocabulary. As implemented in `lessons/5-NLP/17-GenerativeNetworks/GenerativePyTorch.ipynb`, the decoder uses softmax outputs to generate text by sampling from the probability distribution. ### The Context Vector The context vector refers to the final hidden states passed from encoder to decoder. This fixed-size representation must capture all relevant information from the variable-length input sequence, creating an information bottleneck that attention mechanisms (covered in advanced lessons) later address. ## Implementing Seq2Seq with PyTorch ### Vocabulary and Dataset Preparation First, define special tokens for padding, sequence start, sequence end, and unknown characters. The curriculum uses character-level tokenization for simplicity, as shown in the generative networks notebook. ```python import torch import torch.nn as nn import torch.optim as optim from torch.utils.data import DataLoader, Dataset PAD_TOKEN = "" SOS_TOKEN = "" EOS_TOKEN = "" UNK_TOKEN = "" class Vocab: def __init__(self, tokens): self.itos = [PAD_TOKEN, SOS_TOKEN, EOS_TOKEN, UNK_TOKEN] + sorted(set(tokens)) self.stoi = {tok: i for i, tok in enumerate(self.itos)} def encode(self, seq): return [self.stoi.get(tok, self.stoi[UNK_TOKEN]) for tok in seq] def decode(self, ids): return [self.itos[i] for i in ids] class TextDataset(Dataset): def __init__(self, text, seq_len, vocab): self.seq_len = seq_len self.vocab = vocab self.pairs = [] for i in range(0, len(text) - seq_len): inp = text[i : i + seq_len] tgt = text[i + 1 : i + seq_len + 1] self.pairs.append((vocab.encode(inp), vocab.encode(tgt))) def __len__(self): return len(self.pairs) def __getitem__(self, idx): src, tgt = self.pairs[idx] return torch.tensor(src), torch.tensor(tgt) ``` ### Building the Encoder The encoder embeds input tokens and processes them through an LSTM layer, returning the final hidden and cell states: ```python class Encoder(nn.Module): def __init__(self, vocab_size, embed_dim, hidden_dim, num_layers=1): super().__init__() self.emb = nn.Embedding(vocab_size, embed_dim, padding_idx=0) self.rnn = nn.LSTM(embed_dim, hidden_dim, num_layers, batch_first=True) def forward(self, x): emb = self.emb(x) outputs, (h, c) = self.rnn(emb) return h, c ``` ### Building the Decoder The decoder mirrors the encoder structure but projects the hidden state to vocabulary logits: ```python class Decoder(nn.Module): def __init__(self, vocab_size, embed_dim, hidden_dim, num_layers=1): super().__init__() self.emb = nn.Embedding(vocab_size, embed_dim, padding_idx=0) self.rnn = nn.LSTM(embed_dim, hidden_dim, num_layers, batch_first=True) self.fc = nn.Linear(hidden_dim, vocab_size) def forward(self, x, hidden, cell): emb = self.emb(x) out, (h, c) = self.rnn(emb, (hidden, cell)) logits = self.fc(out.squeeze(1)) return logits, h, c ``` ### Combining Encoder and Decoder The `Seq2Seq` class orchestrates the forward pass, handling teacher forcing during training: ```python import random class Seq2Seq(nn.Module): def __init__(self, encoder, decoder, device): super().__init__() self.encoder = encoder self.decoder = decoder self.device = device def forward(self, src, tgt, teacher_forcing_ratio=0.5): batch_size, tgt_len = tgt.shape vocab_size = self.decoder.fc.out_features outputs = torch.zeros(batch_size, tgt_len, vocab_size).to(self.device) hidden, cell = self.encoder(src) input_tok = tgt[:, 0].unsqueeze(1) for t in range(1, tgt_len): logits, hidden, cell = self.decoder(input_tok, hidden, cell) outputs[:, t] = logits teacher_force = random.random() < teacher_forcing_ratio top1 = logits.argmax(1).unsqueeze(1) input_tok = tgt[:, t].unsqueeze(1) if teacher_force else top1 return outputs ``` ### Training with Teacher Forcing **Teacher forcing** accelerates convergence by feeding the true previous token instead of the model's prediction during training. The loss function sums categorical cross-entropy across all decoder timesteps, ignoring padding tokens: ```python def train(model, dataloader, criterion, optimizer, device, epochs=5): model.train() for epoch in range(1, epochs + 1): epoch_loss = 0 for src, tgt in dataloader: src, tgt = src.to(device), tgt.to(device) optimizer.zero_grad() output = model(src, tgt) loss = criterion(output[:, 1:].reshape(-1, output.size(-1)), tgt[:, 1:].reshape(-1)) loss.backward() optimizer.step() epoch_loss += loss.item() print(f"Epoch {epoch} – loss: {epoch_loss/len(dataloader):.4f}") ``` ### Inference and Text Generation During inference, the model generates text autoregressively starting with a `` token: ```python def generate(model, vocab, start_seq, max_len=100): model.eval() device = next(model.parameters()).device src = torch.tensor(vocab.encode(start_seq)).unsqueeze(0).to(device) hidden, cell = model.encoder(src) input_tok = torch.tensor([vocab.stoi[SOS_TOKEN]]).unsqueeze(0).to(device) generated = [] for _ in range(max_len): logits, hidden, cell = model.decoder(input_tok, hidden, cell) next_id = logits.argmax(1) token = vocab.itos[next_id.item()] if token == EOS_TOKEN: break generated.append(token) input_tok = next_id.unsqueeze(1) return "".join(generated) ``` ## Complete End-to-End Code Example Here is the full implementation combining all components for character-level text generation: ```python if __name__ == "__main__": raw = open("data/shakespeare.txt", "r", encoding="utf-8").read() chars = list(set(raw)) vocab = Vocab(chars) seq_len = 100 dataset = TextDataset(raw, seq_len, vocab) loader = DataLoader(dataset, batch_size=64, shuffle=True) device = torch.device("cuda" if torch.cuda.is_available() else "cpu") enc = Encoder(len(vocab.itos), embed_dim=128, hidden_dim=256).to(device) dec = Decoder(len(vocab.itos), embed_dim=128, hidden_dim=256).to(device) model = Seq2Seq(enc, dec, device).to(device) criterion = nn.CrossEntropyLoss(ignore_index=vocab.stoi[PAD_TOKEN]) optimizer = optim.Adam(model.parameters(), lr=0.001) train(model, loader, criterion, optimizer, device, epochs=8) print(generate(model, vocab, "To be", max_len=200)) ``` ## Key Files in the AI-For-Beginners Repository The implementation above draws directly from these curriculum files: - `lessons/5-NLP/16-RNN/RNNPyTorch.ipynb` – Introduces LSTM cells, hidden state handling, and recurrent processing fundamentals that power the encoder. - `lessons/5-NLP/17-GenerativeNetworks/GenerativePyTorch.ipynb` – Demonstrates character-level text generation and softmax sampling techniques used in the decoder. - `lessons/5-NLP/20-LangModels/GPT-PyTorch.ipynb` – Covers beam search strategies that can replace greedy decoding for higher-quality generation. ## Summary - **Encoder-decoder architecture** separates input processing from output generation, using the encoder's final hidden states to initialize the decoder. - **Teacher forcing** during training uses ground-truth tokens instead of model predictions to stabilize and accelerate learning. - **Greedy decoding** selects the highest probability token at each step, while beam search (discussed in `GPT-PyTorch.ipynb`) maintains multiple candidate sequences for better results. - Character-level modeling provides a simple vocabulary but can be extended to word-level or subword tokenization for complex tasks. - The microsoft/AI-For-Beginners notebooks provide foundational concepts for LSTM handling and generative text sampling that apply directly to seq2seq implementation. ## Frequently Asked Questions ### What is teacher forcing in seq2seq models? Teacher forcing is a training technique where the decoder receives the actual ground-truth token from the previous timestep rather than its own prediction. This reduces error accumulation during early training stages and speeds convergence. In the code example, the `teacher_forcing_ratio` parameter controls how often this ground-truth feeding occurs versus using the model's own predictions. ### How does the context vector work in an encoder-decoder architecture? The context vector consists of the final hidden state $h_T$ and cell state $c_T$ produced by the encoder LSTM after processing the entire input sequence. These tensors encode the semantic meaning of the source text into a fixed-size representation that serves as the decoder's initial state. The decoder uses this vector as its "memory" when generating the output sequence, though basic seq2seq models struggle with long sequences due to this compression bottleneck. ### What is the difference between greedy decoding and beam search? Greedy decoding selects the single most probable token at each generation step, which is fast but can lead to suboptimal sequences because early mistakes cannot be corrected. Beam search maintains $k$ candidate sequences (beams) at each step and expands all possibilities, keeping only the top $k$ most likely partial sequences. As noted in `lessons/5-NLP/20-LangModels/GPT-PyTorch.ipynb`, beam search generally produces higher quality text but requires more computation. ### Can I use GRU instead of LSTM for sequence-to-sequence models? Yes, you can substitute GRU (Gated Recurrent Unit) layers for LSTM layers in both the encoder and decoder. GRUs have fewer parameters than LSTMs because they lack a separate cell state, making them faster to train while often achieving similar performance. To use GRU, replace `nn.LSTM` with `nn.GRU` in the encoder and decoder classes, and adjust the state handling to manage only the hidden state $h_T$ without the cell state $c_T$.