Building Recurrent Neural Networks (RNNs) for Text Generation: A Complete Guide with PyTorch and TensorFlow

Use gated RNNs (LSTM or GRU) with an embedding layer, stacked recurrent layers, and a linear output projection; train with next-token prediction, then generate text via temperature-scaled softmax sampling or beam search.

Recurrent Neural Networks (RNNs) and their gated variants LSTM and GRU remain foundational architectures for character-level and word-level text generation. This guide walks through the complete implementation used in the Microsoft AI-For-Beginners curriculum, from data preparation to stochastic text generation with temperature control.

Understanding the RNN Text Generation Pipeline

The microsoft/AI-For-Beginners repository provides a structured pathway through lessons/5-NLP/16-RNN/ and lessons/5-NLP/17-GenerativeNetworks/. The core insight is simple: by training an RNN to predict the next token given all previous tokens, the network learns a probability distribution over sequences that enables generative sampling.

Why RNNs for Generation?

RNNs maintain a hidden state that evolves as each token is processed. This state acts as a compressed memory of the sequence so far, allowing the model to capture long-range dependencies without a fixed context window—critical for coherent text generation.

Step-by-Step Implementation

1. Data Preparation

The first step in GenerativePyTorch.ipynb and GenerativeTF.ipynb involves:

• Loading a raw text corpus
• Building a character or token vocabulary (character-level is more common for beginners)
• Slicing the text into overlapping sequences of fixed length nchars

Typical sequence lengths range from 100–200 characters, with vocabulary sizes of 50–100 for simple character-level models.

2. Model Architecture

The standard architecture follows Embedding → RNN → Linear:

import torch
import torch.nn as nn
import torch.nn.functional as F

class CharRNN(nn.Module):
    def __init__(self, vocab_size, embed_dim=128, hidden_dim=256):
        super().__init__()
        self.embed = nn.Embedding(vocab_size, embed_dim)
        self.lstm = nn.LSTM(embed_dim, hidden_dim, batch_first=True)
        self.fc = nn.Linear(hidden_dim, vocab_size)

    def forward(self, x, hidden=None):
        x = self.embed(x)                     # (B, T, embed_dim)

        out, hidden = self.lstm(x, hidden)    # out: (B, T, hidden_dim)

        logits = self.fc(out)                 # (B, T, vocab_size)

        return logits, hidden

Key architectural choices from the curriculum:

Component	Typical Value	Purpose
Embedding size	128–256	Dense vector representation of tokens
Hidden size	256–512	Capacity of the recurrent state
Layers	2–3	Depth for hierarchical feature learning
Dropout	0.2–0.5	Regularization between stacked layers

3. Training Loop

The training objective is next-token prediction using cross-entropy loss:

def train_step(model, optimizer, x, y):
    model.train()
    optimizer.zero_grad()
    logits, _ = model(x)                     # predict all timesteps

    loss = F.cross_entropy(
        logits.view(-1, logits.size(-1)),
        y.view(-1)
    )
    loss.backward()
    optimizer.step()
    return loss.item()

Input x and target y are offset by one position: the model learns to predict y[i] given x[:i+1].

4. Text Generation with Temperature

The generation loop in GenerativePyTorch.ipynb demonstrates soft sampling with temperature control:

def generate_text(model, start_seq, stoi, itos,
                  max_len=500, temperature=1.0):
    model.eval()
    idxs = torch.tensor(
        [stoi[c] for c in start_seq],
        dtype=torch.long
    ).unsqueeze(0)
    hidden = None
    generated = list(start_seq)

    for _ in range(max_len):
        logits, hidden = model(idxs, hidden)
        logits = logits[:, -1, :] / temperature    # temperature scaling

        probs = F.softmax(logits, dim=-1)
        
        next_idx = torch.multinomial(probs, num_samples=1)
        next_char = itos[next_idx.item()]
        
        generated.append(next_char)
        idxs = next_idx.unsqueeze(0)               # autoregressive feedback

        
        if next_char == '<eos>':
            break
    return ''.join(generated)

Temperature effects:

• T < 1 (e.g., 0.5): Sharper distribution, more deterministic, repetitive output
• T = 1: True model distribution
• T > 1 (e.g., 1.2–1.5): Flatter distribution, more creative but potentially less coherent

Advanced Generation: Beam Search

For higher-quality generation, GPT-PyTorch.ipynb in lessons/5-NLP/20-LangModels/ demonstrates beam search. Instead of greedy or stochastic sampling, beam search maintains the top-k most promising partial sequences at each step, selecting the highest-scoring complete sequence at the end.

Beam widths of 3–5 typically balance quality against computational cost.

Complete File Reference

Path	Description
lessons/5-NLP/16-RNN/RNNTF.ipynb	RNN fundamentals and theory (TensorFlow)
lessons/5-NLP/16-RNN/RNNPyTorch.ipynb	RNN fundamentals and theory (PyTorch)
lessons/5-NLP/17-GenerativeNetworks/GenerativeTF.ipynb	Full TensorFlow text generation pipeline
lessons/5-NLP/17-GenerativeNetworks/GenerativePyTorch.ipynb	Complete PyTorch implementation with temperature sampling
lessons/5-NLP/20-LangModels/GPT-PyTorch.ipynb	Beam search and advanced generation techniques

Summary

• Building RNNs for text generation requires three components: an embedding layer, gated recurrent layers (LSTM/GRU), and a linear output projection
• Training uses next-token prediction with cross-entropy loss on overlapping sequences
• Generation employs temperature scaling to control the randomness/creativity trade-off
• Beam search from GPT-PyTorch.ipynb provides an alternative for higher-quality deterministic output
• The microsoft/AI-For-Beginners repository provides complete, runnable notebooks in both PyTorch and TensorFlow

Frequently Asked Questions

What is the difference between LSTM and GRU for text generation?

LSTM (Long Short-Term Memory) uses separate input, forget, and output gates with a cell state, providing more flexible memory control. GRU (Gated Recurrent Unit) merges the cell and hidden states with only reset and update gates, making it computationally cheaper with fewer parameters. Both are implemented in RNNPyTorch.ipynb and RNNTF.ipynb; GRU trains faster while LSTM sometimes captures longer dependencies better.

How does temperature affect text generation quality?

Temperature scales logits before softmax: logits / T. Lower temperature (T < 1) concentrates probability mass on the most likely tokens, producing repetitive but coherent text. Higher temperature (T > 1) flattens the distribution, increasing diversity but risking grammatical errors or nonsense. The curriculum recommends 0.5–1.5 as a practical range.

Can I use word-level instead of character-level tokens?

Yes. Replace character vocabulary with a word tokenizer (or subword tokenizer like BPE). The architecture remains identical—only vocab_size increases significantly. Character-level models have smaller vocabularies (~100) but longer sequences; word-level models have larger vocabularies (10K–100K) but shorter sequences. The GenerativePyTorch.ipynb notebook uses characters for simplicity.

When should I use beam search instead of temperature sampling?

Use beam search when you want the most likely sequence under the model (e.g., for completion tasks where correctness matters). Use temperature sampling when you want creative, diverse outputs (e.g., story generation, brainstorming). Beam search is deterministic; temperature sampling is stochastic. Both techniques are covered in the AI-For-Beginners curriculum.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}