Building Recurrent Neural Networks (RNNs) for Text Generation: A Practical Guide

Recurrent Neural Networks (RNNs) process sequential data by maintaining a hidden state that updates at each time step, making them ideal for text generation tasks that require modeling dependencies across tokens.

The microsoft/AI-For-Beginners curriculum introduces RNN architecture in the NLP module as the foundation for modeling sequences. In lessons/5-NLP/16-RNN/, the notebooks demonstrate how to implement these networks using both TensorFlow/Keras and PyTorch, starting with classification tasks and extending to text generation workflows.

How RNNs Model Sequential Data

At each time step, an RNN unit receives two inputs: the current token embedding Xᵢ and the previous hidden state Sᵢ. The layer computes a new state Sᵢ₊₁ through a linear transformation and non-linear activation (typically tanh):

S₍i+1₎ = f(W_H·S_i + W_I·X_i + b)

When the task requires an output at every step—such as predicting the next token in a sequence—the model applies an additional output matrix W_O:

Y_i = f(W_O·S_i + b_O)

This recurrent structure allows information to persist across time steps, enabling the network to capture contextual dependencies in text.

Architectural Variants Explored

The repository explores several RNN variants, each addressing specific limitations of the basic architecture.

Simple RNN

The vanilla RNN uses a single hidden layer with shared weights across time steps. While easy to implement, it struggles with long-range dependencies due to the vanishing gradient problem. In the TensorFlow notebook (RNNTF.ipynb), this is implemented using keras.layers.SimpleRNN.

LSTM

Long Short-Term Memory networks introduce forget, input, and output gates to control information flow. As implemented in RNNPyTorch.ipynb, the nn.LSTM layer mitigates vanishing gradients by maintaining a separate cell state alongside the hidden state. For theoretical background, the notebooks reference Christopher Olah’s visual guide to LSTMs and a practical PyTorch multilayer LSTM tutorial.

Bidirectional RNN

These networks process sequences both forward and backward, concatenating the hidden states. This approach provides context from both directions, improving performance on tasks where surrounding text matters.

Multilayer RNN

By stacking recurrent layers with return_sequences=True (TensorFlow) or the num_layers parameter (PyTorch), the model learns hierarchical representations. The PyTorch tutorial specifically demonstrates multilayer configurations for complex pattern extraction.

Packed-Sequence RNN (PyTorch)

For efficient training on variable-length sequences, the PyTorch implementation in torchnlp.py uses nn.utils.rnn.pack_padded_sequence. This avoids computation on padded tokens, significantly improving training speed on datasets like AG News.

From Classification to Text Generation

While the microsoft/AI-For-Beginners notebooks focus on classification using the AG-News dataset (tfds.load('ag_news_subset') in RNNTF.ipynb lines 58-59), the same building blocks adapt naturally to text generation:

1. Train a language model: Configure the RNN to predict the next token given previous tokens.
2. Implement sampling: After training, feed the model's predictions back as inputs, applying temperature scaling or top-k filtering.
3. Post-process: Convert token IDs back to strings, handling special start/end tokens.

Implementation Examples

TensorFlow with Masking

The RNNTF.ipynb notebook demonstrates how to handle variable-length sequences using masking:

import tensorflow as tf
from tensorflow import keras

vocab_size = 20000
embed_dim  = 64
batch_sz   = 16

vectorizer = keras.layers.experimental.preprocessing.TextVectorization(
    max_tokens=vocab_size, input_shape=(1,)
)

model = keras.Sequential([
    vectorizer,
    keras.layers.Embedding(vocab_size, embed_dim, mask_zero=True),
    keras.layers.SimpleRNN(16),
    keras.layers.Dense(vocab_size, activation='softmax')
])

model.compile(loss='sparse_categorical_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

# ds_train / ds_test are the TF-DS splits from the notebook

model.fit(ds_train.map(tupelize).batch(batch_sz),
          validation_data=ds_test.map(tupelize).batch(batch_sz))

PyTorch with Packed Sequences

The RNNPyTorch.ipynb notebook and torchnlp.py utility file show efficient handling of variable lengths:

import torch
import torch.nn as nn
import torch.nn.utils.rnn as rnn_utils

class LSTMPack(nn.Module):
    def __init__(self, vocab_sz, emb_dim, hidden_dim, n_classes):
        super().__init__()
        self.emb = nn.Embedding(vocab_sz, emb_dim)
        self.lstm = nn.LSTM(emb_dim, hidden_dim, batch_first=True)
        self.fc   = nn.Linear(hidden_dim, n_classes)

    def forward(self, x, lengths):
        x = self.emb(x)
        packed = rnn_utils.pack_padded_sequence(x, lengths,
                                                batch_first=True,
                                                enforce_sorted=False)
        packed_out, (h, _) = self.lstm(packed)
        return self.fc(h[-1])

# Example usage

net = LSTMPack(vocab_size, 64, 32, len(classes)).to('cuda')

# train_loader_len yields (labels, padded_sequences, lengths)

train(net, train_loader_len, lr=1e-3, use_pack=True)

Text Generation Loop

To convert a classification model into a generative one, add a softmax output layer and implement an autoregressive sampling loop:

def generate_text(model, start_tokens, max_len=100):
    generated = start_tokens
    for _ in range(max_len):
        probs = model(generated)[0, -1]        # logits for next token

        next_id = tf.random.categorical(tf.math.log([probs]), 1)[0,0]
        generated = tf.concat([generated,
                               tf.expand_dims([next_id], 0)], axis=1)
        if next_id == tokenizer.end_token:      # optional stop token

            break
    return tokenizer.decode(generated.numpy())

This pattern appears in the TensorFlow examples, where the model predicts probability distributions over the vocabulary at each step.

Summary

• RNNs maintain hidden state: The architecture updates an internal memory at each time step using weights W_H, W_I, and bias b, allowing sequential dependency modeling.
• Multiple variants exist: Simple RNNs serve educational purposes, while LSTMs handle long-range dependencies through gating mechanisms, and packed sequences optimize PyTorch training.
• Generation requires autoregressive sampling: Convert classification models by feeding predictions back as inputs, optionally controlling diversity with temperature or top-k sampling.
• Reference implementations available: The microsoft/AI-For-Beginners repository provides working examples in RNNTF.ipynb and RNNPyTorch.ipynb, with utilities in torchnlp.py.

Frequently Asked Questions

What is the difference between SimpleRNN and LSTM for text generation?

SimpleRNN updates its hidden state using a single tanh activation, which causes gradients to vanish over long sequences. LSTM introduces cell states and three gates (forget, input, output) that regulate information flow, preserving gradients across hundreds of time steps. For text generation requiring long-term coherence, LSTM is preferred, as demonstrated in the PyTorch notebook (RNNPyTorch.ipynb).

How do I handle variable-length sequences when training RNNs?

In TensorFlow, set mask_zero=True in the Embedding layer to skip padded tokens automatically. In PyTorch, use torch.nn.utils.rnn.pack_padded_sequence to compress batches before feeding them to the LSTM, which avoids wasted computation on padding—this technique appears in the torchnlp.py utilities file.

Can I use bidirectional RNNs for text generation?

Bidirectional RNNs process sequences in both directions simultaneously, which requires access to the entire input sequence upfront. For text generation—where tokens are produced sequentially and future context is unavailable—standard unidirectional RNNs or causal masking are required. Bidirectional architectures excel in classification tasks like sentiment analysis where full context is available.

How do temperature scaling and top-k sampling improve generated text?

Temperature scaling divides logits by a constant T before applying softmax; values T < 1 increase confidence in high-probability tokens (focused text), while T > 1 flatten the distribution (random text). Top-k sampling restricts selection to the k most likely tokens, preventing rare vocabulary choices. These techniques, referenced in the microsoft/AI-For-Beginners curriculum, control the diversity-quality tradeoff in the generation loop.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}