# Building Recurrent Neural Networks (RNNs) for Text Generation: A Practical Guide

> Learn to build Recurrent Neural Networks RNNs for text generation. This practical guide explains how RNNs process sequential data to create new text. Microsoft AI For Beginners.

- Repository: [Microsoft/AI-For-Beginners](https://github.com/microsoft/AI-For-Beginners)
- Tags: how-to-guide
- Published: 2026-08-02

---

**Recurrent Neural Networks (RNNs) process sequential data by maintaining a hidden state that updates at each time step, making them ideal for text generation tasks that require modeling dependencies across tokens.**

The microsoft/AI-For-Beginners curriculum introduces RNN architecture in the NLP module as the foundation for modeling sequences. In `lessons/5-NLP/16-RNN/`, the notebooks demonstrate how to implement these networks using both TensorFlow/Keras and PyTorch, starting with classification tasks and extending to text generation workflows.

## How RNNs Model Sequential Data

At each time step, an RNN unit receives two inputs: the current token embedding **Xᵢ** and the previous hidden state **Sᵢ**. The layer computes a new state **Sᵢ₊₁** through a linear transformation and non-linear activation (typically **tanh**):

```

S₍i+1₎ = f(W_H·S_i + W_I·X_i + b)

```

When the task requires an output at every step—such as predicting the next token in a sequence—the model applies an additional output matrix **W_O**:

```

Y_i = f(W_O·S_i + b_O)

```

This recurrent structure allows information to persist across time steps, enabling the network to capture contextual dependencies in text.

## Architectural Variants Explored

The repository explores several RNN variants, each addressing specific limitations of the basic architecture.

### Simple RNN

The vanilla RNN uses a single hidden layer with shared weights across time steps. While easy to implement, it struggles with long-range dependencies due to the vanishing gradient problem. In the TensorFlow notebook (`RNNTF.ipynb`), this is implemented using `keras.layers.SimpleRNN`.

### LSTM

Long Short-Term Memory networks introduce *forget*, *input*, and *output* gates to control information flow. As implemented in `RNNPyTorch.ipynb`, the `nn.LSTM` layer mitigates vanishing gradients by maintaining a separate cell state alongside the hidden state. For theoretical background, the notebooks reference Christopher Olah’s visual guide to LSTMs and a practical PyTorch multilayer LSTM tutorial.

### Bidirectional RNN

These networks process sequences both forward and backward, concatenating the hidden states. This approach provides context from both directions, improving performance on tasks where surrounding text matters.

### Multilayer RNN

By stacking recurrent layers with `return_sequences=True` (TensorFlow) or the `num_layers` parameter (PyTorch), the model learns hierarchical representations. The PyTorch tutorial specifically demonstrates multilayer configurations for complex pattern extraction.

### Packed-Sequence RNN (PyTorch)

For efficient training on variable-length sequences, the PyTorch implementation in [`torchnlp.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/torchnlp.py) uses `nn.utils.rnn.pack_padded_sequence`. This avoids computation on padded tokens, significantly improving training speed on datasets like AG News.

## From Classification to Text Generation

While the microsoft/AI-For-Beginners notebooks focus on classification using the AG-News dataset (`tfds.load('ag_news_subset')` in `RNNTF.ipynb` lines 58-59), the same building blocks adapt naturally to text generation:

1. **Train a language model**: Configure the RNN to predict the next token given previous tokens.
2. **Implement sampling**: After training, feed the model's predictions back as inputs, applying temperature scaling or top-k filtering.
3. **Post-process**: Convert token IDs back to strings, handling special start/end tokens.

## Implementation Examples

### TensorFlow with Masking

The `RNNTF.ipynb` notebook demonstrates how to handle variable-length sequences using masking:

```python
import tensorflow as tf
from tensorflow import keras

vocab_size = 20000
embed_dim  = 64
batch_sz   = 16

vectorizer = keras.layers.experimental.preprocessing.TextVectorization(
    max_tokens=vocab_size, input_shape=(1,)
)

model = keras.Sequential([
    vectorizer,
    keras.layers.Embedding(vocab_size, embed_dim, mask_zero=True),
    keras.layers.SimpleRNN(16),
    keras.layers.Dense(vocab_size, activation='softmax')
])

model.compile(loss='sparse_categorical_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

# ds_train / ds_test are the TF-DS splits from the notebook

model.fit(ds_train.map(tupelize).batch(batch_sz),
          validation_data=ds_test.map(tupelize).batch(batch_sz))

```

### PyTorch with Packed Sequences

The `RNNPyTorch.ipynb` notebook and [`torchnlp.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/torchnlp.py) utility file show efficient handling of variable lengths:

```python
import torch
import torch.nn as nn
import torch.nn.utils.rnn as rnn_utils

class LSTMPack(nn.Module):
    def __init__(self, vocab_sz, emb_dim, hidden_dim, n_classes):
        super().__init__()
        self.emb = nn.Embedding(vocab_sz, emb_dim)
        self.lstm = nn.LSTM(emb_dim, hidden_dim, batch_first=True)
        self.fc   = nn.Linear(hidden_dim, n_classes)

    def forward(self, x, lengths):
        x = self.emb(x)
        packed = rnn_utils.pack_padded_sequence(x, lengths,
                                                batch_first=True,
                                                enforce_sorted=False)
        packed_out, (h, _) = self.lstm(packed)
        return self.fc(h[-1])

# Example usage

net = LSTMPack(vocab_size, 64, 32, len(classes)).to('cuda')

# train_loader_len yields (labels, padded_sequences, lengths)

train(net, train_loader_len, lr=1e-3, use_pack=True)

```

### Text Generation Loop

To convert a classification model into a generative one, add a softmax output layer and implement an autoregressive sampling loop:

```python
def generate_text(model, start_tokens, max_len=100):
    generated = start_tokens
    for _ in range(max_len):
        probs = model(generated)[0, -1]        # logits for next token

        next_id = tf.random.categorical(tf.math.log([probs]), 1)[0,0]
        generated = tf.concat([generated,
                               tf.expand_dims([next_id], 0)], axis=1)
        if next_id == tokenizer.end_token:      # optional stop token

            break
    return tokenizer.decode(generated.numpy())

```

This pattern appears in the TensorFlow examples, where the model predicts probability distributions over the vocabulary at each step.

## Summary

- **RNNs maintain hidden state**: The architecture updates an internal memory at each time step using weights **W_H**, **W_I**, and bias **b**, allowing sequential dependency modeling.
- **Multiple variants exist**: Simple RNNs serve educational purposes, while LSTMs handle long-range dependencies through gating mechanisms, and packed sequences optimize PyTorch training.
- **Generation requires autoregressive sampling**: Convert classification models by feeding predictions back as inputs, optionally controlling diversity with temperature or top-k sampling.
- **Reference implementations available**: The microsoft/AI-For-Beginners repository provides working examples in `RNNTF.ipynb` and `RNNPyTorch.ipynb`, with utilities in [`torchnlp.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/torchnlp.py).

## Frequently Asked Questions

### What is the difference between SimpleRNN and LSTM for text generation?

SimpleRNN updates its hidden state using a single tanh activation, which causes gradients to vanish over long sequences. LSTM introduces cell states and three gates (forget, input, output) that regulate information flow, preserving gradients across hundreds of time steps. For text generation requiring long-term coherence, LSTM is preferred, as demonstrated in the PyTorch notebook (`RNNPyTorch.ipynb`).

### How do I handle variable-length sequences when training RNNs?

In TensorFlow, set `mask_zero=True` in the Embedding layer to skip padded tokens automatically. In PyTorch, use `torch.nn.utils.rnn.pack_padded_sequence` to compress batches before feeding them to the LSTM, which avoids wasted computation on padding—this technique appears in the [`torchnlp.py`](https://github.com/microsoft/AI-For-Beginners/blob/main/torchnlp.py) utilities file.

### Can I use bidirectional RNNs for text generation?

Bidirectional RNNs process sequences in both directions simultaneously, which requires access to the entire input sequence upfront. For text generation—where tokens are produced sequentially and future context is unavailable—standard unidirectional RNNs or causal masking are required. Bidirectional architectures excel in classification tasks like sentiment analysis where full context is available.

### How do temperature scaling and top-k sampling improve generated text?

Temperature scaling divides logits by a constant T before applying softmax; values T < 1 increase confidence in high-probability tokens (focused text), while T > 1 flatten the distribution (random text). Top-k sampling restricts selection to the k most likely tokens, preventing rare vocabulary choices. These techniques, referenced in the microsoft/AI-For-Beginners curriculum, control the diversity-quality tradeoff in the generation loop.