# How to Work with Recurrent Neural Networks (RNNs) for Sequential Data: A Complete Guide

> Learn how to work with recurrent neural networks RNNs for sequential data. Understand hidden states and temporal dependencies in text, time-series, and audio with this complete guide.

- Repository: [Microsoft/AI-For-Beginners](https://github.com/microsoft/AI-For-Beginners)
- Tags: how-to-guide
- Published: 2026-08-03

---

**RNNs process sequential data by maintaining a hidden state that persists across timesteps, enabling them to model temporal dependencies in text, time-series, and audio.**

Recurrent Neural Networks (RNNs) are a foundational neural architecture designed specifically for data where order matters—sequences such as sentences, stock prices, or speech signals. Unlike feed-forward networks that process inputs independently, RNNs retain memory of previous inputs through a hidden state vector that evolves with each new observation. This guide draws from the `microsoft/AI-For-Beginners` curriculum to explain core RNN mechanics, practical implementations in TensorFlow and PyTorch, and how to apply them to real sequence modeling tasks.

## Core RNN Architecture and Mathematics

The fundamental RNN cell implements two transformations at every timestep *t*:

**Hidden State Update**

The network combines the current input with the previous hidden state:

```

h_t = σ(W_xh · x_t + W_hh · h_{t-1} + b_h)

```

Where **σ** is typically `tanh` or `ReLU`, **W_xh** maps input to hidden space, and **W_hh** defines recurrence—the same weight matrix applied at every step.

**Output Generation**

The hidden state projects to task-specific outputs:

```

y_t = φ(W_ho · h_t + b_o)

```

Here **φ** becomes `softmax` for classification or a linear activation for regression.

### Back-Propagation Through Time (BPTT)

Training unrolls the network across *T* timesteps, creating a computational graph where cells share weights. Gradients flow backward through the entire chain, which creates practical challenges addressed by RNN variants.

## LSTM, GRU, and Bidirectional Variants

Standard RNNs suffer from vanishing or exploding gradients on long sequences. The `AI-For-Beginners` curriculum covers three solutions implemented in `lessons/5-NLP/16-RNN/`:

| Variant | Mechanism | Best For |
|--------|-----------|----------|
| **LSTM** | Input, forget, and output gates regulate information flow | Long document modeling, speech synthesis |
| **GRU** | Reset and update gates with fewer parameters than LSTM | Real-time prediction, resource-constrained devices |
| **Bidirectional RNN** | Two parallel RNNs process sequences forward and backward | Named-entity recognition, context-heavy classification |

In PyTorch, switch from `nn.RNN` to `nn.LSTM` or `nn.GRU` by changing one line. TensorFlow offers `tf.keras.layers.LSTM` and `Bidirectional` wrappers with identical interfaces.

## Complete TensorFlow Implementation

The notebook `lessons/5-NLP/16-RNN/RNNTF.ipynb` demonstrates character-level text generation using `tf.keras.layers.SimpleRNN`. Below is the essential pattern extracted from that implementation:

```python
import tensorflow as tf

# Vocabulary construction from raw text

vocab = sorted(set("hello"))
char2idx = {c: i for i, c in enumerate(vocab)}
idx2char = {i: c for i, c in enumerate(vocab)}

# Sequence preparation: predict next character at each position

seq = [char2idx[c] for c in "hello"]
inputs = tf.expand_dims(seq[:-1], 0)   # (batch=1, timesteps=4)

targets = tf.expand_dims(seq[1:], 0)   # shifted by one position

model = tf.keras.Sequential([
    tf.keras.layers.Embedding(len(vocab), 8),
    tf.keras.layers.SimpleRNN(16, return_sequences=True),
    tf.keras.layers.Dense(len(vocab))
])

model.compile(
    optimizer="adam",
    loss=tf.losses.SparseCategoricalCrossentropy(from_logits=True)
)
model.fit(inputs, targets, epochs=200)

# Autoregressive generation

def predict(start: str, steps: int = 5) -> str:
    input_seq = tf.expand_dims([char2idx[s] for s in start], 0)
    for _ in range(steps):
        logits = model(input_seq)
        next_id = tf.random.categorical(logits[:, -1, :], 1)[0, 0].numpy()
        input_seq = tf.concat([input_seq, [[next_id]]], axis=1)
    return ''.join(idx2char[i] for i in input_seq.numpy().flatten())

print(predict("h"))

```

Key implementation details from the source:

- `return_sequences=True` outputs hidden states at all timesteps, required for sequence-to-sequence training
- `SparseCategoricalCrossentropy(from_logits=True)` avoids redundant softmax computation
- Autoregressive generation feeds predictions back as subsequent inputs

## Complete PyTorch Implementation

The parallel notebook `lessons/5-NLP/16-RNN/RNNPyTorch.ipynb` implements word-level sentiment classification with `nn.LSTM`:

```python
import torch
import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader, Dataset

# Sample data: binary sentiment

texts = ["I love this movie".split(), "Terrible plot".split()]
labels = torch.tensor([1.0, 0.0])

# Vocabulary with padding token for batching

vocab = {w: i + 1 for i, w in enumerate({w for t in texts for w in t})}
vocab["<pad>"] = 0

def encode(sequence, maxlen=5):
    ids = [vocab[w] for w in sequence]
    ids += [vocab["<pad>"]] * (maxlen - len(ids))
    return torch.tensor(ids[:maxlen])

class SentimentDataset(Dataset):
    def __init__(self):
        self.x = [encode(t) for t in texts]
        self.y = labels
    
    def __len__(self):
        return len(self.x)
    
    def __getitem__(self, idx):
        return self.x[idx], self.y[idx]

loader = DataLoader(SentimentDataset(), batch_size=2, shuffle=True)

class LSTMClassifier(nn.Module):
    def __init__(self, vocab_size, embed_dim=8, hidden_dim=16):
        super().__init__()
        self.embedding = nn.Embedding(vocab_size, embed_dim, padding_idx=0)
        self.lstm = nn.LSTM(embed_dim, hidden_dim, batch_first=True)
        self.classifier = nn.Linear(hidden_dim, 1)
    
    def forward(self, x):
        embedded = self.embedding(x)
        _, (final_hidden, _) = self.lstm(embedded)
        return torch.sigmoid(self.classifier(final_hidden.squeeze(0)))

model = LSTMClassifier(len(vocab))
criterion = nn.BCELoss()
optimizer = optim.Adam(model.parameters())

for epoch in range(50):
    for xb, yb in loader:
        predictions = model(xb).squeeze()
        loss = criterion(predictions, yb)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

```

Critical PyTorch patterns from the curriculum:

- `padding_idx=0` in `nn.Embedding` ensures pad tokens contribute zero gradients
- `batch_first=True` matches TensorFlow's default dimension ordering `(batch, seq, features)`
- The LSTM returns `(output, (hidden_n, cell_n))`—we extract `hidden_n` for sentence-level classification

## Curriculum Learning Path for RNNs

The `microsoft/AI-For-Beginners` repository structures RNN education as a progression through `lessons/5-NLP/`:

1. **`13-TextRep/TextRepresentationTF.ipynb`** — Tokenization and numerical encoding pipelines
2. **`14-Embeddings/EmbeddingsTF.ipynb`** — Dense vector representations that feed RNN input layers
3. **`15-LanguageModeling/CBoW-TF.ipynb`** — Non-recurrent baseline (bag-of-words) for comparison
4. **`16-RNN/RNNTF.ipynb` and `16-RNN/RNNPyTorch.ipynb`** — Full RNN implementations with training loops

This sequence ensures understand why recurrence matters and when simpler alternatives suffice.

## Summary

- **RNNs maintain hidden state** across sequence steps, enabling temporal modeling that feed-forward networks cannot achieve
- **LSTM and GRU gates** solve gradient instability in long sequences—prefer these over vanilla `SimpleRNN` for production
- **TensorFlow and PyTorch implementations** follow identical architectural patterns: embedding → recurrent layer → dense output
- **Autoregressive generation** requires feeding predictions as subsequent inputs, demonstrated in `RNNTF.ipynb`
- **Padding and masking** handle variable-length sequences; both frameworks provide built-in support via `padding_idx` (PyTorch) or `Masking` layers (TensorFlow)

## Frequently Asked Questions

### What is the difference between RNN and LSTM?

An LSTM is a specific type of RNN with a more complex internal structure. While a basic RNN directly transforms hidden states through simple activation functions, an LSTM uses three gates—input, forget, and output—to selectively remember or discard information. This gating mechanism prevents vanishing gradients on long sequences, making LSTMs the default choice for most sequential tasks in `lessons/5-NLP/16-RNN/RNNPyTorch.ipynb`.

### How do you handle variable-length sequences in RNNs?

Variable-length sequences require **padding** shorter examples to a common maximum length and **masking** pad positions during loss calculation. In PyTorch, use `padding_idx` in `nn.Embedding` and `pack_padded_sequence` for efficiency. TensorFlow's `Masking` layer automatically propagates mask information through subsequent layers. Both approaches appear in the `AI-For-Beginners` notebooks when processing batches of text.

### When should I use a bidirectional RNN?

Use a **bidirectional RNN** when future context improves prediction quality—applications like named-entity recognition or sentiment analysis where entire sentences are available at inference time. The `Bidirectional` wrapper in TensorFlow or `bidirectional=True` in PyTorch runs two RNNs in parallel and concatenates their hidden states. Avoid bidirectionality for real-time generation tasks where future tokens are unavailable.

### Why does the curriculum teach both TensorFlow and PyTorch?

Framework-agnostic understanding ensures concepts transfer across tools. The `microsoft/AI-For-Beginners` project provides parallel implementations—`RNNTF.ipynb` and `RNNPyTorch.ipynb`—to demonstrate that RNN fundamentals (hidden state propagation, BPTT, sequence padding) remain identical despite different APIs. This dual approach prepares learners for diverse codebases in research and industry.