How Does the Attention Mechanism Work in Transformers vs LSTM/GRU

Attention mechanisms enable models to focus on specific input segments when generating each output, with Transformers employing parallel self-attention across all positions while LSTM/GRU models use sequential encoder-decoder attention between recurrent states.

The scutan90/DeepLearning-500-questions repository provides comprehensive technical explanations of attention implementations across different architectures. According to the source material, attention solves the fixed-size vector bottleneck in sequence-to-sequence models by dynamically computing weighted sums of encoder representations, fundamentally changing how neural networks handle long sequences.

What Is Attention?

Attention allows a model to learn which parts of an input sequence are most relevant for producing each output element. Instead of compressing an entire source sentence into a single fixed-size context vector—the classic encoder-decoder bottleneck—the mechanism computes a weighted sum of all encoder hidden states. The weights (a_ij in the notation used in ch06_循环神经网络(RNN)/第六章_循环神经网络(RNN).md) are learned alignment scores measuring how well the i-th output position matches the j-th input position.

Attention in LSTM/GRU Seq2Seq Models

In recurrent architectures, attention operates between the encoder and decoder through the following mechanism (described in section 6.2.7 of the RNN chapter):

Encoder processing. An LSTM or GRU processes the source sequence and emits a hidden state h_j at each time step.

Score computation. For each decoder step i, a score e_ij = score(s_{i-1}, h_j) is calculated using dot-product, additive (Bahdanau), or scaled dot-product methods. This score measures alignment between the current decoder state s_{i-1} and each encoder output h_j.

Attention weights. The scores pass through a softmax to produce normalized weights a_ij = softmax(e_ij), forming a probability distribution over source positions.

Context vector. The weighted sum c_i = Σ_j a_ij * h_j creates a dynamic summary vector specific to the current output step. The decoder combines this context with its own previous state to generate the next token.

This approach allows every output token to attend to different source subsets, solving the long-distance bottleneck that plagued vanilla seq2seq models (see lines 125-130 in the RNN chapter source).

Self-Attention in Transformer Models

Transformers discard recurrence entirely and rely on self-attention (multi-head attention) for both encoding and decoding, as referenced in the model compression chapter (ch17_模型压缩、加速及移动端部署/第十七章_模型压缩、加速及移动端部署.md, line 1037).

Input embeddings. Tokens convert to vectors enriched with explicit positional encodings, since the model lacks inherent sequential processing.

Scaled dot-product attention. For every position i, the model derives queries Q_i, keys K_j, and values V_j through linear projections. Attention weights follow a_ij = softmax((Q_i·K_j^T)/√d_k), where the scaling factor √d_k prevents softmax saturation.

Parallel processing. Each output computes as output_i = Σ_j a_ij V_j, allowing every token to instantly attend to every other token simultaneously.

Multi-head mechanism. The operation runs in parallel h times with different learned projections, enabling the model to capture diverse relational aspects (syntactic, semantic, positional) within the same layer.

Because the architecture removes sequential dependencies, Transformers enable fully parallel training and maintain direct long-range connections at a computational cost of O(n²) per layer for the attention matrix.

Key Differences Between RNN Attention and Transformer Self-Attention

Aspect	LSTM/GRU with Attention	Transformer
Recurrence	Sequential processing with hidden states passed step-by-step	No recurrence; entire sequence processed in parallel
Complexity	O(n) sequential steps; attention adds O(n²) per decoder step	O(1) sequential depth per layer; O(n²) for attention matrix
Positional encoding	Implicitly encoded in hidden state progression	Explicit sinusoidal or learned positional embeddings required
Parallelism	Limited—later steps depend on earlier computations	Fully parallelizable across the time dimension
Long-range modeling	Degrades over distance due to gradient paths through time	Handles arbitrary distances via direct pairwise query-key interactions

Implementation Examples

Bahdanau Attention for LSTM (PyTorch)

The following implementation reflects the additive attention mechanism described in the RNN chapter, computing alignment scores between decoder states and encoder outputs:

import torch
import torch.nn as nn
import torch.nn.functional as F

class BahdanauAttention(nn.Module):
    def __init__(self, enc_dim, dec_dim, att_dim):
        super().__init__()
        self.W_enc = nn.Linear(enc_dim, att_dim, bias=False)
        self.W_dec = nn.Linear(dec_dim, att_dim, bias=False)
        self.v = nn.Linear(att_dim, 1, bias=False)

    def forward(self, enc_outputs, dec_hidden):
        # enc_outputs: (seq_len, batch, enc_dim)

        # dec_hidden: (batch, dec_dim)

        seq_len, batch, _ = enc_outputs.size()
        enc_proj = self.W_enc(enc_outputs)                     # (seq_len, batch, att_dim)

        dec_proj = self.W_dec(dec_hidden).unsqueeze(0)          # (1, batch, att_dim)

        scores = self.v(torch.tanh(enc_proj + dec_proj))       # (seq_len, batch, 1)

        attn_weights = F.softmax(scores, dim=0)                # (seq_len, batch, 1)

        context = torch.sum(attn_weights * enc_outputs, dim=0)   # (batch, enc_dim)

        return context, attn_weights.squeeze(-1)               # (batch, enc_dim), (seq_len, batch)

class Seq2Seq(nn.Module):
    def __init__(self, vocab_sz, emb_sz, hid_sz):
        super().__init__()
        self.embed = nn.Embedding(vocab_sz, emb_sz)
        self.encoder = nn.LSTM(emb_sz, hid_sz, bidirectional=False)
        self.attn = BahdanauAttention(hid_sz, hid_sz, att_dim=hid_sz)
        self.decoder = nn.LSTMCell(emb_sz + hid_sz, hid_sz)
        self.out = nn.Linear(hid_sz, vocab_sz)

    def forward(self, src, tgt):
        # src: (src_len, batch)  tgt: (tgt_len, batch)

        src_emb = self.embed(src)
        enc_out, (h, c) = self.encoder(src_emb)               # enc_out: (src_len, batch, hid_sz)

        dec_hidden, dec_cell = h.squeeze(0), c.squeeze(0)
        outputs = []
        for t in range(tgt.size(0)):
            dec_input = self.embed(tgt[t])
            ctx, _ = self.attn(enc_out, dec_hidden)           # context vector per step

            dec_hidden, dec_cell = self.decoder(
                torch.cat([dec_input, ctx], dim=1), (dec_hidden, dec_cell))
            logits = self.out(dec_hidden)
            outputs.append(logits)
        return torch.stack(outputs)                            # (tgt_len, batch, vocab_sz)

The BahdanauAttention module implements the score computation e_ij and produces the context vector c_i that the decoder LSTM consumes at each timestep, exactly matching the mechanism detailed in ch06_循环神经网络(RNN)/第六章_循环神经网络(RNN).md.

Multi-Head Self-Attention in Transformers (PyTorch)

This implementation leverages PyTorch's built-in Transformer modules to achieve the parallel self-attention described in the model compression chapter:

import torch
import torch.nn as nn

class SimpleTransformer(nn.Module):
    def __init__(self, vocab_sz, d_model=512, nhead=8, num_enc=6, num_dec=6):
        super().__init__()
        self.embedding = nn.Embedding(vocab_sz, d_model)
        self.pos_encoder = nn.Sequential(nn.Dropout(0.1))
        self.transformer = nn.Transformer(
            d_model=d_model, nhead=nhead,
            num_encoder_layers=num_enc,
            num_decoder_layers=num_dec,
            dim_feedforward=2048, dropout=0.1)
        self.fc_out = nn.Linear(d_model, vocab_sz)

    def forward(self, src, tgt, src_mask=None, tgt_mask=None,
                src_key_padding_mask=None, tgt_key_padding_mask=None):
        # src, tgt: (seq_len, batch)

        src_emb = self.embedding(src) * (self.embedding.embedding_dim ** 0.5)
        tgt_emb = self.embedding(tgt) * (self.embedding.embedding_dim ** 0.5)
        src_emb = self.pos_encoder(src_emb)
        tgt_emb = self.pos_encoder(tgt_emb)

        output = self.transformer(
            src_emb, tgt_emb,
            src_mask=src_mask, tgt_mask=tgt_mask,
            src_key_padding_mask=src_key_padding_mask,
            tgt_key_padding_mask=tgt_key_padding_mask)

        return self.fc_out(output)            # (tgt_len, batch, vocab_sz)

The nn.Transformer module implements multi-head self-attention for both encoder and decoder stacks, supporting the parallel training approach mentioned in the repository's discussion of WMT14 translation models.

Summary

• Transformers use self-attention to process entire sequences in parallel without recurrence, while LSTM/GRU models rely on sequential attention between encoder and decoder hidden states.
• The context vector in RNN attention computes as c_i = Σ_j a_ij * h_j, creating dynamic, position-specific summaries that bypass the fixed-size bottleneck.
• Multi-head attention in Transformers applies parallel attention operations with different learned projections, capturing diverse relational patterns without sequential processing steps.
• Transformers achieve O(1) sequential depth per layer with O(n²) pairwise attention complexity, whereas RNN attention requires O(n) sequential steps with additional attention overhead at each decoder timestep.

Frequently Asked Questions

How does attention differ between RNNs and Transformers in terms of computation?

In RNNs with attention, computation proceeds sequentially where each decoder timestep calculates attention weights over all encoder states, resulting in O(n) sequential complexity. Transformers compute attention weights for all positions simultaneously using matrix multiplications, enabling full parallelization across the sequence length but incurring O(n²) complexity for the attention matrix calculations.

Why can Transformers handle long-range dependencies better than LSTM/GRU models?

Transformers connect any two positions directly through self-attention with a constant number of operations regardless of distance, while LSTM/GRU models must propagate information through recurrent steps, causing gradient degradation over long sequences. The direct query-key-value interactions in Transformers preserve signals across arbitrary distances without intermediate state bottlenecks.

What replaces positional information in Transformers since they lack recurrence?

Since Transformers process all tokens in parallel without inherent sequence order, they require explicit positional encodings—usually sinusoidal functions or learned embeddings—added to input embeddings. RNNs encode position implicitly through sequential state propagation where later positions naturally accumulate earlier context through hidden state updates.

Can you use attention mechanisms with GRU instead of LSTM?

Yes, the attention mechanism operates identically with GRU cells as with LSTM—the attention scores compute alignment between the decoder state and encoder outputs, producing a context vector that feeds into the GRU decoder at each step. Both architectures benefit from attention to overcome fixed-context limitations, though GRU's gating mechanisms offer computational efficiency compared to LSTM's separate cell and hidden states.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}