# How FNet Is Different From Standard Transformer Attention: A Deep Dive into Fourier-Based Token Mixing

> Discover how FNet revolutionizes Transformers by replacing attention with Fourier transforms. Experience linear complexity, reduced computation, and faster training. Learn the key differences today.

- Repository: [labml.ai/annotated_deep_learning_paper_implementations](https://github.com/labmlai/annotated_deep_learning_paper_implementations)
- Tags: deep-dive
- Published: 2026-03-04

---

**FNet replaces the self-attention mechanism in standard Transformers with discrete Fourier transforms, eliminating learned projection matrices and masking support while reducing computational complexity from quadratic to linear.**

The labmlai/annotated_deep_learning_paper_implementations repository provides a complete PyTorch implementation demonstrating exactly how FNet diverges from traditional transformer attention. Understanding these architectural differences is essential for optimizing transformer efficiency in production environments.

## Core Architectural Differences

### Self-Attention vs Fourier Mixing

Standard transformers rely on the query-key-value paradigm implemented in [`labml_nn/transformers/mha.py`](https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/main/labml_nn/transformers/mha.py), where the `MultiHeadAttention` class computes similarity scores through matrix multiplication of learned projections. In contrast, FNet's `FNetMix` class in [`labml_nn/transformers/fnet/__init__.py`](https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/main/labml_nn/transformers/fnet/__init__.py) applies two-dimensional discrete Fourier transforms—first across the **embedding dimension**, then across the **sequence dimension**—to mix token representations globally without learned attention weights.

### Query-Key-Value Equivalence

One fundamental distinction lies in the tensor relationships. Standard attention maintains independent linear projections for queries, keys, and values through separate weight matrices. FNet enforces that **query = key = value = x**, with the implementation explicitly asserting this equivalence via `assert query is key and key is value`, effectively removing all learned parameters from the mixing operation itself.

### Masking Limitations

Standard self-attention supports arbitrary masking through the `mask` parameter, allowing causal autoregressive training and padding token suppression. FNet fundamentally **does not support masking** because the Fourier transform is a global mathematical operation where every token inherently influences every other token; the code enforces this with `assert mask is None`.

## Computational Complexity and Performance Trade-offs

### Complexity Analysis

Standard self-attention scales quadratically with sequence length due to the `Q·Kᵀ` matrix multiplication, resulting in `O(N²·d)` complexity where N is sequence length and d is embedding dimension. FNet leverages the Fast Fourier Transform (FFT) to achieve **linear complexity** of `O(N·log N·d)`, delivering approximately a **7× speed-up** on typical hardware configurations.

### Accuracy Considerations

While standard transformers achieve state-of-the-art performance across NLP benchmarks, FNet achieves roughly **92% of BERT's GLUE score**—representing a modest accuracy trade-off for significant computational gains.

## Implementation Details from the Annotated Code

The `FNetMix.forward()` method in [`labml_nn/transformers/fnet/__init__.py`](https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/main/labml_nn/transformers/fnet/__init__.py) implements the mixing operation through PyTorch's FFT functions:

```python
def forward(self, query, key, value, mask=None):
    assert query is key and key is value   # same tensor x

    assert mask is None                    # no masking supported

    x = query
    fft_hidden = torch.fft.fft(x, dim=2)   # Fourier across embedding dim

    fft_seq = torch.fft.fft(fft_hidden, dim=0)  # Fourier across sequence dim

    return torch.real(fft_seq)             # keep real part

```

Compare this to the standard attention mechanism in [`labml_nn/transformers/mha.py`](https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/main/labml_nn/transformers/mha.py):

```python
def forward(self, query, key, value, mask=None):
    q = self.query(query)      # learned linear projection

    k = self.key(key)          # learned linear projection  

    v = self.value(value)      # learned linear projection

    scores = torch.einsum('ibhd,jbhd->ijbh', q, k) * self.scale
    if mask is not None:
        scores = scores.masked_fill(mask == 0, float('-inf'))
    attn = self.softmax(scores)
    attn = self.dropout(attn)
    return torch.einsum('ijbh,jbhd->ibhd', attn, v)

```

## Practical Code Examples

Swapping standard attention for FNet requires minimal architectural changes because `FNetMix` deliberately mirrors the `MultiHeadAttention` interface:

```python
from labml_nn.transformers.mha import MultiHeadAttention
from labml_nn.transformers.fnet import FNetMix

class FNetTransformerLayer(nn.Module):
    def __init__(self, d_model, heads):
        super().__init__()
        # No heads or d_model parameters needed for FNet

        self.mixing = FNetMix()

    def forward(self, x, mask=None):
        # Mask must be None for FNet

        return self.mixing(query=x, key=x, value=x, mask=None)

```

Running a comparison between both approaches:

```python
import torch
from labml_nn.transformers.mha import MultiHeadAttention
from labml_nn.transformers.fnet import FNetMix

x = torch.randn(10, 4, 64)   # [seq_len, batch, d_model]

# Standard attention with masking

attn = MultiHeadAttention(heads=8, d_model=64)
mask = torch.ones(10, 10, 4)
out_attn = attn(query=x, key=x, value=x, mask=mask)

# FNet without masking

fnet = FNetMix()
out_fnet = fnet(query=x, key=x, value=x, mask=None)

# Both output: (10, 4, 64)

```

## Summary

- **FNet eliminates learned attention weights** by replacing the query-key-value mechanism with discrete Fourier transforms across embedding and sequence dimensions.
- **No masking support** exists in FNet because Fourier transforms are global operations, unlike the selective masking available in standard self-attention.
- **Linear complexity** `O(N·log N·d)` provides significant speed improvements over the quadratic `O(N²·d)` cost of standard attention.
- **92% of BERT accuracy** represents the trade-off for 7× faster computation on typical sequence lengths.

## Frequently Asked Questions

### Can FNet replace attention in existing transformer models?

Yes. According to the labmlai implementation, `FNetMix` maintains the same interface as `MultiHeadAttention`, accepting `query`, `key`, `value`, and `mask` parameters. This design allows drop-in replacement of attention layers in existing architectures, though you must ensure `mask=None` and that query, key, and value are identical tensors.

### Why doesn't FNet support causal masking?

FNet applies discrete Fourier transforms across both the embedding and sequence dimensions, which are global mathematical operations where every position influences every other position. Unlike self-attention where you can zero-out specific positions in the attention matrix, the Fourier transform cannot selectively ignore future tokens, making causal autoregressive training impossible.

### What are the specific file paths for FNet implementation in the repository?

The core implementation resides in [`labml_nn/transformers/fnet/__init__.py`](https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/main/labml_nn/transformers/fnet/__init__.py), which contains the `FNetMix` class. The standard attention comparison is in [`labml_nn/transformers/mha.py`](https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/main/labml_nn/transformers/mha.py). End-to-end training experiments using FNet on the AG News dataset are located in [`labml_nn/transformers/fnet/experiment.py`](https://github.com/labmlai/annotated_deep_learning_paper_implementations/blob/main/labml_nn/transformers/fnet/experiment.py).

### How much faster is FNet compared to standard transformers?

FNet achieves roughly a **7× speed-up** over standard self-attention implementations on typical hardware, scaling linearly with sequence length `O(N·log N·d)` rather than quadratically `O(N²·d)`. This efficiency gain comes at the cost of approximately 8% accuracy reduction compared to BERT on GLUE benchmarks.