# Does Nanotron Support Flash Attention and Flex Attention?

> Yes Nanotron supports Flash Attention v1 v2 and Flex Attention. Learn how the unified registry architecture and config._attn_implementation enable runtime kernel swapping for faster inference.

- Repository: [Hugging Face/nanotron](https://github.com/huggingface/nanotron)
- Tags: how-to-guide
- Published: 2026-03-03

---

**Yes, Nanotron natively supports both Flash Attention (v1/v2) and Flex Attention through a unified registry architecture in [`src/nanotron/nn/attention.py`](https://github.com/huggingface/nanotron/blob/main/src/nanotron/nn/attention.py), enabling runtime kernel swapping via the `config._attn_implementation` configuration field.**

Huggingface's **Nanotron** framework implements a modular attention system designed for high-performance distributed training. The library provides first-class integration for both **Flash Attention** (via the optional `flash-attn` package) and **Flex Attention** (via PyTorch 2.x native APIs), allowing developers to select optimized kernels without modifying model source code.

## How Nanotron Implements Attention Kernels

### The Unified Attention Registry

In [`src/nanotron/nn/attention.py`](https://github.com/huggingface/nanotron/blob/main/src/nanotron/nn/attention.py), Nanotron maintains an `ATTN_IMPLEMENTATIONS` dictionary that maps symbolic implementation names to concrete forward functions. This registry includes `"flash_attention_2"` pointing to `flash_attention_forward` and `"flex_attention"` pointing to `flex_attention_forward`. During the forward pass, the model dispatcher queries this registry using `self._attn_implementation` to execute the selected kernel.

### Conditional Import Strategy

Nanotron uses defensive import patterns to handle optional dependencies gracefully. The utility function `is_flash_attn_greater_or_equal_2_10()` validates the installed `flash-attn` version before importing `flash_attn.flash_attn_interface.flash_attn_func`. Similarly, `is_torch_flex_attn_available()` probes for `torch.nn.attention.flex_attention` availability. If neither accelerator package is present, the system automatically falls back to `"torch_attn"`, ensuring the library functions out-of-the-box without mandatory dependencies.

## Enabling Flash Attention in Nanotron

To activate Flash Attention v2, install the required package and configure the model:

```python
from nanotron.config import Config
from nanotron.trainer import Trainer

# Requires: pip install flash-attn>=2.1.0

config = Config(
    _attn_implementation="flash_attention_2",
    # Additional model settings...

)

trainer = Trainer(config)
trainer.train()

```

When enabled, the `flash_attention_forward` routine in [`src/nanotron/nn/attention.py`](https://github.com/huggingface/nanotron/blob/main/src/nanotron/nn/attention.py) invokes the highly-optimized CUDA kernels from the flash-attn library for every transformer attention block. For older flash-attn versions, use `"flash_attention"` instead of `"flash_attention_2"`.

## Configuring Flex Attention with Custom Masks

Flex Attention, introduced in PyTorch 2.0, supports dynamic mask patterns including causal, sliding window, and document masking. Configure it through the configuration schema defined in [`src/nanotron/config/models_config.py`](https://github.com/huggingface/nanotron/blob/main/src/nanotron/config/models_config.py):

```python
from nanotron.config import Config
from nanotron.models.qwen import QwenModel

config = Config(
    _attn_implementation="flex_attention",
    flex_attention_mask="sliding_window",  # Options: "causal", "sliding_window", "document"

    sliding_window_size=128,
)

model = QwenModel(config)
output = model(input_ids)

```

The `flex_attention_forward` function in [`src/nanotron/nn/flex_attention.py`](https://github.com/huggingface/nanotron/blob/main/src/nanotron/nn/flex_attention.py) handles low-level dispatch to `torch.nn.attention.flex_attention`, applying the specified block-mask type during inference.

## Runtime Kernel Switching

You can dynamically swap attention implementations without rebuilding the model graph:

```python
def set_attention_impl(model, impl_name):
    assert impl_name in {"flash_attention_2", "flex_attention", "torch_attn"}
    model._attn_implementation = impl_name

# Example: Prefer Flex Attention on PyTorch 2.0+, fallback to standard

import torch
if hasattr(torch.nn, "attention") and torch.__version__ >= "2.0":
    set_attention_impl(model, "flex_attention")
else:
    set_attention_impl(model, "torch_attn")

```

This pattern enables hardware-specific optimizations during inference, allowing you to select Flash Attention on NVIDIA GPUs with flash-attn installed, or Flex Attention on newer PyTorch builds.

## Key Implementation Files

- [`src/nanotron/nn/attention.py`](https://github.com/huggingface/nanotron/blob/main/src/nanotron/nn/attention.py): Contains the `ATTN_IMPLEMENTATIONS` registry and `flash_attention_forward` wrapper for Flash Attention v1/v2 integration.
- [`src/nanotron/nn/flex_attention.py`](https://github.com/huggingface/nanotron/blob/main/src/nanotron/nn/flex_attention.py): Implements `flex_attention_forward` and mask handling logic for PyTorch's native flex attention API.
- [`src/nanotron/models/starcoder2.py`](https://github.com/huggingface/nanotron/blob/main/src/nanotron/models/starcoder2.py): Reference implementation demonstrating Flash Attention wiring with explicit `flash-attn>=2.5.0` dependency checks.
- [`src/nanotron/models/qwen.py`](https://github.com/huggingface/nanotron/blob/main/src/nanotron/models/qwen.py): Model architecture supporting both Flash and Flex Attention, exposing the `flex_attention_mask` configuration parameter.
- [`src/nanotron/config/models_config.py`](https://github.com/huggingface/nanotron/blob/main/src/nanotron/config/models_config.py): Configuration schema defining `_attn_implementation` validation and default values.

## Summary

- Nanotron supports **Flash Attention** via the optional `flash-attn>=2.1.0` package and **Flex Attention** via PyTorch 2.x native APIs.
- Kernel selection is controlled by `config._attn_implementation`, accepting `"flash_attention_2"`, `"flex_attention"`, or `"torch_attn"`.
- The `ATTN_IMPLEMENTATIONS` registry in [`src/nanotron/nn/attention.py`](https://github.com/huggingface/nanotron/blob/main/src/nanotron/nn/attention.py) provides the dispatch mechanism for all attention variants.
- Flash Attention availability is validated through `is_flash_attn_greater_or_equal_2_10()` before importing optimized kernels.
- Flex Attention supports configurable mask types (causal, sliding window, document) via `config.flex_attention_mask`.
- The architecture gracefully degrades to standard PyTorch attention when optional dependencies are absent.

## Frequently Asked Questions

### How do I enable Flash Attention v2 in Nanotron?

Install the `flash-attn` package (version 2.1.0 or higher) and set `config._attn_implementation = "flash_attention_2"` in your Nanotron configuration. The `flash_attention_forward` function in [`src/nanotron/nn/attention.py`](https://github.com/huggingface/nanotron/blob/main/src/nanotron/nn/attention.py) automatically dispatches to the optimized CUDA kernels when available.

### What PyTorch version is required for Flex Attention?

Flex Attention requires PyTorch 2.0 or later with the `torch.nn.attention.flex_attention` module present. Nanotron checks compatibility via `is_torch_flex_attn_available()` before exposing the flex attention code path.

### Can I switch attention implementations after model initialization?

Yes. You can modify `model._attn_implementation` at runtime to switch between `"flash_attention_2"`, `"flex_attention"`, and `"torch_attn"` without reinitializing model weights. Ensure the target kernel's dependencies are installed on the current machine before switching.

### What happens if flash-attn is not installed?

If the `flash-attn` package is missing or below version 2.1.0, Nanotron defaults to the standard PyTorch attention implementation (`"torch_attn"`). The conditional import logic in [`src/nanotron/nn/attention.py`](https://github.com/huggingface/nanotron/blob/main/src/nanotron/nn/attention.py) ensures training or inference continues without errors, albeit without Flash Attention optimizations.