Does Nanotron Support Flash Attention and Flex Attention?

Yes, Nanotron natively supports both Flash Attention (v1/v2) and Flex Attention through a unified registry architecture in src/nanotron/nn/attention.py, enabling runtime kernel swapping via the config._attn_implementation configuration field.

Huggingface's Nanotron framework implements a modular attention system designed for high-performance distributed training. The library provides first-class integration for both Flash Attention (via the optional flash-attn package) and Flex Attention (via PyTorch 2.x native APIs), allowing developers to select optimized kernels without modifying model source code.

How Nanotron Implements Attention Kernels

The Unified Attention Registry

In src/nanotron/nn/attention.py, Nanotron maintains an ATTN_IMPLEMENTATIONS dictionary that maps symbolic implementation names to concrete forward functions. This registry includes "flash_attention_2" pointing to flash_attention_forward and "flex_attention" pointing to flex_attention_forward. During the forward pass, the model dispatcher queries this registry using self._attn_implementation to execute the selected kernel.

Conditional Import Strategy

Nanotron uses defensive import patterns to handle optional dependencies gracefully. The utility function is_flash_attn_greater_or_equal_2_10() validates the installed flash-attn version before importing flash_attn.flash_attn_interface.flash_attn_func. Similarly, is_torch_flex_attn_available() probes for torch.nn.attention.flex_attention availability. If neither accelerator package is present, the system automatically falls back to "torch_attn", ensuring the library functions out-of-the-box without mandatory dependencies.

Enabling Flash Attention in Nanotron

To activate Flash Attention v2, install the required package and configure the model:

from nanotron.config import Config
from nanotron.trainer import Trainer

# Requires: pip install flash-attn>=2.1.0

config = Config(
    _attn_implementation="flash_attention_2",
    # Additional model settings...

)

trainer = Trainer(config)
trainer.train()

When enabled, the flash_attention_forward routine in src/nanotron/nn/attention.py invokes the highly-optimized CUDA kernels from the flash-attn library for every transformer attention block. For older flash-attn versions, use "flash_attention" instead of "flash_attention_2".

Configuring Flex Attention with Custom Masks

Flex Attention, introduced in PyTorch 2.0, supports dynamic mask patterns including causal, sliding window, and document masking. Configure it through the configuration schema defined in src/nanotron/config/models_config.py:

from nanotron.config import Config
from nanotron.models.qwen import QwenModel

config = Config(
    _attn_implementation="flex_attention",
    flex_attention_mask="sliding_window",  # Options: "causal", "sliding_window", "document"

    sliding_window_size=128,
)

model = QwenModel(config)
output = model(input_ids)

The flex_attention_forward function in src/nanotron/nn/flex_attention.py handles low-level dispatch to torch.nn.attention.flex_attention, applying the specified block-mask type during inference.

Runtime Kernel Switching

You can dynamically swap attention implementations without rebuilding the model graph:

def set_attention_impl(model, impl_name):
    assert impl_name in {"flash_attention_2", "flex_attention", "torch_attn"}
    model._attn_implementation = impl_name

# Example: Prefer Flex Attention on PyTorch 2.0+, fallback to standard

import torch
if hasattr(torch.nn, "attention") and torch.__version__ >= "2.0":
    set_attention_impl(model, "flex_attention")
else:
    set_attention_impl(model, "torch_attn")

This pattern enables hardware-specific optimizations during inference, allowing you to select Flash Attention on NVIDIA GPUs with flash-attn installed, or Flex Attention on newer PyTorch builds.

Key Implementation Files

• src/nanotron/nn/attention.py: Contains the ATTN_IMPLEMENTATIONS registry and flash_attention_forward wrapper for Flash Attention v1/v2 integration.
• src/nanotron/nn/flex_attention.py: Implements flex_attention_forward and mask handling logic for PyTorch's native flex attention API.
• src/nanotron/models/starcoder2.py: Reference implementation demonstrating Flash Attention wiring with explicit flash-attn>=2.5.0 dependency checks.
• src/nanotron/models/qwen.py: Model architecture supporting both Flash and Flex Attention, exposing the flex_attention_mask configuration parameter.
• src/nanotron/config/models_config.py: Configuration schema defining _attn_implementation validation and default values.

Summary

• Nanotron supports Flash Attention via the optional flash-attn>=2.1.0 package and Flex Attention via PyTorch 2.x native APIs.
• Kernel selection is controlled by config._attn_implementation, accepting "flash_attention_2", "flex_attention", or "torch_attn".
• The ATTN_IMPLEMENTATIONS registry in src/nanotron/nn/attention.py provides the dispatch mechanism for all attention variants.
• Flash Attention availability is validated through is_flash_attn_greater_or_equal_2_10() before importing optimized kernels.
• Flex Attention supports configurable mask types (causal, sliding window, document) via config.flex_attention_mask.
• The architecture gracefully degrades to standard PyTorch attention when optional dependencies are absent.

Frequently Asked Questions

How do I enable Flash Attention v2 in Nanotron?

Install the flash-attn package (version 2.1.0 or higher) and set config._attn_implementation = "flash_attention_2" in your Nanotron configuration. The flash_attention_forward function in src/nanotron/nn/attention.py automatically dispatches to the optimized CUDA kernels when available.

What PyTorch version is required for Flex Attention?

Flex Attention requires PyTorch 2.0 or later with the torch.nn.attention.flex_attention module present. Nanotron checks compatibility via is_torch_flex_attn_available() before exposing the flex attention code path.

Can I switch attention implementations after model initialization?

Yes. You can modify model._attn_implementation at runtime to switch between "flash_attention_2", "flex_attention", and "torch_attn" without reinitializing model weights. Ensure the target kernel's dependencies are installed on the current machine before switching.

What happens if flash-attn is not installed?

If the flash-attn package is missing or below version 2.1.0, Nanotron defaults to the standard PyTorch attention implementation ("torch_attn"). The conditional import logic in src/nanotron/nn/attention.py ensures training or inference continues without errors, albeit without Flash Attention optimizations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}