How to Configure FlashAttention for Eagle Inference on Different GPUs

Install a GPU-specific FlashAttention 2 wheel and set use_flash_attn=True when calling load_pretrained_model() to enable optimized attention kernels in NVlabs/Eagle.

Eagle (and its LocateAnything branch) supports FlashAttention 2 for accelerated transformer inference across NVIDIA GPU architectures. This guide covers the specific package versions, configuration flags, and verification steps required to enable FlashAttention on compatible hardware while ensuring graceful fallback on unsupported systems.

Understanding GPU Compatibility for FlashAttention

FlashAttention 2 requires specific CUDA compute capabilities and library versions depending on your GPU architecture. The library refuses to load on unsupported hardware, triggering Eagle's automatic fallback mechanism.

Hopper and Blackwell (H100, GH100)

For NVIDIA Hopper (H100) and Blackwell (GH100) GPUs, install flash-attn>=2.1 with CUDA 12.1 or newer. This version includes the "top-left → bottom-right" mask alignment fixes required for causal decoding, as noted in Embodied/eaglevl/utils/locany/modeling_qwen2.py.

Ampere Architecture (A100, A30)

Use flash-attn==2.0.* with CUDA 11.8 or 12.0. These kernels support compute capability 8.0 and above. The standard wheels contain the required kernels for Ampere GPUs without additional compilation.

Ada and RTX 30 Series (RTX 4090, RTX 3090)

Install flash-attn==2.0.* or 2.1 (both compile for CC 8.9). These work with CUDA 11.8 or 12.1. If you encounter an "Unsupported GPU" error on RTX 4090, force reinstall with pip install --force-reinstall flash-attn==2.0.4+cu121 (adjusting the CUDA suffix to match your installation).

Older GPUs (T4, V100)

Not supported. FlashAttention 2 requires compute capabilities not present in these architectures. Eagle automatically falls back to PyTorch's SDPA (sdpa or eager) implementation when FlashAttention is unavailable.

Enabling FlashAttention in Eagle Model Loading

The load_pretrained_model function in Eagle/eagle/model/builder.py serves as the central configuration point. When use_flash_attn=True is passed, the function injects attn_implementation='flash_attention_2' into the Hugging Face from_pretrained call.

from eagle.eagle.model.builder import load_pretrained_model

tokenizer, model, image_processor, ctx_len = load_pretrained_model(
    model_path="nvidia/LocateAnything-3B",
    model_name="LocateAnything-3B",
    device_map="auto",
    use_flash_attn=True,  # Enables FlashAttention 2

)

print(model.config._attn_implementation)  # Output: flash_attention_2

Configuration Methods

You can configure FlashAttention through multiple interfaces depending on your workflow.

Python API

Pass use_flash_attn=True to load_pretrained_model, or manually set the configuration after loading:

model.config._attn_implementation = "flash_attention_2"

Command-Line Interface

Training and inference scripts expose the --attn_implementation flag. For example, in Eagle2_5/eaglevl/train/eagle_2_5_vl_finetune.py:

python -m Eagle2_5.eaglevl.train.eagle_2_5_vl_finetune \
    --model_name_or_path nvidia/LocateAnything-3B \
    --attn_implementation flash_attention_2 \
    --device_map auto

YAML Configuration Files

Set the implementation in your configuration YAML:

model_name_or_path: nvidia/LocateAnything-3B
attn_implementation: flash_attention_2
device_map: auto

Verifying FlashAttention Activation

Check that the library is available and correctly configured:

from transformers.utils import is_flash_attn_2_available

print(is_flash_attn_2_available())  # Should return True

print(model.config._attn_implementation)  # Should show 'flash_attention_2'

GPU-Specific Configuration Notes

A100 and A30

No additional steps required. The default flash-attn wheels contain kernels optimized for compute capability 8.0.

RTX 4090 (Compute Capability 8.9)

Use the same wheels as Ampere. If compilation fails, specify the CUDA version explicitly: pip install flash-attn==2.0.4+cu121.

Hopper and Blackwell

Version 2.1.x or newer is mandatory for correct causal mask handling during decoding. The implementation in Embodied/eaglevl/utils/locany/modeling_qwen2.py relies on these kernel fixes.

Mixed-GPU Setups

When using device_map="auto" across different GPU generations, FlashAttention loads only on compatible devices. Unsupported GPUs automatically use the SDPA fallback while supported ones run FlashAttention kernels.

Automatic Fallback to SDPA

According to the source code in Embodied/eaglevl/utils/locany/modeling_vit.py (lines 84-86), Eagle checks is_flash_attn_2_available() before assigning the implementation. If the library is missing or the GPU lacks support, the system logs a warning and switches to the standard SDPA path without manual intervention.

Summary

• Install GPU-specific wheels: Match flash-attn version to your architecture (2.0.* for Ampere/Ada, 2.1+ for Hopper/Blackwell).
• Use the loader flag: Set use_flash_attn=True in load_pretrained_model() from Eagle/eagle/model/builder.py.
• Configure via CLI or YAML: Pass --attn_implementation flash_attention_2 or set the YAML field.
• Verify activation: Check model.config._attn_implementation equals 'flash_attention_2'.
• Graceful degradation: Unsupported GPUs automatically fall back to SDPA.

Frequently Asked Questions

What FlashAttention version should I use for RTX 4090?

Use flash-attn==2.0.* or newer. Both versions compile for compute capability 8.9. If you encounter installation errors, force reinstall with the specific CUDA wheel: pip install --force-reinstall flash-attn==2.0.4+cu121 (adjusting the CUDA suffix to match your environment).

Does Eagle automatically fall back if FlashAttention fails to load?

Yes. As implemented in Embodied/eaglevl/utils/locany/modeling_vit.py, the code checks is_flash_attn_2_available() before assignment. If the library is unavailable or the GPU is unsupported, Eagle logs a warning and automatically switches to the SDPA (sdpa) or eager implementation without requiring code changes.

How do I check if FlashAttention is actually running during inference?

Verify by inspecting model.config._attn_implementation, which should return 'flash_attention_2'. Additionally, transformers.utils.is_flash_attn_2_available() returns True when the library is properly installed. The model's attention classes (such as LlamaFlashAttention2 in LLaMA-based architectures) will invoke the native FlashAttention kernels instead of standard PyTorch attention.

Can I use FlashAttention on V100 or T4 GPUs?

No. FlashAttention 2 requires compute capabilities 8.0 or higher (Ampere and newer). V100 and T4 GPUs lack the necessary hardware features. Eagle will automatically fall back to PyTorch's SDPA implementation on these older architectures.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}