# How to Configure FlashAttention for Eagle Inference on Different GPUs

> Learn how to configure FlashAttention for Eagle inference across diverse GPUs. Install the right FlashAttention 2 wheel and enable optimized kernels for faster performance.

- Repository: [NVIDIA Research Projects/Eagle](https://github.com/NVlabs/Eagle)
- Tags: how-to-guide
- Published: 2026-06-28

---

**Install a GPU-specific FlashAttention 2 wheel and set `use_flash_attn=True` when calling `load_pretrained_model()` to enable optimized attention kernels in NVlabs/Eagle.**

Eagle (and its LocateAnything branch) supports **FlashAttention 2** for accelerated transformer inference across NVIDIA GPU architectures. This guide covers the specific package versions, configuration flags, and verification steps required to enable FlashAttention on compatible hardware while ensuring graceful fallback on unsupported systems.

## Understanding GPU Compatibility for FlashAttention

FlashAttention 2 requires specific CUDA compute capabilities and library versions depending on your GPU architecture. The library refuses to load on unsupported hardware, triggering Eagle's automatic fallback mechanism.

### Hopper and Blackwell (H100, GH100)

For NVIDIA Hopper (H100) and Blackwell (GH100) GPUs, install `flash-attn>=2.1` with CUDA 12.1 or newer. This version includes the "top-left → bottom-right" mask alignment fixes required for causal decoding, as noted in [`Embodied/eaglevl/utils/locany/modeling_qwen2.py`](https://github.com/NVlabs/Eagle/blob/main/Embodied/eaglevl/utils/locany/modeling_qwen2.py).

### Ampere Architecture (A100, A30)

Use `flash-attn==2.0.*` with CUDA 11.8 or 12.0. These kernels support compute capability 8.0 and above. The standard wheels contain the required kernels for Ampere GPUs without additional compilation.

### Ada and RTX 30 Series (RTX 4090, RTX 3090)

Install `flash-attn==2.0.*` or `2.1` (both compile for CC 8.9). These work with CUDA 11.8 or 12.1. If you encounter an "Unsupported GPU" error on RTX 4090, force reinstall with `pip install --force-reinstall flash-attn==2.0.4+cu121` (adjusting the CUDA suffix to match your installation).

### Older GPUs (T4, V100)

**Not supported.** FlashAttention 2 requires compute capabilities not present in these architectures. Eagle automatically falls back to PyTorch's SDPA (`sdpa` or `eager`) implementation when FlashAttention is unavailable.

## Enabling FlashAttention in Eagle Model Loading

The `load_pretrained_model` function in [`Eagle/eagle/model/builder.py`](https://github.com/NVlabs/Eagle/blob/main/Eagle/eagle/model/builder.py) serves as the central configuration point. When `use_flash_attn=True` is passed, the function injects `attn_implementation='flash_attention_2'` into the Hugging Face `from_pretrained` call.

```python
from eagle.eagle.model.builder import load_pretrained_model

tokenizer, model, image_processor, ctx_len = load_pretrained_model(
    model_path="nvidia/LocateAnything-3B",
    model_name="LocateAnything-3B",
    device_map="auto",
    use_flash_attn=True,  # Enables FlashAttention 2

)

print(model.config._attn_implementation)  # Output: flash_attention_2

```

## Configuration Methods

You can configure FlashAttention through multiple interfaces depending on your workflow.

### Python API

Pass `use_flash_attn=True` to `load_pretrained_model`, or manually set the configuration after loading:

```python
model.config._attn_implementation = "flash_attention_2"

```

### Command-Line Interface

Training and inference scripts expose the `--attn_implementation` flag. For example, in [`Eagle2_5/eaglevl/train/eagle_2_5_vl_finetune.py`](https://github.com/NVlabs/Eagle/blob/main/Eagle2_5/eaglevl/train/eagle_2_5_vl_finetune.py):

```bash
python -m Eagle2_5.eaglevl.train.eagle_2_5_vl_finetune \
    --model_name_or_path nvidia/LocateAnything-3B \
    --attn_implementation flash_attention_2 \
    --device_map auto

```

### YAML Configuration Files

Set the implementation in your configuration YAML:

```yaml
model_name_or_path: nvidia/LocateAnything-3B
attn_implementation: flash_attention_2
device_map: auto

```

## Verifying FlashAttention Activation

Check that the library is available and correctly configured:

```python
from transformers.utils import is_flash_attn_2_available

print(is_flash_attn_2_available())  # Should return True

print(model.config._attn_implementation)  # Should show 'flash_attention_2'

```

## GPU-Specific Configuration Notes

### A100 and A30

No additional steps required. The default `flash-attn` wheels contain kernels optimized for compute capability 8.0.

### RTX 4090 (Compute Capability 8.9)

Use the same wheels as Ampere. If compilation fails, specify the CUDA version explicitly: `pip install flash-attn==2.0.4+cu121`.

### Hopper and Blackwell

Version 2.1.x or newer is mandatory for correct causal mask handling during decoding. The implementation in [`Embodied/eaglevl/utils/locany/modeling_qwen2.py`](https://github.com/NVlabs/Eagle/blob/main/Embodied/eaglevl/utils/locany/modeling_qwen2.py) relies on these kernel fixes.

### Mixed-GPU Setups

When using `device_map="auto"` across different GPU generations, FlashAttention loads only on compatible devices. Unsupported GPUs automatically use the SDPA fallback while supported ones run FlashAttention kernels.

## Automatic Fallback to SDPA

According to the source code in [`Embodied/eaglevl/utils/locany/modeling_vit.py`](https://github.com/NVlabs/Eagle/blob/main/Embodied/eaglevl/utils/locany/modeling_vit.py) (lines 84-86), Eagle checks `is_flash_attn_2_available()` before assigning the implementation. If the library is missing or the GPU lacks support, the system logs a warning and switches to the standard SDPA path without manual intervention.

## Summary

- **Install GPU-specific wheels**: Match `flash-attn` version to your architecture (2.0.* for Ampere/Ada, 2.1+ for Hopper/Blackwell).
- **Use the loader flag**: Set `use_flash_attn=True` in `load_pretrained_model()` from [`Eagle/eagle/model/builder.py`](https://github.com/NVlabs/Eagle/blob/main/Eagle/eagle/model/builder.py).
- **Configure via CLI or YAML**: Pass `--attn_implementation flash_attention_2` or set the YAML field.
- **Verify activation**: Check `model.config._attn_implementation` equals `'flash_attention_2'`.
- **Graceful degradation**: Unsupported GPUs automatically fall back to SDPA.

## Frequently Asked Questions

### What FlashAttention version should I use for RTX 4090?

Use `flash-attn==2.0.*` or newer. Both versions compile for compute capability 8.9. If you encounter installation errors, force reinstall with the specific CUDA wheel: `pip install --force-reinstall flash-attn==2.0.4+cu121` (adjusting the CUDA suffix to match your environment).

### Does Eagle automatically fall back if FlashAttention fails to load?

Yes. As implemented in [`Embodied/eaglevl/utils/locany/modeling_vit.py`](https://github.com/NVlabs/Eagle/blob/main/Embodied/eaglevl/utils/locany/modeling_vit.py), the code checks `is_flash_attn_2_available()` before assignment. If the library is unavailable or the GPU is unsupported, Eagle logs a warning and automatically switches to the SDPA (`sdpa`) or eager implementation without requiring code changes.

### How do I check if FlashAttention is actually running during inference?

Verify by inspecting `model.config._attn_implementation`, which should return `'flash_attention_2'`. Additionally, `transformers.utils.is_flash_attn_2_available()` returns `True` when the library is properly installed. The model's attention classes (such as `LlamaFlashAttention2` in LLaMA-based architectures) will invoke the native FlashAttention kernels instead of standard PyTorch attention.

### Can I use FlashAttention on V100 or T4 GPUs?

No. FlashAttention 2 requires compute capabilities 8.0 or higher (Ampere and newer). V100 and T4 GPUs lack the necessary hardware features. Eagle will automatically fall back to PyTorch's SDPA implementation on these older architectures.