FlashAttention 4 vs xFormers Configuration for LTX-2 Attention Optimization

LTX-2 implements a plug-in architecture that delegates attention computation to swappable backends, automatically selecting FlashAttention 4 on Hopper/Blackwell GPUs while falling back to xFormers for masked operations or when packages are unavailable.

The Lightricks LTX-2 video generation model provides flexible attention optimization through a configurable backend system. Understanding how to choose between FlashAttention 4 and xFormers allows you to maximize throughput on modern NVIDIA hardware while maintaining compatibility for complex masking requirements. This guide examines the internal machinery driving these selections and provides practical configuration patterns for production deployments.

Understanding the Attention Backend Architecture

At the heart of LTX-2's attention system lies the Attention module defined in ltx_core/model/transformer/attention.py. Rather than hard-coding a specific attention implementation, the forward pass delegates to a callable selected from the AttentionFunction or MaskedAttentionFunction enums, creating a truly pluggable architecture.

Available Backend Options

LTX-2 supports four primary attention backends, each with distinct hardware requirements and capabilities:

• FlashAttention 4 – Requires the flash-attn-4 package and provides maximum throughput on Hopper (SM 9) and Blackwell (SM 10) architectures. Limitation: Unmasked attention only.
• FlashAttention 3 – Requires the flash-attn package (flash_attn_interface). Optimized for Hopper (SM 9) but lacks mask support.
• xFormers – Requires the xformers package (memory_efficient_attention). Compatible with all CUDA-capable GPUs and supports both masked and unmasked operations, making it the universal fallback.
• PyTorch SDPA – Built into PyTorch and requires no additional packages. Supports all masking modes and runs on any GPU or CPU.

The enum definitions reside at lines 324-332 in attention.py, where each backend maps to a specific implementation class.

How Automatic Backend Selection Works

LTX-2 determines the optimal backend through a multi-stage selection process that occurs at import time and runtime.

Import-Time Availability Checks

During module initialization (lines 30-46 in attention.py), LTX-2 attempts to import optional packages including xFormers and FlashAttention. Missing packages are gracefully handled by setting their import references to None, ensuring the module loads even without specialized attention libraries installed.

The Automatic Selection Logic

The automatic_attention() function (cached for performance) delegates to _select_primary_attention() at lines 62-74. This logic follows a hardware-aware priority chain:

1. On Hopper GPUs (SM 9), the system prefers FlashAttention 3, then xFormers, then FlashAttention 4
2. On Blackwell GPUs (SM 10), FlashAttention 4 takes precedence if available
3. On other devices, the system falls back to the full-priority SDPA dispatcher (_sdpa_full_priority)

For masked operations, automatic_masked_attention() (lines 118-124) exclusively prefers xFormers because FlashAttention kernels lack native mask support, as documented in the MaskedAttentionFunction docstring.

Forcing FlashAttention 4 or xFormers Manually

While automatic selection suits most workflows, production deployments often require explicit backend control. LTX-2 provides the DiffusionStage.with_attention() method (defined in ltx_pipelines/utils/blocks.py, lines 69-78) to inject specific attention implementations at the pipeline level.

This method utilizes set_attention_module_op (from ltx_core/loader/attention_ops.py, lines 15-42) to create a ModuleOps object that swaps the attention_function and masked_attention_function of every Attention submodule when the transformer builds.

Practical Configuration Examples

Force FlashAttention 4 for Unmasked Path

To maximize throughput on H100 or B200 GPUs where FlashAttention 4 is available:

from ltx_core.model.transformer.attention import AttentionFunction
from ltx_pipelines.utils.blocks import DiffusionStage

# stage is an existing DiffusionStage instance

stage_fa4 = stage.with_attention(AttentionFunction.FLASH_ATTENTION_4)

Calling AttentionFunction.FLASH_ATTENTION_4.to_callable() instantiates a FlashAttention4 class (lines 74-86 in attention.py), which set_attention_module_op injects into all attention modules during transformer construction.

Use xFormers for Masked Operations

When your pipeline requires attention masking (such as regional prompting or padding masks), explicitly select xFormers:

from ltx_core.model.transformer.attention import AttentionFunction

stage_xformers = stage.with_attention(AttentionFunction.XFORMERS)

Because xFormers implements both masked and unmasked kernels in the XFormersAttention class (lines 104-118), the same callable handles both paths efficiently.

Combine Explicit Unmasked and Masked Choices

For maximum performance with mixed workloads, assign different backends to each path:

from ltx_core.model.transformer.attention import AttentionFunction, MaskedAttentionFunction

# FlashAttention 4 for unmasked, xFormers for masked

stage_custom = stage.with_attention(AttentionFunction.FLASH_ATTENTION_4) \
                    .with_attention(MaskedAttentionFunction.XFORMERS)

The with_attention method accepts either an AttentionFunction or MaskedAttentionFunction, filling the appropriate slot without affecting the other configuration.

Verify Runtime Backend Selection

After building the transformer, inspect the actual assigned backends:

with stage_fa4.model_context() as transformer:
    # Find any Attention submodule

    attn = next(m for m in transformer.modules() 
                if isinstance(m, Attention))
    print(attn.attention_function.label)         # "FlashAttention4"

    print(attn.masked_attention_function.label)  # "xFormers"

Each backend class exposes a .label attribute containing human-readable identifiers matching the enum definitions.

Summary

• FlashAttention 4 delivers maximum speed on Hopper and Blackwell GPUs but supports only unmasked attention, requiring the flash-attn-4 package.
• xFormers provides universal compatibility across all CUDA GPUs and handles both masked and unmasked operations, serving as the reliable fallback.
• Automatic selection in ltx_core/model/transformer/attention.py chooses backends based on GPU architecture (SM 9 vs SM 10) and package availability.
• Manual override via DiffusionStage.with_attention() allows explicit backend assignment for specific performance requirements.
• Mixed configurations enable FlashAttention 4 for unmasked paths while retaining xFormers for masked operations, optimizing throughput without sacrificing functionality.

Frequently Asked Questions

Does FlashAttention 4 support attention masking in LTX-2?

No, FlashAttention 4 in LTX-2 does not support attention masking. According to the source code in attention.py, the FlashAttention kernels (versions 3 and 4) lack mask support, which is why automatic_masked_attention() explicitly prefers xFormers or SDPA for masked operations. If your workflow requires masking, use xFormers via MaskedAttentionFunction.XFORMERS.

How do I check if FlashAttention 4 is actually being used at runtime?

Inspect the .label attribute on the attention function after model construction. Access any Attention submodule and check module.attention_function.label to see strings like "FlashAttention4" or "xFormers". This verification confirms which backend the set_attention_module_op helper injected during the build phase.

Can I use FlashAttention 4 on GPUs older than Hopper (SM 9)?

No, FlashAttention 4 requires Hopper (SM 9) or Blackwell (SM 10) architectures. The automatic selection logic in _select_primary_attention() detects your GPU compute capability and falls back to xFormers or PyTorch SDPA on older hardware. Attempting to force FlashAttention 4 on incompatible hardware will result in runtime errors or automatic fallback behavior.

What is the performance difference between FlashAttention 4 and xFormers?

FlashAttention 4 optimizes specifically for NVIDIA's latest tensor core arrangements on Hopper and Blackwell, offering significantly higher memory efficiency and throughput compared to xFormers on supported hardware. However, xFormers provides broader compatibility and essential masking capabilities that FlashAttention 4 lacks. For unmasked generation on H100 or B200 GPUs, FlashAttention 4 typically outperforms xFormers; for masked training or inference, xFormers becomes the optimal choice despite potentially lower raw throughput.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}