What CUDA Version Is Required to Use FlashKDA?

FlashKDA requires CUDA 12.9 or newer and NVIDIA GPUs with compute capability SM 90 or higher (e.g., Ada Lovelace SM 90a, Hopper SM 100a).

FlashKDA is a high-performance CUDA kernel library developed by MoonshotAI for efficient linear attention computation. To compile and run this library, your system must meet strict CUDA version requirements and GPU architecture constraints that are enforced during the build process.

CUDA 12.9 Minimum Version Requirement

According to the README.md in the MoonshotAI/FlashKDA repository, the library explicitly requires "CUDA 12.9 and above". This is not merely a recommendation but a hard dependency based on the kernel implementations.

The build system validates this requirement through setup.py, which imports CUDA_HOME from torch.utils.cpp_extension to locate the CUDA toolkit. This ensures that the CUDA version used to compile FlashKDA matches the version PyTorch was compiled with, preventing binary incompatibility issues.

GPU Compute Capability Requirements

FlashKDA targets modern NVIDIA architectures with compute capability SM 90 or higher. Specifically, the kernels in csrc/smxx/fwd_launch.cu are optimized for:

• Ada Lovelace (SM 90a)
• Hopper (SM 100a)

Older GPUs with compute capability below SM 90 (such as Ampere SM 80 or Turing SM 75) are not supported by the current kernel implementations.

Build-Time Verification

During installation, setup.py performs implicit verification by attempting to compile the C++ and CUDA extensions. The compilation process requires:

1. A valid CUDA_HOME environment variable pointing to a CUDA 12.9+ installation
2. GCC or Clang compilers compatible with the CUDA toolkit version
3. PyTorch installed with CUDA 12.x support

If your CUDA version is incompatible, the build will fail with errors indicating that the CUDA compiler (nvcc) cannot process the kernel code or that architecture flags are unsupported.

Installation Prerequisites

Before installing FlashKDA, verify your environment meets these specifications:

• CUDA Toolkit: Version 12.9, 12.9.1, 12.10, or newer
• PyTorch: Compiled with CUDA 12.x support
• GPU: NVIDIA Ada Lovelace or Hopper architecture (SM 90+)
• Python: 3.8 or newer

Install FlashKDA from source after confirming your CUDA installation:

# Verify CUDA version

nvcc --version

# Install from source

python setup.py install

Usage Examples

Once you have CUDA 12.9+ installed, you can use the low-level kernel API:

import torch
from flash_kda import fwd

# Dummy tensors (must be CUDA)

B, T, H, K, V = 2, 64, 12, 128, 128
q = torch.randn(B, T, H, K, dtype=torch.bfloat16, device="cuda")
k = torch.randn_like(q)
v = torch.randn(B, T, H, V, dtype=torch.bfloat16, device="cuda")
g = torch.randn_like(q)
beta = torch.randn(B, T, H, dtype=torch.bfloat16, device="cuda")
scale = 1.0

out = torch.empty_like(v)

# Call the kernel directly

fwd(q, k, v, g, beta, scale, out, 
    A_log=torch.zeros(H, dtype=torch.float32, device="cuda"),
    dt_bias=torch.zeros(H, K, dtype=torch.float32, device="cuda"),
    lower_bound=-5.0)

Alternatively, use the higher-level API from flash-linear-attention:

from fla.ops.kda import chunk_kda

with torch.inference_mode():
    out, final_state = chunk_kda(
        q=q, k=k, v=v, g=g, beta=beta,
        scale=scale,
        initial_state=None,
        output_final_state=True,
        use_gate_in_kernel=True,
        use_qk_l2norm_in_kernel=True,
        use_beta_sigmoid_in_kernel=True,
        safe_gate=True,
        A_log=torch.zeros(H, device="cuda"),
        dt_bias=torch.zeros(H, K, device="cuda"),
        lower_bound=-5.0,
        transpose_state_layout=True,
        cu_seqlens=None,
    )

Both entry points require tensors to reside on CUDA devices and will raise runtime errors if CUDA 12.9+ features are unavailable.

Summary

• FlashKDA requires CUDA 12.9 or newer as documented in the repository's README.md
• GPU must support SM 90 or higher (Ada Lovelace, Hopper architectures)
• Build enforcement occurs in setup.py via PyTorch's CUDA_HOME detection
• Kernel implementations reside in csrc/smxx/fwd_launch.cu and csrc/flash_kda.cpp
• Installation fails if CUDA version is below 12.9 or GPU architecture is incompatible

Frequently Asked Questions

Can I use FlashKDA with CUDA 11.8 or 12.1?

No. FlashKDA requires CUDA 12.9 or newer. The kernel implementations utilize features and compiler optimizations only available in CUDA 12.9+, and the setup.py build script enforces this version through PyTorch's CUDA extension mechanisms.

Which specific GPU architectures are supported?

FlashKDA supports NVIDIA GPUs with compute capability SM 90 or higher. This includes Ada Lovelace (SM 90a) and Hopper (SM 100a) architectures. Older GPUs like Ampere (SM 80) or Turing (SM 75) cannot run these kernels.

How do I check if my CUDA version is compatible during installation?

Run nvcc --version before installing. If the output shows 12.9 or higher, your CUDA toolkit is compatible. During installation, setup.py will attempt to compile the kernels; if the version is incompatible, the build will fail with architecture or compiler errors before completing.

Does FlashKDA work with AMD GPUs or CPU-only environments?

No. FlashKDA is specifically implemented for NVIDIA CUDA architecture with kernels written in CUDA C++ (csrc/flash_kda.cpp and csrc/smxx/fwd_launch.cu). It requires NVIDIA hardware with SM 90+ and cannot run on AMD GPUs, CPUs, or without a CUDA-capable device.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}