How Needle 2 Handles Attention Computation on Different Hardware Backends

Needle 2 automatically selects between CUDNN flash attention for compatible GPUs with half-precision data types and a portable JAX implementation for CPUs and TPUs, optimizing performance based on the detected backend and tensor precision.

The cactus-compute/needle repository implements a hardware-aware attention mechanism that dynamically selects the most efficient computation path based on the available JAX backend. In needle/model/architecture.py, the MultiHeadAttention module detects whether it is running on GPU, CPU, or TPU and chooses between optimized flash kernels and fallback implementations accordingly. This design ensures optimal throughput on NVIDIA hardware while maintaining full compatibility with other accelerators and higher-precision training regimes.

Backend Detection and Dispatch Strategy

The MultiHeadAttention class relies on jax.default_backend() to detect the execution environment at runtime. This function returns the strings "gpu", "cpu", or "tpu", which MultiHeadAttention uses to branch between optimized and portable code paths.

GPU Flash Attention with CUDNN

When jax.default_backend() returns "gpu" and the query tensor uses bfloat16 or float16 precision, Needle 2 enables the CUDNN implementation of scaled dot-product attention. The code constructs an impl flag set to "cudnn" and passes it to jax.nn.dot_product_attention, which fuses the matrix multiplication, softmax, and masking operations into a single kernel to minimize memory bandwidth and HBM traffic.

impl = ("cudnn" if jax.default_backend() == "gpu"
        and q.dtype in (jnp.bfloat16, jnp.float16) else None)
out = jax.nn.dot_product_attention(
    q.transpose(0, 2, 1, 3),
    k.transpose(0, 2, 1, 3),
    v.transpose(0, 2, 1, 3),
    mask=mask,
    implementation=impl,
)

Portable Fallback for CPU and TPU

For CPUs, TPUs, or when the query tensor is float32, Needle 2 falls back to an explicit JAX implementation that materializes the full attention matrix. This path computes scaled dot-product attention using standard jnp.matmul and nn.softmax operations, providing broad hardware compatibility at the cost of increased memory usage.

scale = jnp.sqrt(jnp.float32(head_dim))
attn_weights = jnp.matmul(q, k.transpose(0, 1, 3, 2)) / scale
attn_weights = nn.softmax(attn_weights, axis=-1)
out = jnp.matmul(attn_weights, v)

Quantization and Multi-Query Support

Before either computation path, the module applies _quantize.maybe_quant_kv to the key and value tensors when quantization is enabled, reducing memory footprint for inference. After attention computation, the output is re-quantized via _aq when quant=True. The fallback path also supports Multi-Query Attention (MQA) by repeating key and value tensors when num_kv_heads differs from num_heads, allowing memory-efficient inference with shared key-value representations.

Practical Configuration Examples

Enabling Flash Attention on GPU

When running on CUDA-enabled hardware with bfloat16 or float16 tensors, Needle 2 automatically selects the CUDNN kernel:

import jax, jax.numpy as jnp
from needle.model.architecture import MultiHeadAttention

# Assume we have a GPU and the default dtype is bfloat16

x = jnp.ones((1, 128, 768), dtype=jnp.bfloat16)   # (batch, seq_len, d_model)

attn = MultiHeadAttention(
    num_heads=12,
    num_kv_heads=12,
    d_model=768,
    num_layers=24,
    dtype=jnp.bfloat16,
    flash=True,
)
out = attn(x)   # internally selects the cudnn flash kernel

print(out.shape)   # (1, 128, 768)

Forcing CPU Compatibility Mode

To use the fallback implementation on CPU or with higher precision, specify float32 for the dtype:

import jax, jax.numpy as jnp
from needle.model.architecture import MultiHeadAttention

x = jnp.ones((1, 128, 768), dtype=jnp.float32)   # float32 triggers fallback

attn = MultiHeadAttention(
    num_heads=12,
    num_kv_heads=12,
    d_model=768,
    num_layers=24,
    dtype=jnp.float32,
    flash=False,          # optional: can also keep flash=True – fallback will be chosen automatically

)
out = attn(x)
print(out.shape)

Configuring Multi-Query Attention

To reduce memory bandwidth with fewer KV heads than query heads:

attn = MultiHeadAttention(
    num_heads=12,
    num_kv_heads=4,      # 4 KV heads → repeats KV tensors 3×

    d_model=768,
    num_layers=24,
    dtype=jnp.bfloat16,
    flash=True,
)
out = attn(x)

Key Source Files

File	Role	Link
needle/model/architecture.py	Defines MultiHeadAttention and backend-aware attention logic	architecture.py
needle/model/decode.py	Selects flash implementation during generation and decode loops	decode.py
needle/model/quantize.py	Provides _quantize.maybe_quant_kv helper functions for KV cache quantization	quantize.py
needle/model/run.py	Configures JAX settings and default dtypes that influence backend path selection	run.py

Summary

• Needle 2 uses jax.default_backend() to detect GPU, CPU, or TPU execution environments at runtime.
• CUDNN flash attention is activated automatically for GPUs using bfloat16 or float16 precision, fusing operations to reduce memory traffic.
• CPU and TPU platforms use a portable JAX implementation with explicit attention matrix computation and softmax.
• The MultiHeadAttention module supports both standard multi-head and multi-query attention configurations via num_kv_heads.
• Quantization hooks (_quantize.maybe_quant_kv) prepare tensors before computation to reduce memory footprint during inference.

Frequently Asked Questions

What hardware is required to use flash attention in Needle 2?

Flash attention via CUDNN requires an NVIDIA GPU with CUDA support. The query tensors must use either bfloat16 or float16 data types to trigger the optimized kernel path in needle/model/architecture.py. If running on CPU, TPU, or with float32 tensors, Needle 2 automatically falls back to the standard JAX implementation.

Does Needle 2 support TPU acceleration for attention layers?

Yes, Needle 2 supports TPU execution through the fallback JAX implementation in MultiHeadAttention. When jax.default_backend() returns "tpu", the system bypasses the CUDNN-specific code and uses the portable attention computation that explicitly materializes attention weights and applies softmax normalization.

How does Needle 2 handle different numbers of key-value heads?

The attention implementation supports Multi-Query Attention (MQA) and Grouped-Query Attention (GQA) by checking if num_kv_heads differs from num_heads. When fewer KV heads are specified, the tensors are repeated as needed within the fallback path to match the query head count, reducing memory bandwidth requirements during inference.

Can I force the fallback implementation even on a compatible GPU?

Yes, you can force the fallback path by setting dtype=jnp.float32 or by setting flash=False in the MultiHeadAttention constructor. While the flash parameter influences the configuration, the actual backend selection ultimately depends on the combination of jax.default_backend() and the data type of the input tensors.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}