# How Grouped Query Attention Reduces Memory Bandwidth in Needle 2

> Discover how Grouped Query Attention in Needle 2 slashes memory bandwidth by sharing key value heads across query groups. Reduce KV cache memory traffic effectively.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-08-29

---

**Grouped Query Attention (GQA) in Needle 2 reduces memory bandwidth by generating fewer key/value heads than query heads and sharing them across query groups, cutting KV cache memory traffic by a factor of `num_heads / num_kv_heads`.**

Needle 2, an open-source transformer implementation by **cactus-compute/needle**, optimizes attention mechanisms for efficient inference. By implementing **Grouped Query Attention memory bandwidth** optimizations directly in the `MultiHeadAttention` class, the framework significantly reduces the data movement bottleneck that dominates transformer performance at scale.

## What Is Grouped Query Attention in Needle 2?

Grouped Query Attention is a memory-efficient variant of multi-head attention where the number of key/value heads (`num_kv_heads`) is smaller than the number of query heads (`num_heads`). Instead of maintaining separate key and value tensors for every query head, query heads are partitioned into groups that share the same key/value pairs.

In Needle 2's implementation in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), this is controlled by two distinct parameters:

- **`num_heads`**: The total number of query attention heads
- **`num_kv_heads`**: The number of distinct key/value head groups

### The Dimension Calculation Strategy

The core of the bandwidth reduction lies in how dimensions are computed inside `MultiHeadAttention.__call__`. The source code calculates head dimensions as follows:

```python
head_dim = attn_dim // self.num_heads
kv_dim = self.num_kv_heads * head_dim

```

This ensures that the key and value projection layers output `kv_dim` features rather than the full attention dimension, immediately reducing the memory footprint of the KV cache by the grouping factor.

## How GQA Reduces Memory Bandwidth

Memory bandwidth pressure in transformers primarily stems from reading and writing the key/value caches during the attention operation. By storing fewer unique KV head states, Needle 2 proportionally reduces the amount of data transferred across the memory bus.

### KV Cache Dimension Reduction

Standard multi-head attention requires storing `num_heads` sets of keys and values. In Needle 2's GQA implementation, only `num_kv_heads` sets are materialized:

```python
k = nn.Dense(kv_dim, kernel_init=default_init("k"))(x)
v = nn.Dense(kv_dim, kernel_init=default_init("v"))(x)

```

When `num_kv_heads = 6` and `num_heads = 12`, the KV cache size is halved compared to standard attention, directly reducing memory bandwidth requirements by 50% during cache updates and attention lookups.

### The Repeat Strategy for Non-Flash Attention

When Flash Attention is disabled (`self.flash = False`), Needle 2 uses a tiling strategy to share the reduced KV set across query heads without materializing redundant data in memory. The source code explicitly repeats the KV tensors along the head axis to match the query head count for computation purposes only:

```python
repeats = self.num_heads // self.num_kv_heads
if repeats > 1:
    k = jnp.repeat(k, repeats, axis=1)
    v = jnp.repeat(v, repeats, axis=1)

```

This `jnp.repeat` operation happens during the forward pass but operates on a significantly smaller base tensor, ensuring that the memory bandwidth-intensive cache reads only touch the compressed `num_kv_heads` representation.

## Implementation Details in MultiHeadAttention

The GQA logic is encapsulated entirely within the `MultiHeadAttention` class in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py). The implementation respects the `flash` configuration flag to determine whether to use the repetition strategy or native grouped attention kernels.

Key implementation characteristics include:

- **Dynamic shape handling**: The code computes `repeats = self.num_heads // self.num_kv_heads` dynamically, supporting arbitrary grouping ratios
- **Dense layer optimization**: The `kv_dim` calculation ensures that `nn.Dense` layers for keys and values operate on the reduced dimension
- **Preset configurations**: Needle 2 includes optimized presets such as `num_heads=12, num_kv_heads=6` that provide a 2× bandwidth reduction out of the box

## Practical Code Example: Configuring GQA in Needle 2

To leverage Grouped Query Attention for reduced memory bandwidth, instantiate the `MultiHeadAttention` module with asymmetric head counts:

```python
import jax.numpy as jnp
from needle.model.architecture import MultiHeadAttention

# Configure GQA with 12 query heads and 6 KV heads (2× bandwidth reduction)

attention_layer = MultiHeadAttention(
    num_heads=12,
    num_kv_heads=6,
    d_model=768,
    num_layers=27,
    dtype=jnp.bfloat16,
    flash=False  # Non-flash path demonstrates the repeat strategy

)

# Example input: (batch_size, sequence_length, d_model)

x = jnp.ones((2, 128, 768), dtype=jnp.bfloat16)

# Forward pass uses reduced KV bandwidth automatically

output = attention_layer(x)
print(f"Output shape: {output.shape}")  # (2, 128, 768)

```

For debugging or verification, you can inspect the internal dimension reduction:

```python
def inspect_gqa_dimensions(layer, x):
    """Demonstrate the KV dimension reduction in Needle 2 GQA."""
    attn_dim = layer.d_model
    head_dim = attn_dim // layer.num_heads
    kv_dim = layer.num_kv_heads * head_dim
    
    print(f"Query heads: {layer.num_heads}")
    print(f"KV heads: {layer.num_kv_heads}")
    print(f"Head dimension: {head_dim}")
    print(f"KV dimension: {kv_dim} (vs {attn_dim} for standard attention)")
    print(f"Bandwidth reduction factor: {layer.num_heads / layer.num_kv_heads}x")

# Usage

inspect_gqa_dimensions(attention_layer, x)

```

## Summary

- **Needle 2** implements Grouped Query Attention in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) via the `MultiHeadAttention` class, allowing `num_kv_heads` to be smaller than `num_heads`.
- The technique reduces **memory bandwidth** by storing and transferring only `num_kv_heads` key/value tensors rather than one per query head.
- The bandwidth reduction factor equals the ratio `num_heads / num_kv_heads`, with common configurations like 12 query heads and 6 KV heads achieving a **2× reduction**.
- When Flash Attention is disabled, the framework uses `jnp.repeat` to tile the compressed KV tensors for computational compatibility without increasing memory traffic.
- Preset configurations in the repository provide optimized GQA settings for immediate deployment.

## Frequently Asked Questions

### How does Grouped Query Attention differ from standard Multi-Head Attention?

Standard Multi-Head Attention generates distinct key and value tensors for every query head, resulting in `num_heads` separate KV projections. Grouped Query Attention generates only `num_kv_heads` KV projections, where multiple query heads share the same key/value pair. In Needle 2, this is implemented by computing `kv_dim = self.num_kv_heads * head_dim` and projecting keys/values into this reduced space.

### What is the memory bandwidth reduction factor when using GQA?

The memory bandwidth reduction factor is exactly `num_heads / num_kv_heads`. For example, if `num_heads=12` and `num_kv_heads=6`, the KV cache memory traffic is reduced by 50%. This directly translates to lower memory bandwidth utilization during the attention computation, which is often the bottleneck in transformer inference.

### Does Needle 2 support Flash Attention with GQA?

Yes, Needle 2 supports both Flash Attention and the standard attention path with GQA. When `flash=True`, the implementation leverages optimized kernels that handle grouped attention natively. When `flash=False`, the code explicitly uses `jnp.repeat` to expand the reduced KV tensors to match query heads, maintaining correctness while keeping the underlying cache storage minimal.

### Where is the GQA logic implemented in the Needle codebase?

The core GQA implementation resides in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) within the `MultiHeadAttention` class. The key logic for dimension calculation (`kv_dim = self.num_kv_heads * head_dim`) and the conditional repetition strategy (`k = jnp.repeat(k, repeats, axis=1)`) are both located in the `__call__` method of this class, as confirmed by the source code analysis of the cactus-compute/needle repository.