# How Grouped Query Attention (GQA) Improves Efficiency in Needle 2

> Discover how Grouped Query Attention (GQA) boosts Needle 2 efficiency by 50% through shared KV projections, reducing computation without sacrificing model power.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-08-14

---

**Grouped Query Attention (GQA) improves efficiency in Needle 2 by grouping query heads and sharing key-value projections across fewer KV heads, reducing computational overhead by approximately 50% while maintaining model capacity.**

Grouped Query Attention (GQA) serves as a core architectural optimization in the Needle 2 transformer framework, fundamentally redesigning how attention mechanisms handle query, key, and value projections. Unlike conventional multi-head attention where each head maintains independent QKV triplets, GQA strategically shares key and value computations across grouped query heads to minimize memory bandwidth bottlenecks. This implementation, found in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), enables Needle 2 to achieve faster inference speeds without sacrificing the expressive power of multi-head attention.

## The Mechanics of Grouped Query Attention in Needle 2

### Reduced Projection Work

In standard multi-head attention layers, every attention head computes independent key and value projections, resulting in computational cost proportional to `num_heads`. GQA inverts this paradigm by allowing `num_kv_heads` to be set to a fraction of `num_heads`, typically halving the KV head count. According to the Needle 2 source code, the expensive K/V projection matrices are computed only once per KV-head group, while query projections remain lightweight linear operations on the hidden dimension. When you configure `num_heads=8` and `num_kv_heads=4`, the framework cuts K/V projection costs by approximately 50% because the same key and value tensors serve two query heads each.

### Lower Memory Traffic

Because K and V tensors are generated once per KV-head rather than once per query head, the volume of intermediate data stored and shuffled through memory drops dramatically. In [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), the attention layer implementation reuses cached key-value pairs across grouped query heads, reducing RAM consumption and memory bandwidth pressure. This optimization proves particularly effective on hardware where memory bandwidth constrains throughput, such as GPUs and CPU architectures with limited cache hierarchies.

### Faster Inference Performance

Fewer matrix multiplications combined with reduced data movement translate directly into lower latency during the attention computation step. The Needle 2 implementation leverages this efficiency gain during autoregressive decoding, where KV-cache bandwidth often dominates inference time. By sharing KV projections across groups, the framework minimizes the memory-bound operations that typically bottleneck transformer inference.

## Configuration and Implementation

The GQA mechanism is explicitly configured through the `TransformerConfig` dataclass defined in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) (lines 61-65). This configuration exposes the trade-off between computational efficiency and attention granularity:

```python

# needle/model/architecture.py – TransformerConfig definition

# (lines 61-65)

@dataclass
class TransformerConfig:
    vocab_size: int = 8192
    d_model: int = 512
    attn_dim: int = 0
    num_heads: int = 8               # total query heads

    num_kv_heads: int = 4            # KV heads shared among query heads

    num_layers: int = 12
    …

```

During model construction, the attention module uses `num_kv_heads` to instantiate shared key/value projection layers, while maintaining separate query projections for each of the `num_heads`. This grouping strategy allows queries to attend to the same set of keys and values while distributing the attention computation across multiple query heads, preserving model capacity despite the reduced KV computation.

## Practical Code Examples

To build a Needle 2 model with GQA, specify the ratio between query heads and KV heads through the configuration object:

```python

# example.py – building a Needle 2 model with GQA

from needle.model.architecture import TransformerConfig, build_model

# Define a model that uses 12 query heads but only 6 shared KV heads

cfg = TransformerConfig(
    d_model=768,
    num_heads=12,
    num_kv_heads=6,          # ← GQA: 2 query heads per KV head

    num_layers=27,
    max_seq_len=2048,
)

model = build_model(cfg)     # <-- creates the attention layers with grouped KV

print(model)                 # shows the layer shapes, e.g. (batch, seq, d_model)

```

For inference, the grouped KV heads remain cached and shared across generation steps:

```python

# inference.py – using the model for a forward pass

import jax
from needle import Needle

# Load a pre-trained checkpoint (the KV heads are already grouped)

needle = Needle(weights="needle-2-base.cact", config=cfg)

prompt = "Explain the benefits of Grouped Query Attention."
output = needle(prompt)      # Fast inference thanks to reduced KV work

print(output)

```

## Summary

- **Grouped Query Attention reduces KV projection costs by 50%** when `num_kv_heads` is set to half of `num_heads`, as implemented in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py).
- **Memory bandwidth bottlenecks are alleviated** by generating K and V tensors once per group rather than once per head, minimizing data movement during attention computation.
- **Configuration is controlled via `TransformerConfig`** using the `num_kv_heads` parameter, which defines how many query heads share each key-value projection.
- **Inference speed improvements** are most pronounced on memory-constrained hardware, where reduced KV-cache traffic directly translates to lower latency.

## Frequently Asked Questions

### What is the difference between GQA and standard multi-head attention?

Standard multi-head attention computes independent key, value, and query projections for every attention head, resulting in `num_heads` separate QKV triplets. GQA modifies this by computing keys and values for a smaller number of heads (`num_kv_heads`) and sharing these projections across multiple query heads, reducing computational redundancy while maintaining attention diversity through independent queries.

### How do I configure GQA in Needle 2?

Configure GQA by instantiating a `TransformerConfig` object from [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) and setting `num_kv_heads` to a value smaller than `num_heads`. For example, setting `num_heads=12` and `num_kv_heads=6` creates two query heads per KV-head group. The `build_model()` function automatically constructs attention layers that respect this grouping ratio.

### Does GQA reduce model quality compared to full multi-head attention?

No, GQA maintains model capacity because each query head retains independent query projections and can attend to shared keys and values from different perspectives. The Needle 2 implementation preserves the expressive power of multi-head attention while reducing only the redundant KV computation, as evidenced by the architecture's ability to distribute multiple queries across shared KV representations.

### Where is the GQA mechanism implemented in the Needle 2 codebase?

The GQA configuration is defined in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) within the `TransformerConfig` dataclass (lines 61-65), specifically through the `num_kv_heads` parameter. The actual attention layer implementation that performs the grouping and sharing of KV projections resides in the same file's attention module construction logic, where it uses these configuration values to instantiate shared projection layers.