Understanding the Role of Grouped Query Attention (GQA) in Needle 2

Grouped Query Attention (GQA) decouples the number of query heads from key/value heads in Needle 2’s transformer architecture, allowing multiple query heads to share KV representations to reduce memory usage and computation while maintaining model expressiveness.

Grouped Query Attention (GQA) serves as a fundamental efficiency mechanism in the Needle 2 transformer architecture from cactus-compute/needle. By configuring disparate counts for query and key/value heads, the system significantly reduces the memory footprint of the KV cache without sacrificing the diversity of attention patterns. This architectural choice is controlled through the TransformerConfig class and implemented in the MultiHeadAttention module.

How Grouped Query Attention Works in Needle 2

In standard multi-head attention, the number of query heads (num_heads) equals the number of key and value heads. Needle 2 breaks this constraint through GQA, implemented in needle/model/architecture.py lines 209-236 within the MultiHeadAttention class.

The mechanism operates through two distinct parameters:

• num_heads – Controls the total number of query heads that generate distinct attention patterns
• num_kv_heads – Determines how many key/value head groups exist; when this value is less than num_heads, multiple query heads share the same KV representation

When num_kv_heads is set lower than num_heads, the model forms query groups where several query heads attend to a single shared KV head. For example, with 12 query heads and 6 KV heads, queries are grouped 2-to-1, effectively halving the KV cache requirements.

Benefits of GQA in Needle 2 Architecture

Reduced KV Cache Memory and Computation

By grouping queries to share KV heads, Needle 2 stores and processes fewer key/value matrices. This reduction is critical for long-context inference and low-memory environments, as the KV cache often dominates memory consumption in transformer models. The computational savings scale directly with the ratio between num_heads and num_kv_heads.

Preserved Expressive Power Through Diverse Queries

Unlike methods that simply reduce head counts uniformly, GQA maintains distinct linear projections for each query head. Individual query heads retain unique weight matrices, allowing the model to capture diverse attention patterns and semantic relationships. This ensures that efficiency gains do not come at the cost of representational capacity.

Flexible Scaling for Production Workloads

Developers can increase num_heads to enrich query diversity without a proportional increase in KV memory costs. The TransformerConfig class (lines 58-78 in architecture.py) defaults num_kv_heads to num_heads // 2 for the standard preset, but supports manual adjustment. This flexibility enables precise trade-offs between inference speed and model quality based on hardware constraints.

Implementing GQA in Your Models

Configure GQA through the TransformerConfig object when instantiating SimpleAttentionNetwork:

# Standard GQA configuration: 12 query heads, 6 KV heads (2:1 grouping)

from needle.model.architecture import TransformerConfig, SimpleAttentionNetwork

cfg = TransformerConfig(
    d_model=768,
    num_heads=12,          # 12 distinct query heads

    num_kv_heads=6,        # 6 KV heads → queries grouped 2-to-1

    num_layers=27,
    flash=True,
)
model = SimpleAttentionNetwork(cfg)

To disable GQA and provide each query head with its own KV representation, set num_kv_heads equal to num_heads:

# Disabling GQA for maximum KV capacity (1:1 mapping)

cfg_gqa_off = TransformerConfig(
    d_model=768,
    num_heads=12,
    num_kv_heads=12,      # Each query head gets dedicated KV head

    num_layers=27,
    flash=True,
)
model_no_gqa = SimpleAttentionNetwork(cfg_gqa_off)

The internal reshaping logic in MultiHeadAttention.__call__ (lines 218-239) automatically handles the grouping:

# From architecture.py lines 218-239

head_dim = attn_dim // self.num_heads
kv_dim   = self.num_kv_heads * head_dim

# Queries reshaped to (B, num_heads, T, head_dim)

# Keys/Values reshaped to (B, num_kv_heads, T, head_dim)

# When num_kv_heads < num_heads, KV heads are repeated so each

# query group attends to the same shared representation

Core Files Implementing GQA

File	Contents
needle/model/architecture.py	Defines TransformerConfig (lines 58-78) and MultiHeadAttention (lines 209-236) with GQA-aware attention flow
needle/model/__init__.py	Exposes model classes that consume GQA configuration parameters
needle/model/run.py	Demonstrates instantiation and inference usage of SimpleAttentionNetwork with GQA settings

Summary

• Grouped Query Attention in Needle 2 allows num_heads to exceed num_kv_heads, creating query groups that share KV representations.
• Memory efficiency scales with the grouping ratio, reducing KV cache size by sharing keys and values across multiple query heads.
• Expressiveness is preserved because individual query heads maintain distinct projection matrices even when sharing KV heads.
• Configuration occurs through TransformerConfig in architecture.py, with num_kv_heads defaulting to half of num_heads but adjustable for custom trade-offs.
• Implementation is handled automatically in MultiHeadAttention, requiring no manual intervention beyond parameter setting.

Frequently Asked Questions

What is the difference between num_heads and num_kv_heads in Needle 2?

num_heads controls the total count of query heads that generate distinct attention patterns, while num_kv_heads specifies how many key/value head groups exist. When num_kv_heads is less than num_heads, multiple query heads share KV representations through Grouped Query Attention, reducing memory usage while keeping query diversity intact.

How does GQA reduce memory usage without sacrificing accuracy?

GQA reduces the size of the KV cache by storing fewer key and value matrices—only num_kv_heads sets instead of num_heads sets. Since individual query heads retain their own linear projections, they can still capture diverse attention patterns. The shared KV representations provide sufficient signal for accurate attention computation while consuming less memory.

Can I disable Grouped Query Attention in Needle 2?

Yes. Set num_kv_heads equal to num_heads in your TransformerConfig to establish a one-to-one mapping between query and KV heads. This configuration effectively disables GQA and maximizes KV capacity, which may be desirable when memory constraints are not a concern and maximum representational capacity is required.

Which file contains the core MultiHeadAttention implementation?

The MultiHeadAttention class is defined in needle/model/architecture.py, specifically lines 209-236, with the grouping logic implemented in the __call__ method around lines 218-239. This module handles the automatic reshaping and repetition of KV heads when num_kv_heads is less than num_heads.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}