How Needle 2's Memory Management Differs from Traditional Transformers

Needle 2 replaces the exponential memory growth of traditional transformers with a deterministic, budget-constrained hybrid system that combines a sliding-window KV cache and a persistent Engram table.

Needle 2, developed in the cactus-compute/needle repository, implements a Simple Attention Network that fundamentally restructures how transformer architectures handle memory. While conventional models store key-value pairs for every token in the sequence—leading to linear memory growth and quadratic attention costs—Needle 2 caps memory usage at approximately 28 MiB through specialized caching mechanisms and fixed-size embedding tables. This technical analysis explores the specific source code implementations that enable Needle 2's constant-memory scaling.

The Sliding-Window KV Cache vs. Full-Context Storage

Traditional transformers maintain a full KV cache that stores key and value tensors for every token in the sequence. This approach consumes memory proportional to $O(L \times d_{\text{model}})$, where $L$ represents sequence length, and incurs quadratic attention computation costs during generation.

Needle 2 eliminates this bottleneck through a budget-constrained sliding window. In needle/model/architecture.py, the effective_kv_window() function (lines 614-618) calculates the maximum number of recent tokens to retain based on a fixed byte budget rather than sequence length. The implementation restricts the KV cache to a few hundred tokens—typically around 400 tokens—regardless of how long the conversation grows.

from needle.model.architecture import effective_kv_window
from needle.model.run import NeedleModel

# Load model with default 28 MiB memory budget

model = NeedleModel.from_pretrained("cactus-needle-2")

# Query the deterministic KV window size

kv_win = effective_kv_window(model.config)
print(f"Effective KV window: {kv_win} tokens")  # Output: ~400 tokens

This sliding window ensures that RAM usage remains constant during inference, contrasting sharply with traditional transformers where memory requirements grow indefinitely with context length.

The Engram Module: Persistent Long-Range Memory

While sliding windows limit memory, they risk losing distant context. Needle 2 solves this through the Engram module, a fixed-size embedding table that provides persistent long-range storage without the memory overhead of full KV caching.

Located in needle/model/architecture.py (lines 81-96), the Engram module hashes token sequences into a fixed-size table and retrieves them on-the-fly during attention computation. The engram_indices() function (lines 46-58) generates hash-based indices that can be reused across layers, enabling the model to reference distant tokens without storing their full key-value representations.

Key characteristics:

• Fixed-size storage: The Engram table maintains constant capacity regardless of sequence length.
• Cross-layer reuse: Engram indices generated once can be shared across multiple transformer layers.
• Low bandwidth: Retrieval operates on compressed hash representations rather than full-dimensional KV tensors.

This architecture allows Needle 2 to capture long-range dependencies that would normally require the full KV cache in traditional transformers, but with bandwidth comparable to retrieval-augmented models.

Deterministic Memory Budget Calculation

Unlike traditional transformers where memory limits depend on available GPU/CPU RAM, Needle 2 implements explicit memory budgeting. The kv_budget_window formula in needle/model/architecture.py (lines 1004-1011) calculates the exact memory footprint based on:

• Model width (hidden dimensions)
• Number of transformer layers
• KV-grouping configuration (GQA/MQA settings)
• Allocated Engram slots

This calculation guarantees that the model fits within approximately 28 MiB of RAM for a complete inference session—a stark contrast to traditional models where memory scales linearly with sequence length.

# The budget automatically accounts for all memory consumers:

# - Sliding-window KV cache

# - Engram table slots

# - Quantization overhead

# Result: deterministic ~28 MiB usage regardless of prompt length

output = model.generate("Extremely long prompt..." * 1000)  # Still ~28 MiB

Per-Token Quantization Strategy

Traditional transformer quantization typically applies uniform weight compression, but Needle 2 extends compression to per-token quantization of activation caches. The maybe_quant_kv function in needle/model/architecture.py (lines 42-44) applies dynamic quantization to both the KV cache and Engram outputs.

This approach reduces the memory footprint of cached representations beyond standard compression techniques. By quantizing on a per-token basis rather than applying global schemes, Needle 2 maintains precision for recent tokens while aggressively compressing older cached values.

Attention Mechanism Implementation

The attention computation itself reflects these memory constraints. While traditional transformers use standard dot-product attention over the full $L \times L$ matrix, Needle 2 leverages JAX's dot_product_attention with flash-attention optimization—but only over the constrained kv_window and optional Engram-derived keys/values.

This implementation keeps attention complexity close to linear in sequence length $O(L \times \text{window_size})$ rather than quadratic $O(L^2)$. When the backend supports GPU acceleration, the flash-attention path further optimizes the memory-bandwidth bound operations over the limited KV window.

Summary

• Sliding-window caching: effective_kv_window() enforces a fixed token budget (typically ~400 tokens) instead of storing all past KV pairs.
• Engram persistence: Fixed-size hash tables provide long-range memory without linear growth, implemented in the Engram module (lines 81-96).
• Deterministic budgeting: The kv_budget_window formula (lines 1004-1011) guarantees ~28 MiB RAM usage regardless of sequence length.
• Per-token quantization: maybe_quant_kv (lines 42-44) compresses both cache and Engram outputs dynamically.
• Linear attention cost: Flash-attention operates only over the constrained window and Engram indices, avoiding quadratic scaling.

Frequently Asked Questions

What is the Engram table in Needle 2?

The Engram table is a fixed-size embedding storage mechanism that hashes token sequences into persistent memory slots. Located in needle/model/architecture.py (lines 81-96), it allows the model to retrieve distant context without storing full key-value tensors for every past token. The engram_indices() function (lines 46-58) generates reusable hash indices that work across layers, providing long-range dependency modeling while maintaining constant memory usage.

How does Needle 2 maintain constant memory usage during long conversations?

Needle 2 combines three mechanisms: a sliding-window KV cache that retains only recent tokens (calculated by effective_kv_window()), a fixed-size Engram table for distant context, and per-token quantization via maybe_quant_kv(). The kv_budget_window formula explicitly calculates memory requirements based on model architecture rather than sequence length, capping usage at approximately 28 MiB regardless of input size.

How does sliding-window attention differ from standard transformer attention?

Standard transformers compute attention over the full sequence length, requiring $O(L^2)$ memory and computation. Needle 2 restricts attention to a budget-constrained window (typically ~400 tokens) using dot_product_attention with flash-attention optimization. This limits computation to $O(L \times \text{window_size})$, while the Engram module provides additional context without requiring full attention over distant tokens.

What is the default memory budget for Needle 2 models?

According to the source code in needle/model/architecture.py, the default configuration targets approximately 28 MiB of RAM for a complete inference session. This budget accommodates the sliding-window KV cache, Engram table slots, and quantization overhead through the deterministic kv_budget_window calculation (lines 1004-1011), ensuring the model fits on resource-constrained devices.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}