Bounded Memory Implementation with 256-Token Sliding Window and KV Sinks in Needle 2

Needle 2 caps KV cache memory usage by applying a fixed-size sliding-window attention mask (commonly 256 tokens) combined with optional KV sinks that force specific tokens to persist outside the window, limiting memory growth while preserving essential context.

Needle 2 introduces a bounded-memory attention mechanism designed for resource-constrained environments. By implementing a configurable sliding window over the key-value (KV) cache and supplementing it with persistent "KV sinks," the architecture ensures strict memory bounds without sacrificing critical context. This article examines the implementation details found in the cactus-compute/needle repository, specifically how the 256-token window is calculated, stored, and enforced at runtime.

KV Window Budgeting and Storage

The sliding window size is determined at two stages: first during model configuration via budgeting functions, and second during export via the CACT header format.

Calculating the Budgeted Window

The model derives a theoretical KV cache budget based on architectural parameters. The kv_budget_window function computes this from the hidden size, number of layers, KV-heads, and engram sites. This yields a "budgeted" window size representing the maximum sustainable KV cache length for the target hardware.

Resolving the Effective Window

The final window width is resolved by effective_kv_window, implemented in needle/model/architecture.py at lines 14-16. This function selects the actual window size used during training and inference, which can be overridden via the kv_window configuration field. When specified, this value determines the exact number of recent tokens retained—in practice often set to 256 tokens (2⁸), though the source demonstrates the logic with 64 tokens (2⁶).

Persisting Window Size in CACT Headers

To ensure runtime consistency, the effective window is serialized into the model checkpoint. The write_export function in needle/model/export.py (lines 27-30) embeds the kv_window value directly into the CACT header. During loading, read_export restores this value, exposing it via the configuration object so the inference engine knows the exact window width the model was trained with.

Runtime Sliding Window Masking

During forward passes, Needle applies the window constraint through attention masking rather than physical cache eviction. This occurs in the hidden_cells function within needle/model/architecture.py.

Implementing the Window Constraint

The mask construction combines causal, padding, and window masks. When the window argument is provided, the implementation limits attention to the last W positions:

mask = (make_causal_mask(tokens.shape[1])
        & make_padding_mask(tokens, cfg.pad_token_id))
if window:
    pos = jnp.arange(tokens.shape[1])
    recent = ((pos[:, None] - pos[None, :]) < window)[None, None, :, :]
    keep = recent if sink is None else (recent | sink[:, None, None, :])
    mask = mask & keep

This logic (found at lines 44-49 of architecture.py) creates a boolean mask where positions outside the sliding window receive a negative infinity attention score. For a 256-token window, passing window=256 ensures only the most recent 256 positions participate in attention, bounding memory complexity to O(256×d) per head rather than O(n×d).

KV Sinks for Persistent Context

While the sliding window discards old tokens to save memory, certain tokens—such as document prefixes or tool headers—must persist across all windows. Needle implements KV sinks to mark these positions for retention.

Defining Sink Masks

Sinks are optional boolean arrays passed as the sink argument to hidden_cells. When provided, the mask construction modifies the keep tensor:

keep = recent if sink is None else (recent | sink[:, None, None, :])

This OR operation ensures sink positions remain attended to even when they fall outside the recent window.

Integration with Packing Masks

For batched inference with packing, the make_causal_packing_mask function (lines 24-30 of architecture.py) handles sinks via segment IDs:

if window:
    pos = jnp.arange(T)
    recent = (pos[:, None] - pos[None, :]) < window
    sink = (jnp.zeros_like(seg_ids, dtype=jnp.bool_) if prefix is None
            else prefix > 0)
    mask = mask & (recent[None, :, :] | sink[:, None, :])

Here, the prefix flag indicates which positions should serve as sinks. This allows the model to maintain attention to static context while processing new tokens within the rolling buffer.

Practical Code Examples

The following patterns demonstrate how to leverage the bounded-memory features in Needle 2.

Loading a Checkpoint and Discovering its KV Window

Retrieve the baked-in window size from the CACT header to know the model's training configuration:

from needle.model.export import read_export

metadata, tensors = read_export("my_model.cact")
print("KV window baked into the model:", metadata["kv_window"])  # e.g., 256

Running Inference with a 256-Token Sliding Window

Apply the window constraint during the forward pass to limit memory usage:

from needle.model.run import load_checkpoint, build_model
import jax.numpy as jnp

params, cfg, _ = load_checkpoint("my_model.cact", return_run=True)
model = build_model(cfg)          
model = model.apply(params)       

# tokens: (batch, seq_len) int32

tokens = jnp.array([[...]])  # Your input tokens

# Enforce a 256-token KV window (2^8)

logits = model(tokens, window=256)

Preserving KV for Special Tokens with Sinks

Force specific positions (e.g., a tool-use marker) to remain in cache indefinitely:

# Create a sink mask preserving the first token (e.g., a special header)

sink_mask = jnp.zeros_like(tokens, dtype=bool)
sink_mask = sink_mask.at[:, 0].set(True)   # Keep first token forever

# Apply both the 256-token window and the sink mask

logits = model(tokens, window=256, sink=sink_mask)

Summary

• Budgeted Calculation: kv_budget_window and effective_kv_window in needle/model/architecture.py determine the optimal window size based on architecture and configuration.
• Persistent Storage: The kv_window value is embedded in the CACT header via write_export in needle/model/export.py, ensuring the runtime uses training-consistent bounds.
• Mask-Based Enforcement: The hidden_cells function applies the window via tensor masking (recent < window) rather than cache deletion, supporting dynamic window sizes including 256 tokens.
• KV Sinks: Optional boolean masks allow specific tokens to bypass the sliding window, enabling long-range attention to prefixes or special markers without breaking memory bounds.
• Edge Optimization: This bounded-memory design enables deployment on mobile and edge GPUs where unbounded KV caches would cause OOM errors.

Frequently Asked Questions

How does the 2S⁶ notation relate to the 256-token window mentioned in the question?

The 2S⁶ notation indicates scientific notation for powers of two used in the Needle 2 codebase, where 2S⁶ equals 2⁶ or 64 tokens. A 256-token window follows the same pattern as 2⁸ (2S⁸). The implementation is size-agnostic; the window parameter in hidden_cells accepts any integer value, allowing models to be configured for 64, 256, or other power-of-two boundaries depending on memory constraints.

What happens to tokens that slide outside the 256-token window?

Tokens outside the recent 256 positions are masked out in the attention computation within hidden_cells. Their key-value pairs remain physically allocated in device memory but are effectively ignored by the attention scores due to the boolean mask (recent tensor). If no KV sinks are defined, these tokens no longer influence the model's output. With sinks, specified tokens remain attended to via the OR-based mask combination even after falling outside the window.

Can KV sinks be used for dynamic content or only static prefixes?

KV sinks are controlled by boolean masks that can be constructed dynamically at runtime. While the make_causal_packing_mask example uses prefix > 0 to identify static document headers, you can construct arbitrary sink tensors in Python. For example, you could mark positions containing specific entity mentions or tool calls based on runtime logic, allowing the model to maintain "sticky" attention to dynamically identified critical tokens across the 256-token rolling window.

Where is the sliding window logic implemented if I need to modify the masking behavior?

The core masking logic resides in needle/model/architecture.py. Specifically, the hidden_cells function (lines 44-49) handles the standard causal window masking, while make_causal_packing_mask (lines 24-30) handles packed sequence scenarios. Both functions accept the window and sink arguments. To modify behavior—such as implementing a strided window or attention sink pooling—you would edit the mask construction logic in these functions before the mask is applied to attention scores.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}