How Needle Maintains Bounded Memory Using a 256-Token Sliding Window

Needle caps runtime memory at roughly 28 MiB regardless of conversation length by pinning tool-related KV entries as permanent sinks and restricting the attention cache to a 256-token sliding window that rotates old tokens out of memory.

The cactus-compute/needle repository implements a transformer architecture specifically designed for constant-memory inference. Unlike standard language models that suffer from linear memory growth as context length increases, Needle maintain bounded memory using a 256-token sliding window mechanism that keeps the KV cache size fixed while preserving access to critical tool functions.

Configuring the Sliding Window in Model Headers

The sliding window size is baked into the model binary during export. In needle/model/export.py, the header structure stores kv_window as a persistent field that defines the maximum attention span the model was trained to handle.

When loading a .cact model file, you can inspect this configuration to verify the bounded memory constraints:

from needle.model.export import read_header

header = read_header("needle2.cact")
print("KV sliding window:", header.kv_window)   # → 256 (tokens)

This value defaults to 256 tokens across standard Needle distributions, establishing the foundation for the constant-memory guarantee.

Runtime Budget and Effective Window Calculation

During inference, the system calculates the actual window size by balancing hardware constraints against the model's training configuration. The kv_budget_window() function in needle/model/architecture.py (lines 998-1016) computes a hardware-budgeted KV size of approximately 11 MiB, then derives the effective_kv_window by selecting the smaller of the budget limit and the header's kv_window value.

This dual-layer constraint ensures that even if a user specifies a larger window, the runtime will clamp the effective size to prevent memory overflow:

from needle.model.architecture import TransformerConfig
from needle.model.export import export

cfg = TransformerConfig(
    d_model=1024,
    num_heads=8,
    num_layers=12,
    kv_window=256,          # Enforce 256-token sliding window

    # … other hyper-parameters …

)

export(params, cfg, tokenizer, out_path="my_needle.cact")

Pinning Tools as Permanent KV Sinks

To maintain tool accessibility without breaking the memory bound, Needle implements KV sinks for tool-related keys and values. While the sliding window aggressively evicts old conversation tokens, the decoder preserves tool KV entries in fixed memory locations outside the rotating buffer.

According to the repository's README, this design pins "the tools pinned as KV sinks, so total memory stays near 28 MB no matter how long the conversation runs." This separation allows the model to reference tools throughout arbitrarily long interactions without expanding the cache footprint.

Attention Masking in the Decoder

The sliding window is enforced at the attention mechanism level through strategic masking. In needle/model/decode.py (lines 81-108), the decoder constructs an attention mask that restricts each position to attend only to:

• Tokens within the last kv_window positions (the sliding window)
• The pinned tool KV entries (permanent sinks)

This masking prevents the model from attending to evicted tokens, effectively creating a circular buffer where new tokens overwrite the oldest cached values once the 256-token limit is reached.

You can observe this bounded behavior during inference:

import needle

agent = needle.Needle(weights="my_needle.cact", tools=[my_tool])

# The internal KV cache never grows beyond the configured 256-token window

response = agent.run("Continue the story for another paragraph.") 
print(response["results"])

Summary

• Fixed header configuration: The kv_window field in needle/model/export.py stores the 256-token limit in the model binary.
• Runtime enforcement: kv_budget_window() in needle/model/architecture.py clamps the effective window to the smaller of hardware budget (≈11 MiB) or configured size.
• Tool persistence: Tool KV entries bypass the sliding window by being pinned as permanent sinks, ensuring constant-time tool access.
• Decoder masking: needle/model/decode.py applies sliding-window masks that restrict attention to recent tokens plus pinned sinks, preventing memory growth regardless of conversation length.

Frequently Asked Questions

What happens when a conversation exceeds 256 tokens?

When the conversation length surpasses the kv_window limit of 256 tokens, the oldest conversation tokens are evicted from the KV cache. The decoder in needle/model/decode.py masks out these evicted positions, so the model can only attend to the most recent 256 tokens plus any pinned tool KV sinks. This rotation maintains constant memory usage rather than allocating new buffers.

How does the KV budget calculation prevent memory overflow?

The kv_budget_window() function computes a hardware-specific budget of approximately 11 MiB for the KV cache. When initializing inference, Needle compares this hardware limit against the model's kv_window setting and assigns effective_kv_window to the smaller value. This guarantees that even on memory-constrained devices, the cache never exceeds physical RAM capacity.

Why are tool KV entries pinned as sinks instead of using the sliding window?

Tools require persistent availability throughout the conversation, but the sliding window constantly evicts old tokens. By pinning tool-related keys and values as permanent sinks outside the rotating buffer, Needle ensures that tool functions remain callable even after thousands of tokens have passed, all without expanding the cache size beyond the 28 MiB footprint.

Can I customize the 256-token sliding window size?

Yes. When exporting a custom .cact model using needle/model/export.py, you can specify any kv_window value in the TransformerConfig. However, the runtime will still apply the kv_budget_window() constraint from needle/model/architecture.py, so the effective window may be smaller than your specification if hardware limits are exceeded.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}