How the 256-Token Sliding Window Works with Tools as KV Sinks in Needle

Needle implements a fixed-memory KV cache by combining a 256-token sliding window for conversation history with permanent tool definitions pinned as KV sinks, keeping total cache size near 28 MiB regardless of chat length.

The cactus-compute/needle project solves a critical inference problem: LLM KV caches grow linearly with sequence length, eventually exhausting GPU memory. Needle's architecture keeps the KV cache bounded through a dual mechanism implemented in needle/model/architecture.py and needle/model/decode.py. This article explains exactly how the 256-token sliding window interacts with tool-based KV sinks to achieve constant memory usage.

The Two-Part Memory Architecture

Needle's memory management relies on two cooperating components that operate at different stages of the inference pipeline.

Effective KV Window: Bounding the Cache Size

The effective KV window determines how many recent tokens retain their key-value entries in the cache. The computation happens in needle/model/architecture.py through two functions.

First, kv_budget_window converts a fixed byte budget into a token limit:

def kv_budget_window(config):
    head_dim = (config.attn_dim or config.d_model) // config.num_heads
    kv = config.num_kv_heads * head_dim
    d, L = config.d_model, config.num_layers
    sites = len(tuple(getattr(config, "engram_layers", (2, 15))))
    per_pos = (L * (2 * kv + 2 * (kv // KV_GROUP) * 4)
               + sites * (d + (KV_GROUP) * 4))
    # 11 MiB + 0.5 MiB budget for KV cache

    window = (KV_BUDGET_BYTES // per_pos) // KV_GROUP * KV_GROUP
    return max(KV_WINDOW_MIN, min(window, config.max_seq_len))

Key parameters from the source:

• KV_BUDGET_BYTES ≈ 11 MiB — the hard memory ceiling for KV storage
• KV_WINDOW_MIN = 160 — guaranteed minimum window size
• Default configuration yields 256 tokens (2⁶), matching the documented behavior

The effective window applies user overrides:

def effective_kv_window(config):
    budget = kv_budget_window(config)
    return min(budget, config.kv_window) if config.kv_window else budget

If config.kv_window is unset, the budget-derived 256-token window applies.

KV Sinks for Tools: Pinning Critical Context

While the sliding window drops old tokens, tool definitions must remain permanently accessible. Needle solves this by marking tool tokens as KV sinks — entries that bypass eviction.

Tool delimiters are defined in needle/model/tokenizer.py:

TOOLS_START = "<tools>"
TOOLS_END = "</tools>"

The masking logic in needle/model/decode.py implements the sink mechanism:

def make_causal_packing_mask(seg_ids, prefix=None, window=0):
    T = seg_ids.shape[1]
    causal = jnp.tril(jnp.ones((T, T), dtype=jnp.bool_))
    block  = (seg_ids[:, :, None] == seg_ids[:, None, :]) & (seg_ids[:, :, None] > 0)
    mask   = block & causal[None, :, :]

    if window:
        pos   = jnp.arange(T)
        recent = (pos[:, None] - pos[None, :]) < window          # ← sliding window

        sink   = (jnp.zeros_like(seg_ids, dtype=jnp.bool_) if prefix is None
                  else prefix > 0)                               # ← KV sink

        mask = mask & (recent[None, :, :] | sink[:, None, :])
    return mask[:, None, :, :]

The critical logic: mask & (recent | sink). A token is attended to if it is either within the recent window or designated as a sink. The prefix parameter receives a boolean mask where tool tokens are marked True.

End-to-End Execution Flow

Understanding the sliding window with tools requires tracing the complete inference path:

1. Tool injection — Needle(tools=tools_json) parses tools in needle/agent/tools.py and wraps them with <tools> … </tools> delimiters
2. Tokenization — needle/model/tokenizer.py assigns segment IDs: tool tokens receive ID 1, conversation tokens receive ID 2
3. Mask construction — Each generation step calls make_causal_packing_mask with prefix=(seg_ids == 1) to identify sinks and window=effective_kv_window(config)
4. Cache management — JAX arrays store KV pairs only for positions passing the mask filter

Memory footprint remains ≈ 28 MiB total: ~11 MiB KV budget plus model weights, invariant to conversation length.

Code Examples

Configuring the 256-Token Window Explicitly

from needle import Needle
from needle.model.architecture import TransformerConfig

cfg = TransformerConfig(kv_window=256)   # force 256-token sliding window

agent = Needle(tools='[]', config=cfg)   # empty tool set for testing

Using Tools with KV Sink Protection

tools_json = """
[
  {"name": "set_lights",
   "description": "Turn lights on/off or dim them.",
   "parameters": {"type": "object",
                  "properties": {"room": {"type": "string"},
                                 "state": {"type": "string", "enum": ["on","off"]},
                                 "brightness": {"type": "integer"}}}}
]
"""

agent = Needle(tools=tools_json, config=cfg)

# Tool tokens now persist in KV cache regardless of window sliding

Inspecting the Attention Mask Internals

import jax.numpy as jnp
from needle.model.decode import make_causal_packing_mask

# Simulate: 3 tool tokens (segment 1) + 5 user tokens (segment 2)

seg_ids = jnp.array([[1, 1, 1, 2, 2, 2, 2, 2]])
mask = make_causal_packing_mask(
    seg_ids, 
    prefix=(seg_ids == 1),   # tool positions are sinks

    window=256
)

print(mask.shape)    # (1, 1, 8, 8) — single-head attention matrix

print(mask[0, 0])

# Output: tool rows are all True; user rows True only for last 256 positions

The printed mask shows:

• Rows 0-2 (tool tokens): all columns True — full attention allowed to all positions
• Rows 3-7 (user tokens): only recent positions True — sliding window enforced

Key Implementation Files

File	Purpose
needle/model/architecture.py	kv_budget_window(), effective_kv_window(), KV-budget constants
needle/model/decode.py	make_causal_packing_mask() — sliding window + sink logic
needle/model/tokenizer.py	TOOLS_START, TOOLS_END delimiters
needle/agent/tools.py	Tool JSON parsing and prompt injection

Summary

• 256-token sliding window — Derived from 11 MiB KV budget in kv_budget_window(), configurable via TransformerConfig(kv_window=N)
• Tool KV sinks — Tool tokens between <tools>/</tools> are marked in prefix and excluded from eviction via sink mask in make_causal_packing_mask()
• Constant memory — ~28 MiB total regardless of conversation turns; tool schemas always accessible
• User control — Override window size with kv_window parameter; minimum 160 tokens enforced

Frequently Asked Questions

What happens if I set kv_window larger than the budget allows?

The effective_kv_window() function applies min(budget, config.kv_window), so the budget-derived window caps the actual size. You cannot exceed the memory budget through configuration.

Can I mark other tokens as KV sinks besides tools?

Currently, the prefix parameter in make_causal_packing_mask() only receives the tool segment mask from the tokenizer. There is no public API for custom sinks, though the architecture supports arbitrary boolean masks.

Why 256 tokens specifically?

The 256-token default emerges from KV_BUDGET_BYTES // per_pos calculation with standard transformer dimensions (hidden size, layer count, head configuration) in the base TransformerConfig. Different model sizes yield different window sizes; 256 is the documented default, not a hardcoded constant.

How does this compare to other sliding window implementations?

Unlike fixed-window approaches that lose all old context, Needle's tool-sink mechanism guarantees critical structured data (tool schemas) remains available. This is essential for function-calling agents where tool definitions must be referenceable across arbitrarily long conversations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}