# How the 256-Token Sliding Window Works with Tools as KV Sinks in Needle

> Discover how Needle's 256-token sliding window and KV sinks manage conversation history efficiently, maintaining a fixed cache size of 28 MiB for optimal performance.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: internals
- Published: 2026-08-20

---

**Needle implements a fixed-memory KV cache by combining a 256-token sliding window for conversation history with permanent tool definitions pinned as KV sinks, keeping total cache size near 28 MiB regardless of chat length.**

The `cactus-compute/needle` project solves a critical inference problem: LLM KV caches grow linearly with sequence length, eventually exhausting GPU memory. Needle's architecture keeps the KV cache bounded through a dual mechanism implemented in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) and [`needle/model/decode.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/decode.py). This article explains exactly how the 256-token sliding window interacts with tool-based KV sinks to achieve constant memory usage.

## The Two-Part Memory Architecture

Needle's memory management relies on two cooperating components that operate at different stages of the inference pipeline.

### Effective KV Window: Bounding the Cache Size

The **effective KV window** determines how many recent tokens retain their key-value entries in the cache. The computation happens in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) through two functions.

First, `kv_budget_window` converts a fixed byte budget into a token limit:

```python
def kv_budget_window(config):
    head_dim = (config.attn_dim or config.d_model) // config.num_heads
    kv = config.num_kv_heads * head_dim
    d, L = config.d_model, config.num_layers
    sites = len(tuple(getattr(config, "engram_layers", (2, 15))))
    per_pos = (L * (2 * kv + 2 * (kv // KV_GROUP) * 4)
               + sites * (d + (KV_GROUP) * 4))
    # 11 MiB + 0.5 MiB budget for KV cache

    window = (KV_BUDGET_BYTES // per_pos) // KV_GROUP * KV_GROUP
    return max(KV_WINDOW_MIN, min(window, config.max_seq_len))

```

Key parameters from the source:

- `KV_BUDGET_BYTES` ≈ **11 MiB** — the hard memory ceiling for KV storage
- `KV_WINDOW_MIN` = **160** — guaranteed minimum window size
- Default configuration yields **256 tokens** (2⁶), matching the documented behavior

The effective window applies user overrides:

```python
def effective_kv_window(config):
    budget = kv_budget_window(config)
    return min(budget, config.kv_window) if config.kv_window else budget

```

If `config.kv_window` is unset, the budget-derived 256-token window applies.

### KV Sinks for Tools: Pinning Critical Context

While the sliding window drops old tokens, **tool definitions must remain permanently accessible**. Needle solves this by marking tool tokens as **KV sinks** — entries that bypass eviction.

Tool delimiters are defined in [`needle/model/tokenizer.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/tokenizer.py):

```python
TOOLS_START = "<tools>"
TOOLS_END = "</tools>"

```

The masking logic in [`needle/model/decode.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/decode.py) implements the sink mechanism:

```python
def make_causal_packing_mask(seg_ids, prefix=None, window=0):
    T = seg_ids.shape[1]
    causal = jnp.tril(jnp.ones((T, T), dtype=jnp.bool_))
    block  = (seg_ids[:, :, None] == seg_ids[:, None, :]) & (seg_ids[:, :, None] > 0)
    mask   = block & causal[None, :, :]

    if window:
        pos   = jnp.arange(T)
        recent = (pos[:, None] - pos[None, :]) < window          # ← sliding window

        sink   = (jnp.zeros_like(seg_ids, dtype=jnp.bool_) if prefix is None
                  else prefix > 0)                               # ← KV sink

        mask = mask & (recent[None, :, :] | sink[:, None, :])
    return mask[:, None, :, :]

```

The critical logic: `mask & (recent | sink)`. A token is attended to if it is **either** within the recent window **or** designated as a sink. The `prefix` parameter receives a boolean mask where tool tokens are marked `True`.

## End-to-End Execution Flow

Understanding the sliding window with tools requires tracing the complete inference path:

1. **Tool injection** — `Needle(tools=tools_json)` parses tools in [`needle/agent/tools.py`](https://github.com/cactus-compute/needle/blob/main/needle/agent/tools.py) and wraps them with `<tools>` … `</tools>` delimiters
2. **Tokenization** — [`needle/model/tokenizer.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/tokenizer.py) assigns segment IDs: tool tokens receive ID 1, conversation tokens receive ID 2
3. **Mask construction** — Each generation step calls `make_causal_packing_mask` with `prefix=(seg_ids == 1)` to identify sinks and `window=effective_kv_window(config)`
4. **Cache management** — JAX arrays store KV pairs only for positions passing the mask filter

Memory footprint remains **≈ 28 MiB** total: ~11 MiB KV budget plus model weights, invariant to conversation length.

## Code Examples

### Configuring the 256-Token Window Explicitly

```python
from needle import Needle
from needle.model.architecture import TransformerConfig

cfg = TransformerConfig(kv_window=256)   # force 256-token sliding window

agent = Needle(tools='[]', config=cfg)   # empty tool set for testing

```

### Using Tools with KV Sink Protection

```python
tools_json = """
[
  {"name": "set_lights",
   "description": "Turn lights on/off or dim them.",
   "parameters": {"type": "object",
                  "properties": {"room": {"type": "string"},
                                 "state": {"type": "string", "enum": ["on","off"]},
                                 "brightness": {"type": "integer"}}}}
]
"""

agent = Needle(tools=tools_json, config=cfg)

# Tool tokens now persist in KV cache regardless of window sliding

```

### Inspecting the Attention Mask Internals

```python
import jax.numpy as jnp
from needle.model.decode import make_causal_packing_mask

# Simulate: 3 tool tokens (segment 1) + 5 user tokens (segment 2)

seg_ids = jnp.array([[1, 1, 1, 2, 2, 2, 2, 2]])
mask = make_causal_packing_mask(
    seg_ids, 
    prefix=(seg_ids == 1),   # tool positions are sinks

    window=256
)

print(mask.shape)    # (1, 1, 8, 8) — single-head attention matrix

print(mask[0, 0])

# Output: tool rows are all True; user rows True only for last 256 positions

```

The printed mask shows:
- Rows 0-2 (tool tokens): all columns `True` — full attention allowed to all positions
- Rows 3-7 (user tokens): only recent positions `True` — sliding window enforced

## Key Implementation Files

| File | Purpose |
|------|---------|
| [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) | `kv_budget_window()`, `effective_kv_window()`, KV-budget constants |
| [`needle/model/decode.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/decode.py) | `make_causal_packing_mask()` — sliding window + sink logic |
| [`needle/model/tokenizer.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/tokenizer.py) | `TOOLS_START`, `TOOLS_END` delimiters |
| [`needle/agent/tools.py`](https://github.com/cactus-compute/needle/blob/main/needle/agent/tools.py) | Tool JSON parsing and prompt injection |

## Summary

- **256-token sliding window** — Derived from 11 MiB KV budget in `kv_budget_window()`, configurable via `TransformerConfig(kv_window=N)`
- **Tool KV sinks** — Tool tokens between `<tools>`/`</tools>` are marked in `prefix` and excluded from eviction via `sink` mask in `make_causal_packing_mask()`
- **Constant memory** — ~28 MiB total regardless of conversation turns; tool schemas always accessible
- **User control** — Override window size with `kv_window` parameter; minimum 160 tokens enforced

## Frequently Asked Questions

### What happens if I set kv_window larger than the budget allows?

The `effective_kv_window()` function applies `min(budget, config.kv_window)`, so the budget-derived window caps the actual size. You cannot exceed the memory budget through configuration.

### Can I mark other tokens as KV sinks besides tools?

Currently, the `prefix` parameter in `make_causal_packing_mask()` only receives the tool segment mask from the tokenizer. There is no public API for custom sinks, though the architecture supports arbitrary boolean masks.

### Why 256 tokens specifically?

The 256-token default emerges from `KV_BUDGET_BYTES // per_pos` calculation with standard transformer dimensions (hidden size, layer count, head configuration) in the base `TransformerConfig`. Different model sizes yield different window sizes; 256 is the documented default, not a hardcoded constant.

### How does this compare to other sliding window implementations?

Unlike fixed-window approaches that lose all old context, Needle's tool-sink mechanism guarantees critical structured data (tool schemas) remains available. This is essential for function-calling agents where tool definitions must be referenceable across arbitrarily long conversations.