# Bounded Memory Implementation with 256-Token Sliding Window and KV Sinks in Needle 2

> Implement bounded memory in Needle 2 with a 256-token sliding window and KV sinks. Control KV cache memory growth and preserve essential context effectively.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-08-25

---

**Needle 2 caps KV cache memory usage by applying a fixed-size sliding-window attention mask (commonly 256 tokens) combined with optional KV sinks that force specific tokens to persist outside the window, limiting memory growth while preserving essential context.**

Needle 2 introduces a bounded-memory attention mechanism designed for resource-constrained environments. By implementing a configurable sliding window over the key-value (KV) cache and supplementing it with persistent "KV sinks," the architecture ensures strict memory bounds without sacrificing critical context. This article examines the implementation details found in the `cactus-compute/needle` repository, specifically how the 256-token window is calculated, stored, and enforced at runtime.

## KV Window Budgeting and Storage

The sliding window size is determined at two stages: first during model configuration via budgeting functions, and second during export via the CACT header format.

### Calculating the Budgeted Window

The model derives a theoretical KV cache budget based on architectural parameters. The `kv_budget_window` function computes this from the hidden size, number of layers, KV-heads, and engram sites. This yields a "budgeted" window size representing the maximum sustainable KV cache length for the target hardware.

### Resolving the Effective Window

The final window width is resolved by `effective_kv_window`, implemented in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) at lines 14-16. This function selects the actual window size used during training and inference, which can be overridden via the `kv_window` configuration field. When specified, this value determines the exact number of recent tokens retained—in practice often set to 256 tokens (2⁸), though the source demonstrates the logic with 64 tokens (2⁶).

### Persisting Window Size in CACT Headers

To ensure runtime consistency, the effective window is serialized into the model checkpoint. The `write_export` function in [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) (lines 27-30) embeds the `kv_window` value directly into the CACT header. During loading, `read_export` restores this value, exposing it via the configuration object so the inference engine knows the exact window width the model was trained with.

## Runtime Sliding Window Masking

During forward passes, Needle applies the window constraint through attention masking rather than physical cache eviction. This occurs in the `hidden_cells` function within [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py).

### Implementing the Window Constraint

The mask construction combines causal, padding, and window masks. When the `window` argument is provided, the implementation limits attention to the last *W* positions:

```python
mask = (make_causal_mask(tokens.shape[1])
        & make_padding_mask(tokens, cfg.pad_token_id))
if window:
    pos = jnp.arange(tokens.shape[1])
    recent = ((pos[:, None] - pos[None, :]) < window)[None, None, :, :]
    keep = recent if sink is None else (recent | sink[:, None, None, :])
    mask = mask & keep

```

This logic (found at lines 44-49 of [`architecture.py`](https://github.com/cactus-compute/needle/blob/main/architecture.py)) creates a boolean mask where positions outside the sliding window receive a negative infinity attention score. For a 256-token window, passing `window=256` ensures only the most recent 256 positions participate in attention, bounding memory complexity to O(256×d) per head rather than O(n×d).

## KV Sinks for Persistent Context

While the sliding window discards old tokens to save memory, certain tokens—such as document prefixes or tool headers—must persist across all windows. Needle implements **KV sinks** to mark these positions for retention.

### Defining Sink Masks

Sinks are optional boolean arrays passed as the `sink` argument to `hidden_cells`. When provided, the mask construction modifies the `keep` tensor:

```python
keep = recent if sink is None else (recent | sink[:, None, None, :])

```

This OR operation ensures sink positions remain attended to even when they fall outside the recent window.

### Integration with Packing Masks

For batched inference with packing, the `make_causal_packing_mask` function (lines 24-30 of [`architecture.py`](https://github.com/cactus-compute/needle/blob/main/architecture.py)) handles sinks via segment IDs:

```python
if window:
    pos = jnp.arange(T)
    recent = (pos[:, None] - pos[None, :]) < window
    sink = (jnp.zeros_like(seg_ids, dtype=jnp.bool_) if prefix is None
            else prefix > 0)
    mask = mask & (recent[None, :, :] | sink[:, None, :])

```

Here, the `prefix` flag indicates which positions should serve as sinks. This allows the model to maintain attention to static context while processing new tokens within the rolling buffer.

## Practical Code Examples

The following patterns demonstrate how to leverage the bounded-memory features in Needle 2.

### Loading a Checkpoint and Discovering its KV Window

Retrieve the baked-in window size from the CACT header to know the model's training configuration:

```python
from needle.model.export import read_export

metadata, tensors = read_export("my_model.cact")
print("KV window baked into the model:", metadata["kv_window"])  # e.g., 256

```

### Running Inference with a 256-Token Sliding Window

Apply the window constraint during the forward pass to limit memory usage:

```python
from needle.model.run import load_checkpoint, build_model
import jax.numpy as jnp

params, cfg, _ = load_checkpoint("my_model.cact", return_run=True)
model = build_model(cfg)          
model = model.apply(params)       

# tokens: (batch, seq_len) int32

tokens = jnp.array([[...]])  # Your input tokens

# Enforce a 256-token KV window (2^8)

logits = model(tokens, window=256)

```

### Preserving KV for Special Tokens with Sinks

Force specific positions (e.g., a tool-use marker) to remain in cache indefinitely:

```python

# Create a sink mask preserving the first token (e.g., a special header)

sink_mask = jnp.zeros_like(tokens, dtype=bool)
sink_mask = sink_mask.at[:, 0].set(True)   # Keep first token forever

# Apply both the 256-token window and the sink mask

logits = model(tokens, window=256, sink=sink_mask)

```

## Summary

- **Budgeted Calculation**: `kv_budget_window` and `effective_kv_window` in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) determine the optimal window size based on architecture and configuration.
- **Persistent Storage**: The `kv_window` value is embedded in the CACT header via `write_export` in [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py), ensuring the runtime uses training-consistent bounds.
- **Mask-Based Enforcement**: The `hidden_cells` function applies the window via tensor masking (`recent < window`) rather than cache deletion, supporting dynamic window sizes including 256 tokens.
- **KV Sinks**: Optional boolean masks allow specific tokens to bypass the sliding window, enabling long-range attention to prefixes or special markers without breaking memory bounds.
- **Edge Optimization**: This bounded-memory design enables deployment on mobile and edge GPUs where unbounded KV caches would cause OOM errors.

## Frequently Asked Questions

### How does the 2S⁶ notation relate to the 256-token window mentioned in the question?

The **2S⁶** notation indicates scientific notation for powers of two used in the Needle 2 codebase, where 2S⁶ equals 2⁶ or 64 tokens. A 256-token window follows the same pattern as 2⁸ (2S⁸). The implementation is size-agnostic; the `window` parameter in `hidden_cells` accepts any integer value, allowing models to be configured for 64, 256, or other power-of-two boundaries depending on memory constraints.

### What happens to tokens that slide outside the 256-token window?

Tokens outside the recent 256 positions are masked out in the attention computation within `hidden_cells`. Their key-value pairs remain physically allocated in device memory but are effectively ignored by the attention scores due to the boolean mask (`recent` tensor). If no KV sinks are defined, these tokens no longer influence the model's output. With sinks, specified tokens remain attended to via the OR-based mask combination even after falling outside the window.

### Can KV sinks be used for dynamic content or only static prefixes?

KV sinks are controlled by boolean masks that can be constructed dynamically at runtime. While the `make_causal_packing_mask` example uses `prefix > 0` to identify static document headers, you can construct arbitrary `sink` tensors in Python. For example, you could mark positions containing specific entity mentions or tool calls based on runtime logic, allowing the model to maintain "sticky" attention to dynamically identified critical tokens across the 256-token rolling window.

### Where is the sliding window logic implemented if I need to modify the masking behavior?

The core masking logic resides in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py). Specifically, the `hidden_cells` function (lines 44-49) handles the standard causal window masking, while `make_causal_packing_mask` (lines 24-30) handles packed sequence scenarios. Both functions accept the `window` and `sink` arguments. To modify behavior—such as implementing a strided window or attention sink pooling—you would edit the mask construction logic in these functions before the mask is applied to attention scores.