# What Is the Sliding Window Size for Bounded-Memory Decode in Needle?

> Discover Needle's 256-token sliding window for bounded-memory decode. Keep your KV cache under 28 MiB, no matter how long your conversation gets.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-09-12

---

**Needle uses a fixed 256-token sliding window for bounded-memory decode, ensuring the KV cache never exceeds approximately 28 MiB regardless of conversation length.**

The `cactus-compute/needle` repository implements a memory-constrained transformer architecture designed for long-running conversations without GPU memory explosion. Understanding the precise sliding window size—and how it interacts with permanent tool storage—is critical for predicting memory usage and context retention in production deployments.

## Default Sliding Window Configuration

Needle’s bounded-memory mode enforces a strict **256-token sliding window** that limits attention to recent context. According to the project documentation in [`README.md`](https://github.com/cactus-compute/needle/blob/main/README.md) at line 13, the system employs "a 256-token sliding window with the tools pinned as KV sinks" to maintain constant memory footprint.

When you initialize a standard `TransformerConfig` without explicit overrides, the architecture automatically applies this 256-token limit. The window acts as a rolling buffer: as new tokens generate beyond the 256-token threshold, the oldest tokens drop out of the KV cache, while the model continues attending only to tokens within that recent history.

## How the Window Size Is Calculated

The actual window size derives from the `kv_window` parameter in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py). If you do not explicitly set this value, Needle computes an effective window using the `effective_kv_window` helper function, which defaults to 256 tokens based on the standard KV budget configuration.

```python
from needle.model.architecture import TransformerConfig, effective_kv_window

# Default configuration (no explicit kv_window)

cfg = TransformerConfig()
window = effective_kv_window(cfg)

print(f"Effective sliding-window size: {window} tokens")   # → 256

```

This design allows the `TransformerConfig` class to maintain flexible configuration while guaranteeing that bounded-memory deployments automatically fall back to the safe 256-token default. The `effective_kv_window` function encapsulates the logic that maps abstract memory budgets to concrete token counts.

## Implementation in the Decode Loop

During inference, [`needle/model/decode.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/decode.py) consumes the `kv_window` value to construct attention masks that enforce the sliding constraint. The decode loop masks out positions beyond the 256-token boundary, ensuring the model physically cannot attend to tokens outside the active window, regardless of how long the conversation history grows.

### KV Sinks for Tool Persistence

While conversation tokens rotate through the 256-token window, tool definitions remain exempt. Needle pins tool representations as **KV sinks**—permanent entries in the KV cache that survive the sliding window eviction. This architectural choice guarantees that function calling capabilities persist indefinitely without consuming the precious 256-token budget reserved for conversational context.

## Memory Implications of the 256-Token Limit

The 256-token sliding window directly bounds the working memory to approximately **28 MiB**, as documented in the repository. This constant-memory guarantee holds because the KV cache dimensions remain fixed at `(batch_size, num_heads, 256, head_dim)` rather than growing with sequence length. The 28 MiB target accommodates the 256 cached positions plus the permanent KV sinks for tools, enabling deployment on memory-constrained edge devices without gradient-based memory management.

## Summary

- **Needle’s sliding window size is 256 tokens** for bounded-memory decode, as defined in `TransformerConfig` and documented in [`README.md`](https://github.com/cactus-compute/needle/blob/main/README.md).
- The `effective_kv_window` function in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) calculates this value when no explicit `kv_window` is provided.
- [`needle/model/decode.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/decode.py) enforces the limit via attention masking during inference.
- **KV sinks** store tools permanently outside the sliding window, preserving function-calling capability without consuming the 256-token budget.
- This architecture guarantees a constant **28 MiB memory footprint** regardless of conversation length.

## Frequently Asked Questions

### What happens when the conversation exceeds 256 tokens?

When the token count surpasses the 256-token window, the oldest conversation tokens are evicted from the KV cache. The model loses the ability to attend to those specific tokens, though tool definitions remain accessible via the permanent KV sinks. This creates a "forgetting" effect for distant conversation history while maintaining functional capabilities.

### How do KV sinks differ from the sliding window?

KV sinks are permanent cache entries that bypass the sliding window eviction policy entirely. While conversation tokens rotate through the 256-token bounded buffer, pinned KV sinks for tools remain in GPU memory indefinitely. As implemented in [`needle/model/decode.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/decode.py), the attention mechanism masks the sliding window portion but always retains access to sink indices.

### Can I customize the sliding window size?

Yes, you can override the default by explicitly setting `kv_window` in your `TransformerConfig` initialization. However, increasing the value beyond 256 tokens violates the bounded-memory guarantee and will raise the constant 28 MiB memory target proportionally. The `effective_kv_window` function will respect custom values you provide, though the standard configuration optimizes for the 256-token constraint.

### Why does Needle use a 28 MiB memory target?

The 28 MiB target represents the precise GPU memory required to store the 256-token KV cache along with permanent tool sinks for the default model configuration. This figure accounts for the tensor dimensions `(batch, heads, 256, head_dim)` at standard precision, ensuring the system runs on embedded devices and edge hardware with severe memory constraints.