How Needle 2 Manages Memory with a 256-Token Sliding Window

Needle 2 caps transformer memory usage by implementing a fixed-size 256-token sliding-window KV-cache that overwrites stale entries via modulo indexing, ensuring constant O(kv_window × num_layers × head_dim) complexity regardless of generation length.

Needle 2 (the current version of the cactus-compute/needle repository) eliminates unbounded memory growth during inference through aggressive KV-cache quantization and a configurable sliding-window attention mechanism. By encoding the window size directly into the exported model binary, the runtime allocates a fixed circular buffer that never grows beyond the specified token limit—commonly set to 256 (2⁸) tokens for balancing context retention with memory efficiency.

KV-Cache Header Configuration in export.py

The sliding-window constraint begins at model export time. In needle/model/export.py, the binary header reserves a kv_window field that dictates how many past tokens the runtime retains in the KV-cache. The header structure defined at lines 20–30 serializes this value as an unsigned 32-bit integer:

# export.py – header definition

# … line 20‑30 …

HEADER  (30 u32-sized fields, rope_theta an f32, everything else u32):
    tag, num_tensors, codebook_len, kv_window, kv_bits,
    vocab, d_model, num_heads, num_kv_heads, num_layers, head_dim, max_seq_len,
    ...

When the model loads, the read_export() function parses this header to extract the window size. At lines 1000–1002, the code unpacks the raw bytes and assigns kv_window from the header tuple:

# export.py – read_export()

hdr = struct.unpack_from(_HDR_FMT, raw, 0)
tag, num_tensors, cb_n, kv_window, kv_bits = hdr[:5]

This value persists throughout the model’s lifecycle, locking the memory footprint at initialization.

Effective Window Size Calculation

Before export, the effective window size is computed in needle/model/architecture.py via the effective_kv_window(config) helper. This function evaluates the model’s sliding_window attribute—if explicitly set to 256 (2⁸), it returns that value; otherwise, it defaults to 64 tokens (2⁶).

The power-of-two constraint enables efficient bitwise operations during indexing while keeping the KV-cache compact. Developers can configure larger windows by adjusting the architecture configuration before exporting to .cact format.

Circular Buffer Implementation in run.py

Inside the inference engine (needle/model/run.py), the KV-cache is allocated once as a contiguous buffer sized exactly to kv_window slots. As the model generates new tokens, the cache index advances via modulo arithmetic:

idx = position % kv_window

This ring-buffer approach automatically overwrites the oldest key/value pairs once the 256-token limit is reached. Consequently, the memory cost remains fixed at O(kv_window × num_layers × head_dim) even when processing arbitrarily long sequences.

Practical Configuration Examples

To inspect the sliding-window behavior of an exported model:

# Inspect KV-window size from an exported model

from needle.model.export import read_export

meta, tensors = read_export("model.cact")
print(f"KV-window size: {meta['kv_window']} tokens")  # Output: 256

To configure a 256-token window before export:

# Force 256-token (2^8) sliding window

from needle.model.architecture import TransformerConfig, effective_kv_window

cfg = TransformerConfig.from_dict({"sliding_window": 256})
kv_window = effective_kv_window(cfg)  # Returns 256

The underlying C++ single-mega-kernel engine implements the update logic as follows:

# Simulated ring-buffer update mechanism

def update_kv_cache(cache, new_key, new_value, pos, kv_window):
    idx = pos % kv_window           # Circular index calculation

    cache[:, idx, :] = new_key      # Overwrite stale key

    cache[:, idx, :] = new_value    # Overwrite stale value

    return pos + 1                  # Advance position

Summary

• Binary Header Encoding: The kv_window field in needle/model/export.py (lines 20–30) persists the window size into the .cact checkpoint format, read at lines 1000–1002 during model loading.
• Fixed Allocation: needle/model/run.py allocates the KV-cache exactly once based on this header value, eliminating dynamic memory growth.
• Constant Complexity: Memory usage scales as O(kv_window × num_layers × head_dim), independent of generation length.
• Configurable Trade-off: While defaulting to 64 tokens (2⁶), the system supports 256-token (2⁸) or larger windows via the sliding_window configuration parameter in needle/model/architecture.py.

Frequently Asked Questions

What is the default KV-window size in Needle 2?

The default configuration uses a 64-token (2⁶) window to minimize memory footprint for edge deployments. However, the architecture supports arbitrary power-of-two sizes including 128, 256, and 512 tokens, configured via the sliding_window attribute in needle/model/architecture.py before model export.

How does the sliding window prevent out-of-memory errors during long generations?

By implementing a circular buffer in needle/model/run.py that overwrites entries modulo kv_window, the system maintains constant memory usage regardless of sequence length. Once the 256-token limit is reached, each new token evicts the oldest cached keys and values rather than appending to an ever-growing tensor.

Can I change the KV-window size without re-exporting the model?

No. The kv_window value is baked into the binary header at export time (defined in needle/model/export.py). To use a different window size, you must re-export the checkpoint from a configuration with the desired sliding_window value, as the runtime allocates the exact buffer size specified in the header during read_export().

Why does Needle 2 use power-of-two window sizes?

Power-of-two sizes (64, 256, 512, etc.) enable efficient bitwise modulo operations and alignment-friendly memory layouts in the C++ inference kernel. This constraint, enforced through effective_kv_window() in needle/model/architecture.py, simplifies the ring-buffer indexing logic while maintaining cache-friendly access patterns across layers.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}