How Needle 2 Implements Bounded Memory: A 256‑Token Sliding Window Design

Needle 2 maintains a constant RAM footprint of approximately 28 MiB by using a fixed 256‑token sliding window with tool definitions pinned as KV sinks, allowing unlimited conversation length without memory growth.

The cactus-compute/needle repository introduces a novel solution to the memory explosion problem in transformer-based conversational AI. Rather than letting the key-value cache grow with each turn, Needle 2's Simple Attention Network (SAN) enforces hard limits on context retention while preserving immediate access to tool definitions. This bounded memory design makes full-session tool calling viable on memory-constrained edge devices.

The 256‑Token Sliding Window Mechanism

At the core of Needle 2's memory strategy is a strictly enforced 256‑token window that governs all conversation history. This constraint is implemented in needle/model/run.py, where the inference engine maintains the sliding buffer during multi-turn dialogue.

When a conversation exceeds 256 tokens, the oldest user tokens are automatically dropped from the KV cache. New tokens take their place without allocating additional memory. Because the window size never changes, total RAM consumption stays near 28 MiB regardless of interaction length—enabling hour-long sessions on microcontrollers and other tiny devices.

The window operates at token granularity, not turn granularity, allowing fine-grained memory management within single responses.

KV Sinks: Pinning Tools in Memory

While user tokens slide out of the cache, tool definitions remain permanently resident. Needle 2 treats declared tools as key-value (KV) sinks—their embeddings are pinned and never evicted.

This design choice serves a critical architectural purpose:

• Instant availability: Tools can be invoked at any moment without reloading embeddings
• Bounded overhead: Tool KV storage counts against the fixed memory budget during initialization, not during conversation
• Deterministic latency: No cache misses or on-demand computation when tools are called

The KV sink implementation resides in the SAN's engram memory layer within needle/model/architecture.py, where hashed n-gram tables organize both sliding and pinned entries.

Simple Attention Network Architecture

The bounded memory guarantee emerges from Needle 2's Simple Attention Network (SAN), defined in needle/model/architecture.py. This architecture replaces conventional growing attention with two key mechanisms:

1. Engram key-value memory: A fixed-capacity storage system limited to the most recent 256 tokens
2. Hashed n-gram tables: Efficient (kₜ, vₜ) row retrieval that operates within the constrained window

The SAN's byte-level grammar enforces these structural limits at the model level, not merely as software policy. This hardware-aware design eliminates the O(n²) memory scaling that plagues standard transformers.

Practical Implementation

Basic Usage with Default Window

By default, needle.Needle() initializes with the 256-token sliding window and automatic KV sink management for tools.

import needle

# Tool definition is automatically pinned as KV sink

@needle.tool
def get_weather(city: str):
    """Get the current weather for a city."""
    return {"city": city, "temp_c": 27, "sky": "clear"}

# Agent starts with bounded memory configuration

agent = needle.Needle(tools=[get_weather])

# Multiple turns—memory stays near 28 MiB

response1 = agent.run("What's the weather in Paris?")
response2 = agent.run("And in Tokyo?")
response3 = agent.run("Now tell me the temperature in New York.")
print(response3["results"])

Explicit Window Configuration

For specialized deployments, the window size can be adjusted via the max_len parameter documented in doc/apis.md. The same sliding-window mechanics apply regardless of the specific limit.

# Explicit 256-token window (matches default)

agent = needle.Needle(tools=[get_weather], max_len=256)

# Bounded memory behavior persists across all interaction lengths

agent.run("... extended multi-turn dialogue ...")

Key Source Files

File	Responsibility
needle/model/architecture.py	Simple Attention Network implementation; hashed n-gram tables and engram KV memory enforcing the sliding window
needle/model/run.py	Inference loop; maintains 256-token sliding window and manages KV sink pinning for tools
README.md	Bounded memory guarantee documentation (lines 13–14, 21–26)
doc/apis.md	Public API reference for max_len and related configuration parameters

Summary

• Fixed 256-token sliding window caps memory usage regardless of conversation duration
• Tool KV sinks remain pinned and instantly accessible while user context slides
• ~28 MiB RAM footprint enables deployment on microcontrollers and edge devices
• SAN architecture replaces unbounded attention with hashed n-gram engram memory
• Implementation spans needle/model/architecture.py and needle/model/run.py

Frequently Asked Questions

How does Needle 2 prevent memory growth during long conversations?

Needle 2 drops the oldest tokens from its KV cache once the 256-token limit is reached. New tokens overwrite freed positions rather than expanding storage. This sliding-window mechanism is enforced in needle/model/run.py and guarantees constant memory usage.

Why are tools treated differently from user tokens?

Tools are pinned as KV sinks because they must remain instantly invocable throughout the session. User context, by contrast, can degrade gracefully—older turns become less relevant. This asymmetry preserves functionality while enabling aggressive memory limits.

Can the sliding window size be changed from 256 tokens?

Yes. The max_len parameter in needle.Needle() accepts custom window sizes. However, the default 256 tokens represents the validated configuration for the 28 MiB memory target. Larger windows increase RAM consumption proportionally.

What happens when a conversation exceeds the token window?

The model continues operating on the most recent 256 tokens. Earlier context is irretrievably discarded—there is no summarization or compression fallback. This hard truncation is the trade-off for deterministic, bounded memory behavior.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}