How Needle's Bounded Memory Design Stays at 28MB Regardless of Conversation Length

Needle maintains a constant ~28 MB RAM footprint by capping the KV cache at 256 tokens with a fixed 11 MiB budget and using pinned engram tables, ensuring memory never grows with conversation length.

The cactus-compute/needle repository implements a bounded memory architecture that defies the typical scaling behavior of large language models. Unlike conventional transformers whose memory usage grows linearly with context length, Needle pins its RAM usage at approximately 28 MB through strict enforcement of fixed-size buffers and sliding-window attention.

The Three Pillars of Needle's Bounded Memory Architecture

Fixed-Size KV Cache with a 256-Token Sliding Window

The attention mechanism in Needle does not accumulate key/value pairs indefinitely. Instead, it maintains a 256-token sliding window that overwrites older KV slots as the conversation progresses. This design choice, documented in the project README, ensures the cache never exceeds its predetermined allocation regardless of how many turns the dialogue spans.

Hard KV Budget Allocation (≈ 11 MiB)

In needle/model/architecture.py, the constant KV_BUDGET_BYTES establishes a rigid memory ceiling for the KV cache. The function kv_budget_window (lines 998–1002) calculates the maximum safe window length that fits within this ≈ 11 MiB constraint after accounting for the model's head dimensions, layer count, and architectural overhead. This computation guarantees that the KV cache cannot accidentally expand beyond its budget.

Pinned Engram Tables for Tool Information

Tool-related key/value pairs bypass the sliding window through dedicated engram tables. As implemented in architecture.py (lines 1414–1419), these engram KV stores are instantiated once and never resized, acting as permanent "sinks" that retain tool definitions without consuming additional memory as the conversation grows.

Technical Implementation: Window Calculation and Masking

The bounded memory guarantee is enforced through a three-stage pipeline:

1. Budget-Driven Window Calculation – kv_budget_window(config) determines the exact token capacity (256) that respects the KV_BUDGET_BYTES limit while considering layer count, head count, and engram slots.

2. Causal Mask Application – make_causal_mask and make_causal_packing_mask generate attention masks that strictly adhere to the computed window, ensuring the model only attends to the most recent 256 tokens.

3. Engram Integration – Engram KV tensors (self.engrams) are allocated during initialization and remain pinned throughout the session, providing persistent tool context without dynamic memory allocation.

Verifying Constant Memory Usage in Practice

You can verify Needle's fixed memory footprint programmatically. The following example runs 100 conversation turns while monitoring RSS memory:

import needle
import psutil
import os

@needle.tool
def echo(txt: str):
    """Return the input text unchanged."""
    return {"result": txt}

agent = needle.Needle(tools=[echo])

def mem_mb():
    proc = psutil.Process(os.getpid())
    return proc.memory_info().rss / 1024**2

for i in range(100):
    reply = agent.run(f"Turn {i}: repeat this phrase.")
    
print(f"Final RAM usage: {mem_mb():.1f} MiB")

Even after processing 100 turns, the process consistently reports approximately 28 MiB, composed of roughly 14 MiB for the model binary, 11 MiB for the fixed KV budget, and minor overhead.

Summary

• Needle's memory usage remains invariant to conversation length through a fixed 256-token sliding window that overwrites stale KV cache entries.
• The KV_BUDGET_BYTES constant and kv_budget_window function in needle/model/architecture.py enforce a hard 11 MiB ceiling on attention cache memory.
• Engram tables provide persistent tool storage without dynamic growth, acting as pinned KV sinks defined in lines 1414–1419 of the architecture module.
• Total RAM consumption stays near 28 MiB (14 MiB model + 11 MiB KV cache + overhead) regardless of dialogue duration.

Frequently Asked Questions

Why does Needle use a 256-token sliding window instead of full context?

The 256-token limit is derived from the kv_budget_window calculation in needle/model/architecture.py, which maximizes context length while fitting within the strict KV_BUDGET_BYTES allocation of approximately 11 MiB. This trade-off prioritizes predictable embedded-systems memory usage over infinite context retention.

How do engram tables prevent memory growth when using tools?

Engram KV tables are instantiated once during model initialization (self.engrams = [...]) and remain permanently pinned in memory. Unlike dynamic KV caches that grow with sequence length, these tables have fixed dimensions and act as static "sinks" for tool-related information, ensuring tool usage does not expand the memory footprint.

Can the 28 MB memory limit be increased for longer context windows?

The memory boundary is configurable through the KV_BUDGET_BYTES constant in needle/model/architecture.py, though modifying this requires recalculating the window size via kv_budget_window to ensure the cache fits within the new budget. Increasing the budget would raise RAM usage above 28 MB proportionally.

Does the sliding window cause the model to forget earlier conversation details?

Yes, the 256-token sliding window inherently limits the model's direct attention span to recent tokens. However, critical information from earlier turns can be preserved through the engram mechanism, which explicitly stores tool definitions and other essential data outside the sliding window in fixed-size tables.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}