How Needle 2 Tool Pinning Maintains 28 MB Memory Usage

Needle 2 maintains a constant memory footprint of approximately 28 MB throughout long-running conversations by pinning tools as key-value (KV) sinks within a fixed 256-token sliding window, ensuring memory consumption never increases with conversation length.

The cactus-compute/needle repository implements a novel inference architecture that eliminates memory bloat during LLM tool use. By treating tool definitions as persistent KV cache entries rather than continuously reloading their definitions, Needle 2 achieves bounded memory usage that remains stable regardless of conversation length.

The 28 MB Memory Architecture

Needle 2 stores its entire inference engine in a single 14 MB binary (libneedle.so) downloaded from Hugging Face. During operation, the system allocates roughly 14 MB additional RAM for the KV cache and sliding window, resulting in a total footprint of approximately 28 MB that does not grow as more tokens are processed.

This memory ceiling persists because the engine never appends tool definitions to the growing conversation history. Instead, tools are pinned once as fixed-size KV slots, allowing the system to discard old conversation tokens while retaining tool accessibility.

How Tool Pinning Works

The pinning mechanism operates through three distinct phases implemented in the native inference engine:

• KV Slot Creation: When a tool first appears in the conversation, the engine allocates fixed-size KV slots for its schema and description.
• Slot Persistence: These slots are pinned within the 256-token window, marking them as non-discardable regardless of their age in the conversation sequence.
• Memory Region Reuse: New conversation tokens overwrite the oldest unpinned tokens in the window, while pinned tool slots remain constant.

Because the retrieval head selects at most five tools per turn, and each tool occupies fixed-size KV storage, the memory consumption remains deterministic according to the README documentation.

The 256-Token Sliding Window

The engine maintains a 256-token sliding window that serves as the working memory for the inference process. This window contains recent conversation text and the pinned tool KV sinks.

When the conversation extends beyond 256 tokens, the engine discards only the oldest unpinned tokens. The pinned tool definitions remain resident in the KV cache, eliminating the need to reload or re-encode tool schemas on subsequent turns. This design is explicitly described as "bounded memory" in the project documentation.

Native Engine Implementation

The core pinning logic resides in the native inference engine (libneedle.so) rather than the Python wrapper. The Python layer in needle/__init__.py simply forwards user-provided tool schemas to the engine via the public API.

The engine enforces fixed-size KV allocations for pinned tools, preventing the memory fragmentation and growth typical of dynamic context windows. As documented in doc/apis.md, the engine manages tool selection and pinning during inference without requiring manual memory management from the Python runtime.

Python API and Practical Usage

Tool registration occurs through the @needle.tool decorator defined in needle/agent/tools.py. When initializing a Needle agent with tools, the Python wrapper transmits the schemas to the native engine, which immediately pins them in the KV cache.

import needle

# Declare tools - only schemas are sent to the engine

@needle.tool
def set_lights(room: str, brightness: int):
    """Set the lights in a room to a specific brightness."""
    return {"room": room, "brightness": brightness}

@needle.tool
def get_weather(city: str):
    """Return the current weather for a city."""
    return {"city": city, "temp_c": 20, "sky": "clear"}

# Initialize agent - engine pins these tools automatically

agent = needle.Needle(tools=[set_lights, get_weather])

# Multi-turn conversation memory stays bounded at ~28 MB

print(agent.run("Dim the kitchen lights to 10%")["results"])
print(agent.run("What's the weather in Nairobi?")["results"])

Even after dozens of additional calls to agent.run(), process memory remains near 28 MB because the tools persist as pinned KV sinks while old conversation tokens rotate out of the sliding window.

Summary

• Constant Memory: Needle 2 maintains approximately 28 MB RAM usage regardless of conversation length by combining a 14 MB binary with a bounded KV cache.
• Tool Pinning: Tools are stored as pinned KV sinks within the 256-token window, preventing repeated reloading of tool definitions.
• Sliding Window: Old tokens are discarded when exceeding 256 tokens, but pinned tools remain resident.
• Native Implementation: The pinning mechanism operates in libneedle.so, with needle/__init__.py and needle/agent/tools.py handling Python-side schema transmission.
• Fixed Limits: Maximum of five tools pinned per turn with fixed-size allocations, ensuring predictable memory consumption.

Frequently Asked Questions

What is the maximum memory usage for a Needle 2 conversation?

A complete Needle 2 session consumes approximately 28 MB of RAM regardless of conversation duration. This consists of the 14 MB inference engine binary plus approximately 14 MB allocated to the KV cache and sliding window. Memory usage does not increase with additional conversation turns because old tokens are discarded from the 256-token window while tool definitions remain pinned.

How many tools can be pinned simultaneously?

The retrieval head selects and pins at most five tools per turn. Because these KV slots use fixed-size allocations established when the tool is first seen, the memory overhead remains constant and predictable. The engine rejects or ignores additional tool candidates beyond this limit to maintain the bounded memory guarantee.

Does conversation length affect memory usage in Needle 2?

No, conversation length has no impact on memory consumption due to the 256-token sliding window architecture. As new tokens arrive, the engine discards the oldest unpinned tokens while retaining tool KV sinks. This design ensures that processing ten turns or ten thousand turns requires identical memory resources.

Where is the tool pinning logic implemented?

The pinning logic is implemented in the native inference engine (libneedle.so) downloaded from Hugging Face, not in the Python codebase. The Python wrapper in needle/__init__.py forwards tool schemas to the engine, which then manages KV slot creation, pinning, and memory region reuse internally. This architecture minimizes Python interpreter overhead and ensures consistent memory performance.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}