How Needle 2 Manages Memory to Stay Within 28 MB RAM: 5 Bounded Allocation Techniques

Needle 2 stays within approximately 28 MB of RAM by combining compact 14 MB binary weight storage, aggressive 2-bit quantization, a fixed 256-token sliding window for KV cache management, pre-allocated fixed-size buffers, and memory-bounded configuration contracts.

Needle 2 delivers on-device LLM inference for severely memory-constrained environments according to the cactus-compute/needle source code. The repository implements a bounded memory design that guarantees predictable RAM usage regardless of conversation length, making it suitable for edge devices and microcontrollers.

Compact Weight Storage: The 14 MB Binary Foundation

The model's 45 million parameters are distributed as a single 14 MB binary file (needle2.cact) hosted on Hugging Face. This file is downloaded once and cached locally.

In needle/model/architecture.py, the TransformerConfig loads this binary directly at runtime. No separate weight files are parsed or loaded into auxiliary structures—the engine memory-maps or reads the binary directly into the inference buffer.

This approach cuts the model's static memory cost in half compared to conventional float16 or float32 storage. The 14 MB binary becomes the baseline upon which all other memory optimizations build.

CQ 2-Bit Quantization: Minimal Precision, Maximum Compression

All weights are quantized to 2 bits using the Cactus Quants (CQ) scheme implemented in needle/model/quantize.py. The quantize_params() function applies this compression, while fake_quant() enables training-time simulation of the quantization effects.

Traditional inference engines often use 16-bit or 32-bit floating-point weights. Needle's 2-bit quantization reduces weight tensor memory by 8× to 16× compared to standard formats. This aggressive compression preserves sufficient accuracy for tool-calling tasks while keeping the weight matrix footprint minimal.

The quantization is not optional—it is baked into the 14 MB binary distribution. Users receive pre-quantized weights; no runtime conversion occurs.

256-Token Sliding Window: Bounding the KV Cache

The KV cache represents the largest variable memory cost in transformer inference. Needle 2 eliminates variance through a fixed-size sliding window mechanism.

In needle/model/run.py, the implementation enforces:

• kv_window = 256 — Only the most recent 256 tokens' key-value pairs are retained
• Tool KV pinning — KV pairs associated with tool calls are marked as "sinks" and never evicted
• Automatic eviction — Older non-tool tokens are discarded when the window fills

This design guarantees the KV cache never exceeds approximately 14 MB, regardless of how long the conversation runs. The 256-token window provides sufficient context for multi-turn tool-using dialogues while eliminating the unbounded growth that plagues standard attention implementations.

Fixed-Size Buffers: Eliminating Heap Fragmentation

Runtime memory allocation introduces unpredictable overhead and fragmentation. Needle 2 pre-allocates all working memory.

In needle/model/run.py, the constant BUF_BUCKET = 128 defines the allocation granularity. Internal buffers for activations, intermediate results, and temporary tensors are allocated once at startup and reused across all inference steps.

This technique:

• Eliminates repeated malloc/free cycles during generation
• Prevents heap fragmentation that would otherwise expand the memory footprint over time
• Keeps the runtime heap size deterministic and predictable

Memory-Bounded Configuration: Enforcing the 28 MB Contract

The TransformerConfig class in needle/model/architecture.py exposes explicit memory parameters:

• kv_window — Sets the sliding window size (default: 256)
• kv_bits — Controls KV cache quantization (default: 8-bit)

These defaults are not suggestions—they are hard contracts. The configuration system rejects or warns on combinations that would exceed the 28 MB target. The API specification in doc/apis.md documents observed peak RAM usage of 28.5 MB, confirming the design achieves its goal.

Practical Usage: Running Within the Memory Budget

import needle

# Default configuration automatically respects 28 MB limit.

# Loads 14 MB binary and applies 256-token sliding window.

agent = needle.Needle(
    tools=[my_tool],
    weights="cactus-needle/needle2.cact",  # 14 MB cached binary

)

# Multi-turn conversation—KV cache bounded regardless of length

response = agent.run("What's the weather in Tokyo?")
print(response["results"])

# Advanced: explicit configuration for custom constraints

from needle.model.architecture import TransformerConfig

config = TransformerConfig(kv_window=256, kv_bits=8)
agent = needle.Needle(tools=[my_tool], config=config)

The README emphasizes this guarantee explicitly: "Bounded memory: a 256-token sliding window with the tools pinned as KV sinks, so total memory stays near 28 MB no matter how long the conversation runs."

Summary

Needle 2 achieves its 28 MB RAM target through five complementary techniques:

• Compact weight storage — Single 14 MB binary eliminates redundant loading overhead
• CQ 2-bit quantization — 8-16× weight compression via needle/model/quantize.py
• 256-token sliding window — Hard cap on KV cache growth with tool-aware pinning
• Fixed-size buffers — Pre-allocated pools via BUF_BUCKET = 128 in needle/model/run.py
• Memory-bounded configuration — Enforced defaults in TransformerConfig

The result is ≈ 14 MB (weights) + ≈ 14 MB (KV/buffers) ≈ 28 MB total, constant across arbitrary conversation lengths.

Frequently Asked Questions

What happens if I increase the KV window beyond 256 tokens?

Increasing kv_window directly increases peak RAM usage. Each additional token requires additional key-value storage. The 28 MB guarantee applies only to the default 256-token configuration. Consult needle/model/architecture.py to calculate memory for custom values: each token's KV pair consumes hidden_dim × kv_bits / 8 bytes.

Does the 2-bit quantization hurt model accuracy?

The Cactus Quants scheme in needle/model/quantize.py is trained with quantization-aware optimization via fake_quant(). For tool-calling tasks, the accuracy loss is negligible according to repository benchmarks. The 2-bit quantization is not user-configurable—the distributed binary is pre-optimized.

Can Needle 2 run on devices with less than 28 MB RAM?

The repository targets 28 MB as a guaranteed ceiling, not a minimum requirement. Actual usage varies slightly based on Python interpreter overhead and operating system. For devices below this threshold, you would need custom quantization or model distillation outside the current cactus-compute/needle implementation scope.

How does tool KV pinning prevent memory growth?

Tool-related KV pairs are marked as persistent "sinks" in the sliding window implementation. While this reserves some fixed slots in the 256-token window, it prevents the unbounded accumulation of tool state that would otherwise occur across extended multi-tool conversations. The pinned slots count toward the fixed 256-token budget, not beyond it.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}