How Needle 2 Maintains Low Memory Usage: Inside the 28 MiB Architecture

Needle 2 runs complete inference sessions in approximately 28 MiB of RAM despite containing 45 million parameters, achieved through a bounded sliding-window KV cache, 2-bit quantization, and a lightweight Simple Attention Network that replaces traditional feed-forward layers.

The cactus-compute/needle repository delivers a radical reimagining of transformer efficiency, engineering a compact language model that maintains fixed memory consumption regardless of conversation length. By systematically replacing memory-heavy components—from full-precision attention caches to dense MLP blocks—with constrained alternatives, the architecture achieves its low memory usage target while preserving tool-use capabilities. Examining the source code reveals six tightly-coupled mechanisms that prevent the typical memory expansion seen in standard transformers.

Bounded Sliding-Window KV Cache

The primary defense against unbounded memory growth is a bounded sliding-window KV cache that retains only the most recent 256 tokens during inference. Unlike standard transformers that accumulate key-value pairs indefinitely as conversations lengthen, Needle 2 discards older tokens automatically, ensuring the cache never exceeds its preset allocation.

Special handling for tool calls prevents context loss: tools are "pinned" as dedicated KV sinks, guaranteeing that critical function definitions remain accessible without expanding the memory window. This design is orchestrated within the Stack module in needle/model/architecture.py, which manages the entire forward pass while enforcing the 256-token constraint.

Simple Attention Network (SAN) with Hadamard MLP

Traditional transformer blocks rely on parameter-intensive feed-forward networks (FFNs) that consume significant memory during activation. Needle 2 replaces these with a Simple Attention Network (SAN) featuring a HadamardMLP—a lightweight transformation that applies element-wise multiplication (the Hadamard product) without learned weight matrices for the hidden-to-hidden mapping.

As implemented in needle/model/architecture.py (lines 87-103), the HadamardMLP class eliminates the storage and computation overhead of dense linear layers. The architecture couples this with Grouped-Query Attention (GQA), reducing the number of attention heads that must be cached simultaneously. Together, these modifications slash the parameter count that must reside in working memory during each forward pass.

Engram KV Memory

Complementing the sliding-window cache is the Engram KV memory system—a compact storage mechanism that replaces full-precision activation tensors with hashed n-gram tables. Located in needle/model/architecture.py (lines 81-105), the Engram class maintains a fixed table of 8,192 slots accessed via fast hash-based indexing.

Rather than storing complete key-value vectors for every token in history, the engram compresses context into compact representations. This shrinks the KV cache footprint dramatically while preserving retrieval capabilities for relevant historical patterns.

CQ2-Bit Quantization

Weight storage represents another major memory sink in large models. Needle 2 addresses this through CQ2-bit quantization, implemented in needle/model/quantize.py (lines 1-30), which compresses the 45M parameters into a 14 MiB binary engine file using the Cactus Quants library.

The fake_quant_act function applies fake-quantization to activations at inference time, allowing computations to proceed in reduced precision without maintaining full float32 buffers for intermediate results. Weights are stored in 2-bit precision, cutting the model storage requirements by roughly 16× compared to standard float32 representations.

Layer-Wise Gating and Flash Attention

Auxiliary optimizations further trim runtime memory allocation. Each transformer block incorporates learnable gating mechanisms (_gate attributes in needle/model/architecture.py, lines 14-16) that can silence entire sub-networks when their contributions are unnecessary, preventing the storage of useless intermediate activations.

When GPU hardware is available, the implementation falls back to cuDNN flash attention (needle/model/architecture.py, lines 45-48), which reuses memory buffers and avoids allocating large intermediate attention matrices. On CPU, the Stack module casts all inputs to bfloat16 by default, halving the activation memory footprint compared to float32 operations.

Code Implementation and Memory Verification

Needle 2 exposes its memory characteristics through the public API, allowing verification of the low memory usage claims. The following example demonstrates tool-augmented inference:

import needle

@needle.tool
def get_weather(city: str):
    "Get the current weather for a city."
    return {"city": city, "temp_c": 27, "sky": "clear"}

agent = needle.Needle(tools=[get_weather])
result = agent.run("What's the weather like in Lagos right now?")
print(result["results"])

# [{'city': 'Lagos', 'temp_c': 27, 'sky': 'clear'}]

To inspect the actual memory constraints:

import needle
print("Engine size (bytes):", needle.engine.size_bytes)   # ~14,000,000

print("Approx. RAM usage (MiB):", needle.engine.approx_mem_mib)  # ~28

These attributes reflect the bounded-memory design enforced by the Stack module, which coordinates the engram KV sink, SAN blocks, and quantization layers to maintain the 28 MiB ceiling.

Summary

• Bounded sliding-window KV cache caps context at 256 tokens with tool pinning to prevent unbounded growth during long conversations.
• Simple Attention Network replaces dense FFNs with HadamardMLP and GQA, reducing parameter storage and multiplication overhead in needle/model/architecture.py.
• Engram KV memory compresses historical context into 8,192 hashed slots rather than full-precision activation tensors.
• CQ2-bit quantization stores the 45M-parameter model in a 14 MiB binary using 2-bit precision via needle/model/quantize.py.
• Layer-wise gating and flash attention eliminate unnecessary activation storage and optimize buffer reuse on GPU hardware.

Frequently Asked Questions

What is the maximum RAM usage for Needle 2 inference?

Needle 2 maintains a fixed memory footprint of approximately 28 MiB during inference, regardless of conversation length or the number of tool calls. This constraint is enforced by the Stack module in needle/model/architecture.py, which coordinates the bounded KV cache and engram memory systems to prevent allocation beyond the preset limit.

How does the sliding-window KV cache handle long conversations?

The cache retains only the most recent 256 tokens as a sliding window, discarding older activations automatically. Critical tool definitions are preserved as pinned KV sinks, ensuring that function signatures remain available even when older conversation history is evicted from the cache.

What is CQ2-bit quantization and why does it save memory?

CQ2-bit quantization is a compression scheme implemented in needle/model/quantize.py that represents model weights using only 2 bits per parameter instead of 32-bit floats. This reduces the 45M-parameter model to a 14 MiB binary file, cutting weight storage requirements by roughly 93% while maintaining inference quality through fake-quantized activations.

Can Needle 2 utilize GPU acceleration without increasing memory usage?

Yes. When a GPU is available, Needle 2 activates flash attention paths (lines 45-48 in needle/model/architecture.py) that leverage cuDNN's memory-efficient kernels. These implementations reuse buffers and avoid materializing large attention matrices, keeping RAM usage at the 28 MiB target even during GPU-accelerated inference.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}