How Does Needle 2 Achieve Extreme Compactness? A Deep Dive into the 14 MB LLM Engine

Needle 2 achieves extreme compactness by combining a Simple Attention Network architecture with Hadamard MLPs, grouped-query attention, engram KV memory, and aggressive CQ2-bit quantization to compress a 45M-parameter model into a 14 MB binary that operates using only 28 MB of RAM.

The cactus-compute/needle repository hosts Needle 2, a foundation tool-calling model that delivers full LLM capabilities in a footprint small enough for edge devices. Understanding how Needle 2 achieves extreme compactness reveals a deliberate architectural shift away from standard Transformers toward parameter-efficient alternatives and extreme quantization strategies.

The Simple Attention Network Architecture

At the core of Needle 2’s efficiency lies the SimpleAttentionNetwork class implemented in needle/model/architecture.py (lines 78-84). This dense-only architecture replaces the conventional Transformer stack by eliminating separate feed-forward networks and introducing several memory-saving mechanisms that drastically reduce parameter count.

Hadamard MLP Implementation

Instead of learned FFN layers, Needle 2 employs a Hadamard MLP that uses a fixed orthonormal Walsh-Hadamard transform (_walsh_matrix) combined with only three learned diagonal vectors (d1, d2, d3) as seen in lines 80-103 of the architecture file. The transform computes in O(n log n) time without requiring stored weights, drastically reducing the feed-forward block's memory footprint while maintaining expressive power.

Grouped-Query Attention (GQA)

The model implements Grouped-Query Attention to minimize key/value projection parameters. According to lines 18-33 of needle/model/architecture.py, this technique shares key and value heads across groups, reducing the number of stored projection matrices while preserving attention quality and context understanding.

Engram KV Memory System

For long-range context without expanding the KV cache, Needle 2 utilizes an Engram KV Memory system. The Engram class (lines 81-88) implements a compact n-gram hashing table that stores key/value pairs in a fixed-size embedding, providing extended context memory without the linear growth typical of standard attention caches.

Multi-Lane Hyper-Connections

To emulate deep multi-layer behavior without stacking numerous layers, the architecture employs Multi-Lane Hyper-Connections. These learned routing matrices (mhc_* parameters defined in lines 94-108) steer information across parallel lanes, allowing a single layer to perform computations that traditionally require deep networks, effectively reusing parameters across the architecture.

Aggressive Quantization Strategy

Beyond architectural innovations, Needle 2 achieves its file size through extreme weight compression that goes beyond standard 8-bit quantization.

CQ2-Bit Quantization

The model weights undergo CQ2-bit quantization using the Cactus Quants library implemented in needle/model/quantize.py. This 4× compression over 8-bit representations reduces the entire 45M-parameter model to approximately 14 MB, storing only 2 bits per weight while maintaining inference accuracy comparable to much larger models.

Single-Binary Engine Design

The deployment strategy eliminates runtime overhead through unified packaging. As documented in the repository's README.md (lines 5-9), the inference runtime loads quantized weights once from a single .cact archive, requiring no external model files or network I/O. This design keeps the on-device footprint at approximately 28 MB of RAM during operation, enabling deployment on memory-constrained edge devices.

Practical Implementation

Deploying the compact engine requires no external dependencies beyond the Needle package:

# Load the ultra‑compact engine (14 MB) and run a query

import needle

# No external model files required – everything lives in the .cact archive

agent = needle.Needle(weights="needle2.cact", tools=[...])

# Simple tool‑calling example

result = agent.run("What’s the weather in Lagos?")["results"]
print(result)          # [{'city': 'Lagos', 'temp_c': 27, 'sky': 'clear'}]

For debugging or downstream tasks, you can access intermediate representations:

# Access the hidden states for debugging or downstream tasks

tokens = agent.tokenizer.encode("Explain quantum entanglement.")
hidden = agent.model.hidden_states(tokens)   # returns a list of layer‑wise representations

print(hidden.shape)      # (num_layers, batch, seq_len, d_model)

Summary

• Simple Attention Network: Replaces standard Transformers with dense-only operations in needle/model/architecture.py (lines 78-84), eliminating parameter-heavy FFN layers.
• Hadamard MLP: Uses fixed Walsh-Hadamard transforms with only three learned diagonal vectors (d1, d2, d3) to reduce feed-forward storage to near zero.
• Grouped-Query Attention: Shares KV heads across groups (lines 18-33) to minimize projection matrix counts.
• Engram Memory: Implements compact n-gram hashing via the Engram class (lines 81-88) for long-range context without cache expansion.
• Multi-Lane Hyper-Connections: Routes information through mhc_* parameters (lines 94-108) to simulate deep networks without layer stacking.
• CQ2-Bit Quantization: Compresses 45M parameters to 14 MB using 2-bit quantization in needle/model/quantize.py.
• Single-Binary Deployment: Packages the entire runtime into one 14 MB file loaded at initialization, reducing RAM usage to 28 MB.

Frequently Asked Questions

How small is the Needle 2 binary compared to similar models?

Needle 2 compresses a 45M-parameter model into a 14 MB binary, making it approximately 5-70× smaller than comparable models like FunctionGemma 270M while maintaining competitive performance on tool-calling tasks. This extreme compactness allows the model to outperform much larger alternatives while requiring only 28 MB of RAM during inference.

What is Hadamard MLP and why does it save space?

Hadamard MLP is a parameter-free feed-forward mechanism that uses fixed orthonormal Walsh-Hadamard transforms instead of learned weight matrices. By combining this transform with only three learned diagonal vectors (d1, d2, d3), the architecture eliminates storage for large FFN layers while preserving computational expressiveness through O(n log n) operations.

Does the 2-bit quantization affect model accuracy?

According to the cactus-compute/needle source code, the CQ2-bit quantization scheme maintains sufficient precision for tool-calling and general LLM tasks despite the aggressive 4× compression ratio. The 2-bit representation allows the 45M-parameter model to fit in 14 MB while outperforming much larger 8-bit models on specific edge-device benchmarks.

Can Needle 2 run on devices with limited RAM?

Yes, the Needle 2 engine requires only approximately 28 MB of RAM during inference, achieved through the combination of the Simple Attention Network architecture, Engram KV memory that prevents cache growth, and the single-binary loading strategy that eliminates duplicate weight storage in memory.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}