# How Does Needle 2 Achieve Extreme Compactness? A Deep Dive into the 14 MB LLM Engine

> Discover how Needle 2 achieves extreme compactness using a novel architecture and aggressive quantization to deliver a powerful 14 MB LLM engine.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-08-26

---

**Needle 2 achieves extreme compactness by combining a Simple Attention Network architecture with Hadamard MLPs, grouped-query attention, engram KV memory, and aggressive CQ2-bit quantization to compress a 45M-parameter model into a 14 MB binary that operates using only 28 MB of RAM.**

The cactus-compute/needle repository hosts Needle 2, a foundation tool-calling model that delivers full LLM capabilities in a footprint small enough for edge devices. Understanding how Needle 2 achieves extreme compactness reveals a deliberate architectural shift away from standard Transformers toward parameter-efficient alternatives and extreme quantization strategies.

## The Simple Attention Network Architecture

At the core of Needle 2’s efficiency lies the `SimpleAttentionNetwork` class implemented in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) (lines 78-84). This dense-only architecture replaces the conventional Transformer stack by eliminating separate feed-forward networks and introducing several memory-saving mechanisms that drastically reduce parameter count.

### Hadamard MLP Implementation

Instead of learned FFN layers, Needle 2 employs a **Hadamard MLP** that uses a fixed orthonormal Walsh-Hadamard transform (`_walsh_matrix`) combined with only three learned diagonal vectors (`d1`, `d2`, `d3`) as seen in lines 80-103 of the architecture file. The transform computes in O(n log n) time without requiring stored weights, drastically reducing the feed-forward block's memory footprint while maintaining expressive power.

### Grouped-Query Attention (GQA)

The model implements **Grouped-Query Attention** to minimize key/value projection parameters. According to lines 18-33 of [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), this technique shares key and value heads across groups, reducing the number of stored projection matrices while preserving attention quality and context understanding.

### Engram KV Memory System

For long-range context without expanding the KV cache, Needle 2 utilizes an **Engram KV Memory** system. The `Engram` class (lines 81-88) implements a compact n-gram hashing table that stores key/value pairs in a fixed-size embedding, providing extended context memory without the linear growth typical of standard attention caches.

### Multi-Lane Hyper-Connections

To emulate deep multi-layer behavior without stacking numerous layers, the architecture employs **Multi-Lane Hyper-Connections**. These learned routing matrices (`mhc_*` parameters defined in lines 94-108) steer information across parallel lanes, allowing a single layer to perform computations that traditionally require deep networks, effectively reusing parameters across the architecture.

## Aggressive Quantization Strategy

Beyond architectural innovations, Needle 2 achieves its file size through extreme weight compression that goes beyond standard 8-bit quantization.

### CQ2-Bit Quantization

The model weights undergo **CQ2-bit quantization** using the Cactus Quants library implemented in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py). This 4× compression over 8-bit representations reduces the entire 45M-parameter model to approximately 14 MB, storing only 2 bits per weight while maintaining inference accuracy comparable to much larger models.

## Single-Binary Engine Design

The deployment strategy eliminates runtime overhead through unified packaging. As documented in the repository's [`README.md`](https://github.com/cactus-compute/needle/blob/main/README.md) (lines 5-9), the inference runtime loads quantized weights once from a single `.cact` archive, requiring no external model files or network I/O. This design keeps the on-device footprint at approximately 28 MB of RAM during operation, enabling deployment on memory-constrained edge devices.

## Practical Implementation

Deploying the compact engine requires no external dependencies beyond the Needle package:

```python

# Load the ultra‑compact engine (14 MB) and run a query

import needle

# No external model files required – everything lives in the .cact archive

agent = needle.Needle(weights="needle2.cact", tools=[...])

# Simple tool‑calling example

result = agent.run("What’s the weather in Lagos?")["results"]
print(result)          # [{'city': 'Lagos', 'temp_c': 27, 'sky': 'clear'}]

```

For debugging or downstream tasks, you can access intermediate representations:

```python

# Access the hidden states for debugging or downstream tasks

tokens = agent.tokenizer.encode("Explain quantum entanglement.")
hidden = agent.model.hidden_states(tokens)   # returns a list of layer‑wise representations

print(hidden.shape)      # (num_layers, batch, seq_len, d_model)

```

## Summary

- **Simple Attention Network**: Replaces standard Transformers with dense-only operations in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) (lines 78-84), eliminating parameter-heavy FFN layers.
- **Hadamard MLP**: Uses fixed Walsh-Hadamard transforms with only three learned diagonal vectors (`d1`, `d2`, `d3`) to reduce feed-forward storage to near zero.
- **Grouped-Query Attention**: Shares KV heads across groups (lines 18-33) to minimize projection matrix counts.
- **Engram Memory**: Implements compact n-gram hashing via the `Engram` class (lines 81-88) for long-range context without cache expansion.
- **Multi-Lane Hyper-Connections**: Routes information through `mhc_*` parameters (lines 94-108) to simulate deep networks without layer stacking.
- **CQ2-Bit Quantization**: Compresses 45M parameters to 14 MB using 2-bit quantization in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py).
- **Single-Binary Deployment**: Packages the entire runtime into one 14 MB file loaded at initialization, reducing RAM usage to 28 MB.

## Frequently Asked Questions

### How small is the Needle 2 binary compared to similar models?

Needle 2 compresses a 45M-parameter model into a 14 MB binary, making it approximately 5-70× smaller than comparable models like FunctionGemma 270M while maintaining competitive performance on tool-calling tasks. This extreme compactness allows the model to outperform much larger alternatives while requiring only 28 MB of RAM during inference.

### What is Hadamard MLP and why does it save space?

**Hadamard MLP** is a parameter-free feed-forward mechanism that uses fixed orthonormal Walsh-Hadamard transforms instead of learned weight matrices. By combining this transform with only three learned diagonal vectors (`d1`, `d2`, `d3`), the architecture eliminates storage for large FFN layers while preserving computational expressiveness through O(n log n) operations.

### Does the 2-bit quantization affect model accuracy?

According to the cactus-compute/needle source code, the CQ2-bit quantization scheme maintains sufficient precision for tool-calling and general LLM tasks despite the aggressive 4× compression ratio. The 2-bit representation allows the 45M-parameter model to fit in 14 MB while outperforming much larger 8-bit models on specific edge-device benchmarks.

### Can Needle 2 run on devices with limited RAM?

Yes, the Needle 2 engine requires only approximately 28 MB of RAM during inference, achieved through the combination of the Simple Attention Network architecture, Engram KV memory that prevents cache growth, and the single-binary loading strategy that eliminates duplicate weight storage in memory.