# Performance Characteristics of Needle 2: How a 45M-Parameter Model Outperforms 200M+ Competitors

> Discover Needle 2's performance. This 45M-parameter model rivals 200M+ competitors in accuracy with a fraction of the resources. Learn how it achieves this efficiency.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: performance
- Published: 2026-08-17

---

**Needle 2 delivers comparable tool-calling accuracy to 200M+ parameter models while using only 14 MB of storage and 28 MB of RAM, achieved through 2-bit quantization and a novel Simple Attention Network architecture.**

The performance characteristics of Needle 2 make it a unique foundation model in the efficient AI landscape. According to the cactus-compute/needle repository, this 45 million-parameter model rivals the functional capabilities of models five to seventy times its size through aggressive quantization and architectural innovations that minimize memory bandwidth and compute requirements.

## Model Size and Memory Footprint

Needle 2 redefines what is possible with tiny transformer architectures by optimizing for binary size and runtime RAM rather than raw parameter count.

### Binary Footprint and Parameter Count

The model compresses **45 million parameters** into a **14 MB binary**, a compression ratio achieved through the custom CQ2-bit quantization scheme implemented in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py). This stands in stark contrast to competitors like FunctionGemma 270M, LFM2.5 230M, and Apple FM, which typically ship as FP16 checkpoints ranging from 500 MB to over 1 GB.

### Runtime Memory Requirements

During inference, Needle 2 consumes approximately **28 MB of RAM** for a full conversation using a 256-token sliding window. This memory ceiling remains constant regardless of conversation length due to the **sliding-window KV sink** mechanism, which pins tool-related key-value pairs while discarding older context. The memory efficiency stems from the architecture defined in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), where the Engram KV Memory stores (kₜ, vₜ) rows in hashed n-gram tables rather than dense matrices.

## Quantization Strategy: CQ2-bit Precision

Unlike standard models using FP16 (16-bit) weights, Needle 2 employs **CQ2-bit quantization** (2-bit per weight) via the Cactus Quants scheme. This radical precision reduction—implemented in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py)—enables the 14 MB footprint without sacrificing tool-calling accuracy. The quantization scheme preserves model expressivity while slashing storage and compute costs, allowing the inference engine to run as a single static binary with no dynamic weight loading overhead.

## Architectural Innovations Driving Efficiency

Needle 2 replaces the traditional transformer stack with a **Simple Attention Network (SAN)** detailed in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py). This architecture eliminates standard feed-forward networks in favor of fixed transforms and shared attention mechanisms.

### Hadamard MLP and Walsh-Hadamard Transform

The **Hadamard MLP** replaces conventional learned feed-forward networks with a fixed Walsh-Hadamard transform operating in O(n log n) time. Because this transform requires no learned weights, it drastically reduces parameter count and memory bandwidth requirements during the forward pass. The implementation in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) applies this transform as a drop-in replacement for standard MLP blocks.

### Grouped-Query Attention (GQA)

**GQA (Grouped-Query Attention)** splits queries into groups that share keys and values across attention heads. This technique, visible in the attention implementation within [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), cuts the attention matrix size while preserving model expressivity. By reducing the KV cache memory pressure, GQA enables the constant 28 MB RAM utilization even during long-form tool-calling conversations.

### Engram KV Memory and Sliding-Window Management

The **Engram KV Memory** system stores token-level context in hashed n-gram tables, enabling constant-time lookups rather than linear scans through context history. Complementing this, the **sliding-window KV sink** explicitly retains tool-related KV pairs while allowing older conversation context to expire. These mechanisms, orchestrated in [`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py), guarantee that memory usage never exceeds the 28 MB ceiling regardless of dialogue length.

### Multi-lane Hyper-connections

Signal routing occurs through **multi-lane hyper-connections** that utilize a doubly-stochastic matrix P computed via Sinkhorn iteration. This routing method—defined in the architecture diagram referenced in [`README.md`](https://github.com/cactus-compute/needle/blob/main/README.md)—improves information flow between layers without inflating the parameter count, avoiding the dense projection matrices typical in standard transformers.

## Benchmark Performance vs. Similar-Size Models

On the benchmark suite included in the repository, Needle 2 "trades wins" with FunctionGemma 270M, LFM2.5 230M, and Apple FM models despite being **5× to 70× smaller**. The model achieves this parity through the **byte-level grammar** compiler enforced in [`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py), which guarantees syntactically correct JSON tool arguments and reduces post-processing overhead that often degrades effective accuracy in larger models.

Per-token latency remains competitive with these larger models because Needle 2 requires no dynamic weight loading from storage. While competitors must page hundreds of megabytes of FP16 weights into memory, Needle 2 executes entirely within its 14 MB static binary and 28 MB working set.

## Running Needle 2: Code Examples

The following examples demonstrate how to leverage Needle 2's efficient inference engine for tool-calling and structured extraction tasks.

### Basic Tool-Calling Inference

```python
import needle

@needle.tool
def get_weather(city: str):
    """Get the current weather for a city."""
    return {"city": city, "temp_c": 27, "sky": "clear"}

agent = needle.Needle(tools=[get_weather])
result = agent.run("What's the weather like in Lagos right now?")
print(result["results"])

# → [{'city': 'Lagos', 'temp_c': 27, 'sky': 'clear'}]

```

### Structured Extraction

```python
from pydantic import BaseModel
import needle

class Invoice(BaseModel):
    vendor: str
    total: float
    due_date: str

text = "Invoice from Acme Corp, $1,200.00, due 2026-09-01"
invoice = needle.extract(text, Invoice)
print(invoice.vendor, invoice.total)

# → Acme Corp 1200.0

```

### Loading a Fine-Tuned Model

```python
import needle

# Load a custom `.cact` checkpoint (still 14 MB binary)

agent = needle.Needle(weights="my_needle.cact", tools=[get_weather])
print(agent.run("Summarize the last three tool calls.")["results"])

```

## Summary

- **Needle 2 operates at 14 MB binary size and 28 MB RAM**, compared to 500 MB+ requirements for similar-capability models.
- **CQ2-bit quantization** (2-bit weights) enables extreme compression without sacrificing tool-calling accuracy.
- The **Simple Attention Network** replaces standard transformers with Hadamard MLPs, Grouped-Query Attention, and Engram KV Memory to minimize compute and memory bandwidth.
- **Performance parity** with FunctionGemma 270M and LFM2.5 230M models demonstrates that architectural efficiency can overcome raw parameter count advantages.
- The implementation in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) and [`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py) uses byte-level grammar enforcement and sliding-window KV sinks to maintain constant memory usage during inference.

## Frequently Asked Questions

### How does Needle 2 achieve such a small memory footprint compared to other LLMs?

Needle 2 achieves its tiny footprint through **CQ2-bit quantization** (2-bit weights versus standard 16-bit FP16) and the **Simple Attention Network** architecture that replaces parameter-heavy MLPs with fixed Walsh-Hadamard transforms. The **Engram KV Memory** system stores context in hashed n-gram tables rather than dense matrices, while the sliding-window KV sink caps total memory at 28 MB regardless of conversation length.

### What is the Simple Attention Network (SAN) architecture?

The **Simple Attention Network** is Needle 2's custom transformer replacement implemented in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py). It combines **Hadamard MLPs** (fixed transforms instead of learned weights), **Grouped-Query Attention** (shared KV heads), and **multi-lane hyper-connections** (Sinkhorn-iterated routing matrices). These components work together to preserve model expressivity while eliminating the parameter bloat typical of standard transformer blocks.

### How does Needle 2's accuracy compare to larger models like FunctionGemma?

Despite being 5× to 70× smaller, Needle 2 "trades wins" with FunctionGemma 270M, LFM2.5 230M, and Apple FM on the repository's benchmark suite. The **byte-level grammar** compiler ensures syntactically valid tool outputs, reducing error rates that often plague larger models. According to the source code analysis, the model maintains comparable win rates on tool-calling tasks while operating at significantly lower latency due to zero dynamic loading overhead.

### Can Needle 2 run on mobile or edge devices?

Yes, Needle 2 is specifically optimized for edge deployment. The **14 MB binary** and **28 MB RAM** requirements fit comfortably within mobile phone constraints, and the single-binary architecture with no external weight dependencies simplifies deployment. The [`needle/cli.py`](https://github.com/cactus-compute/needle/blob/main/needle/cli.py) utilities support building custom `.cact` checkpoints that maintain the same size constraints, making the model ideal for resource-constrained environments where storage and compute are limited.