# Needle 2 45M Parameter Model: Comparing Compact Foundation Models

> Discover how Needle 2's 45M parameter model stacks up against larger foundation models. Achieve efficient tool-calling and extraction with this compact, 4-bit quantized solution.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: comparison
- Published: 2026-09-11

---

**Needle 2 delivers competitive tool-calling and structured extraction capabilities using only 45 million 4-bit quantized parameters, making it 5–6× smaller than comparable foundation models like FunctionGemma 270M while fitting entirely within a 14MB binary footprint.**

The cactus-compute/needle repository positions Needle 2 as a highly efficient alternative to resource-intensive foundation models. By leveraging a custom 4-bit quantization format referred to as "4SM" in the project documentation, this 45M-parameter engine trades wins with models nearly six times its size. According to the source code in [`README.md`](https://github.com/cactus-compute/needle/blob/main/README.md), Needle 2 achieves functional parity with competitors using just **2 bits** against their **f16** precision, enabling deployment on memory-constrained devices.

## Understanding the 45M Parameter Architecture

Needle 2 implements a **Simple Attention Network** architecture optimized for minimal footprint without sacrificing core capabilities. The model's compression strategy centers on aggressive quantization techniques that preserve functional accuracy.

### The 4SM Quantization Format

In [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), the codebase defines the quantization pipeline responsible for compressing the full-precision model down to 4 bits. This "4SM" format (referenced in the project paper) reduces the model footprint to approximately **14MB** for the complete engine binary. The architecture file implements custom kernels that maintain inference speed despite the reduced precision, allowing the 45 million parameters to execute efficiently on standard hardware.

## Parameter Count Comparison Against Foundation Models

The repository's [`README.md`](https://github.com/cactus-compute/needle/blob/main/README.md) explicitly benchmarks Needle 2 against larger foundation models, demonstrating that parameter count does not directly correlate with task performance:

- **FunctionGemma 270M**: At 270 million parameters, this model is approximately **6× larger** than Needle 2, yet the README notes that Needle 2 "trades wins" with FunctionGemma on tool-calling tasks.
- **LFM 2.5 230M**: With 230 million parameters, this competitor is roughly **5× larger** than Needle 2's 45M count.
- **Apple FM**: Falling within the 200M–250M parameter range, Apple's foundation model is **5–6× larger** while offering comparable functionality.

These comparisons, documented in lines 5–6 of [`README.md`](https://github.com/cactus-compute/needle/blob/main/README.md), establish Needle 2 as operating within a niche of **5× to 70× smaller** models relative to mainstream alternatives, depending on specific configuration and precision requirements.

## Running Inference with the Compact Model

Despite its reduced size, Needle 2 exposes a full-featured API through [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) for tool-calling and structured data extraction.

### Loading the 45M Parameter Engine

The following example demonstrates initializing the 4-bit model and executing a multi-tool query:

```python
import needle

# Load the pre‑downloaded 4‑bit engine (14 MB) and a set of tools

agent = needle.Needle(
    weights="needle2.cact",          # 45 M‑parameter, 4‑bit model

    tools=[get_weather, set_lights] # user‑defined tool functions

)

# Simple tool‑calling query

result = agent.run("What’s the weather in Nairobi and dim the living‑room lights to 20 %")
print(result["results"])

# → [{'city': 'Nairobi', 'temp_c': 23, 'sky': 'cloudy'},

#    {'room': 'living_room', 'brightness': 20}]

```

### Structured Extraction

For data parsing tasks, the model supports Pydantic schema validation as implemented in the package entry point:

```python

# Structured extraction using a Pydantic schema

from pydantic import BaseModel

class Invoice(BaseModel):
    vendor: str
    total: float
    due_date: str

extracted = needle.extract(
    "Invoice from Acme Corp, $1,200.00, due 2026-10-15", Invoice
)
print(extracted.vendor, extracted.total)

# → Acme Corp 1200.0

```

## Performance Characteristics at 45M Parameters

Needle 2 challenges the scaling laws that dominate modern foundation model development. While competitors like FunctionGemma 270M and LFM 2.5 230M utilize **f16** (16-bit) floating-point precision, Needle 2 operates at **2-bit** effective compression in its 4SM format. This architectural choice enables the model to load entirely into L1/L2 cache on modern processors, reducing memory bandwidth bottlenecks that typically constrain larger models.

According to the documentation in [`doc/apis.md`](https://github.com/cactus-compute/needle/blob/main/doc/apis.md), the reduced parameter count does not limit the model's ability to handle complex tool definitions or multi-step reasoning chains. The 14MB binary size allows for instant startup times and embedded deployment scenarios impossible with 200M+ parameter alternatives.

## Summary

Needle 2 demonstrates that foundation model effectiveness depends on architectural efficiency rather than raw parameter volume:

- **45 million parameters** with 4-bit quantization deliver comparable performance to 200M+ parameter models
- **14MB binary footprint** enables edge deployment and cache-resident inference
- **5–6× smaller** than FunctionGemma 270M and LFM 2.5 230M while maintaining functional parity
- **Simple Attention Network** architecture defined in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) optimizes for memory-constrained environments

## Frequently Asked Questions

### How does Needle 2 achieve competitive performance with only 45M parameters?

Needle 2 utilizes a **Simple Attention Network** architecture combined with aggressive **4SM quantization** (4-bit) that reduces memory bandwidth requirements. By implementing custom inference kernels in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), the model maximizes the efficiency of each parameter, allowing 45M 4-bit weights to perform comparably to 200M+ 16-bit models on tool-calling tasks.

### What is the 4SM quantization format mentioned in the Needle 2 paper?

The **4SM** format is a proprietary 4-bit quantization scheme referenced in the Needle 2 research documentation and implemented in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py). It compresses the model to approximately **14MB** (compared to 400MB+ for f16 alternatives) while preserving enough precision for accurate function calling and structured extraction.

### Can Needle 2 run on edge devices given its small footprint?

Yes, the **14MB binary size** and **45M parameter count** make Needle 2 ideal for edge deployment. The entire model fits within standard CPU cache levels, eliminating the GPU requirements and large RAM allocations necessary for 200M+ parameter foundation models like Apple FM or FunctionGemma.

### How do I load the 45M parameter model in my application?

Import the `Needle` class from [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) and specify the 4-bit weights file: `agent = needle.Needle(weights="needle2.cact", tools=[your_function])`. This loads the 45M-parameter engine with the 4SM quantization format ready for inference.