# What Is the Needle 2 Language Model? A 45M Parameter On-Device Foundation Model

> Discover Needle 2, a powerful 45M parameter on-device foundation model. Easily perform tool-calling, device control, and data extraction with this compact 14MB binary.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-08-28

---

**Needle 2 is a 45 million-parameter, on-device foundation model designed for tool-calling, device control, and structured data extraction, shipped as a 14 MB binary that runs in approximately 28 MB of RAM.**

Needle 2 represents a new class of edge-optimized transformers developed by Cactus Compute that enables autonomous AI agents without cloud dependency. Unlike traditional large language models requiring substantial GPU resources, this open-source architecture fits entirely on consumer devices such as smartphones, wearables, and embedded edge servers.

## Core Architecture: The Simple Attention Network

At the heart of the Needle 2 language model lies the `SimpleAttentionNetwork` class implemented in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py). This backbone replaces conventional transformer components with specialized efficiency layers that reduce computational overhead while maintaining reasoning capabilities.

### HadamardMLP and Walsh-Hadamard Transforms

The `HadamardMLP` class eliminates traditional dense matrix operations by computing `H·x` using pre-computed Walsh-Hadamard transforms. This matrix-free approach drastically reduces both parameter count and FLOPs compared to standard feed-forward networks. The implementation appears in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) alongside the attention mechanisms.

### Grouped-Query Attention with Flash Support

Attention is handled by the `MultiHeadAttention` class, which automatically utilizes **flash-attention** when GPU backends are available. On CPU-only devices, it falls back to a classic implementation featuring optional KV-compression to minimize memory bandwidth. The design employs **Grouped-Query Attention (GQA)** to share key and value heads across query heads, further reducing cache size.

### Engram Memory for Extended Context

The `Engram` class provides a learned key-value cache that stores compressed token histories via hash-based tables. This system enables Needle 2 to maintain longer context windows without the linear memory growth typical of standard transformers, keeping the working set within the ≈28 MB RAM budget.

## Quantization and Native Engine

Model weights are stored in **CQ2-bit** format (Cactus Quants) and lazily loaded by the inference engine. The quantization logic resides in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py), handling both weight storage and optional activation quantization to minimize data movement during inference.

The high-performance runtime is a compiled binary (approximately 14 MB) auto-downloaded from Hugging Face on first use. In [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py), the `_lib()` function manages engine initialization via `ctypes`, binding the Python API to the native code and loading the CQ2-bit weights into memory.

## Tool-Calling API and Python Interface

The public interface centers on the `Needle` class defined in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py). This thin wrapper accepts tool definitions—either Python functions decorated with `@tool`, Pydantic models, or raw JSON schemas—and orchestrates the inference loop.

When `run()` is invoked, the engine tokenizes input, executes a forward pass through `SimpleAttentionNetwork`, and emits JSON-encoded tool calls. The Python wrapper resolves these calls, executes the corresponding functions, and feeds results back into the model via the `complete` method until the model returns a final response or reaches the configured `max_steps` limit.

### Structured Data Extraction

Beyond agentic tool use, the `extract()` function enables zero-shot structured extraction. By passing text and a Pydantic model, developers can parse unstructured content into typed objects without fine-tuning, leveraging the model's built-in JSON generation capabilities.

## Practical Implementation Examples

The following examples demonstrate common usage patterns with the Needle 2 language model.

### Basic Tool-Calling Agent

```python
from needle import Needle, tool

@tool
def get_weather(city: str) -> str:
    """Return a short weather forecast for *city*."""
    return f"The weather in {city} is sunny."

# Instantiate the model (downloads binary on first use)

agent = Needle(tools=[get_weather])

# Run query - model automatically calls get_weather

response = agent.run("What will the weather be like in Paris tomorrow?")
print(response["results"][0])   # → "The weather in Paris is sunny."

print(response["text"])         # Final natural-language answer

```

### Structured Extraction with Pydantic

```python
from pydantic import BaseModel
from needle import extract

class FlightInfo(BaseModel):
    airline: str
    flight_number: str
    departure: str
    arrival: str

text = "I booked flight AA123 from JFK to LAX tomorrow morning."
info = extract(text, FlightInfo)
print(info)  # → FlightInfo(airline='AA', flight_number='123', ...)

```

### Loading Fine-Tuned Checkpoints

```python

# Load a LoRA-fine-tuned checkpoint (.cact archive)

agent = Needle(tools=[get_weather], weights="my_finetuned.cact")
response = agent.run("Is it raining in Berlin?")

```

### Command-Line Interface

The package includes a CLI entry point in [`needle/cli.py`](https://github.com/cactus-compute/needle/blob/main/needle/cli.py) supporting direct inference and fine-tuning workflows:

```bash

# Run inference from command line

needle run --prompt "What is the capital of France?"

# Fine-tune on custom data (produces .cact archive)

needle finetune --data training.jsonl --output my_model.cact

```

## Summary

- **Needle 2** is a 45M-parameter transformer optimized for on-device inference, requiring only ~28 MB RAM at runtime.
- The **Simple Attention Network** in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) uses HadamardMLP, GQA, and Engram memory to achieve efficiency.
- **CQ2-bit quantization** in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py) compresses weights to enable the 14 MB binary distribution.
- The Python API via [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) provides seamless tool-calling with the `Needle` class and `@tool` decorator.
- Native engine binding through `ctypes` delivers high-performance inference while maintaining a clean Python interface.

## Frequently Asked Questions

### What makes Needle 2 different from other compact language models?

Needle 2 distinguishes itself through the **HadamardMLP** architecture using Walsh-Hadamard transforms instead of dense matrices, the **Engram** memory system for compressed KV caching, and the **CQ2-bit** quantization format. These components—implemented in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) and [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py)—allow the model to operate within severe memory constraints while still supporting tool-calling and structured extraction.

### Can Needle 2 run on smartphones and IoT devices?

Yes. The compiled binary is approximately 14 MB and a full inference session requires roughly 28 MB of RAM. According to the implementation in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py), the engine loads lazily and utilizes CPU-optimized paths when GPUs are unavailable, making it suitable for phones, wearables, and edge servers without dedicated AI accelerators.

### How does the tool-calling mechanism work?

The model generates JSON-encoded tool calls during inference. The `Needle` class in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) parses these calls, looks up the corresponding Python function registered via the `@tool` decorator, executes it, and passes the result back to the model through the `complete` method. This loop continues until the model produces a final response or exceeds the `max_steps` threshold.

### What is the CQ2-bit quantization format?

CQ2-bit (Cactus Quants) is a proprietary 2-bit weight quantization scheme defined in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py). It stores model weights in a highly compressed format that the native engine decompresses on-the-fly during inference, reducing storage requirements and memory bandwidth while maintaining model accuracy for tool-calling tasks.