# What Is the Overall Goal of the Needle 2 Project? Inside the 45M-Parameter Edge AI Engine

> Discover Needle 2 project's goal: ship a 14MB binary with a 45M-parameter AI model, inference engine, and grammar decoding for offline, low-RAM edge devices.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-08-19

---

**The overall goal of the Needle 2 project is to ship a single 14 MiB binary that bundles a 45-million-parameter foundation model, a lightweight inference engine, and grammar-constrained decoding so the entire system runs on devices with roughly 28 MiB of RAM without any network access.**

Needle 2, maintained in the `cactus-compute/needle` repository, targets **tool-calling**, **structured extraction**, and **confidence-gated inference** on the smallest edge devices. Every design choice—from the custom 2-bit weight format to the sliding-window memory—supports the overall goal of the Needle 2 project: deliver a self-contained AI that generates schema-valid JSON and knows when to escalate uncertain answers.

## The 14 MiB Self-Contained Archive

Unlike typical deployments that require separate weight files, runtime libraries, and network endpoints, Needle 2 compresses everything into one `.cact` archive. According to the repository source code, the `needle build` command—implemented in [`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py)—merges quantized weights, the byte-level grammar, and the inference runtime into a single file. At inference time, [`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py) loads this archive locally, which means **no network traffic** occurs during decoding.

This archive contains:

- **Quantized model weights** in a custom CQ2 2-bit format decoded on-the-fly.
- **Tool schemas** compiled into a byte-level grammar that forces valid JSON output.
- **LoRA adapters** (optional) that have been merged back into the base weights at export time.

## Architectural Highlights

The model architecture lives primarily in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) and replaces standard transformer blocks with components optimized for minimal memory use.

### Simple Attention Network (SAN)

The backbone of Needle 2 is the **Simple Attention Network (SAN)**, defined in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) inside the `SimpleAttentionNetwork` class. It replaces the conventional feed-forward network with a **Hadamard-MLP** and uses **Grouped-Query Attention (GQA)** to reduce key-value cache size. These changes cut memory overhead while preserving the model’s ability to attend across tool descriptions and user prompts.

### Engram Memory for Bounded Context

Instead of storing an ever-growing key-value history, Needle 2 uses an **Engram** memory system—also in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) via the `Engram` class and the `engram_kv` pipeline. This is a hashed n-gram key-value store that provides a sliding window of roughly **256 tokens**. Because the window is bounded, the runtime footprint stays near **28 MiB** regardless of conversation length.

### CQ2-Bit Quantization Engine

Weights are stored in a custom **CQ2 2-bit format** and decoded on-the-fly during inference. The quantization helpers in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py)—invoked through `maybe_quant_*` utilities—keep the storage tiny and eliminate the need to decompress full float16 or float32 tensors into RAM. This is how Needle 2 maintains its sub-30 MiB footprint even on long sessions.

### ConfidenceHead for Safe Automation

Needle 2 does not just generate text—it estimates how much it trusts each response. The `ConfidenceHead` class in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) outputs a calibrated confidence score for every answer. Callers can set a threshold and decide whether to accept the output or escalate to a human or larger model, making automated pipelines safer.

## Three Primary Use Cases

The repository demonstrates three main patterns for using Needle 2 in production.

### 1. Tool-Calling with Valid JSON Output

Needle 2 reads tool descriptions through the `@needle.tool` decorator and decides which tool to invoke. The decoder in [`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py) uses a byte-level grammar compiled from the function’s Pydantic schema, so the generated arguments are syntactically correct JSON.

```python
import needle

@needle.tool
def get_weather(city: str):
    """Get the current weather for a city."""
    # In a real app you might call an API here.

    return {"city": city, "temp_c": 27, "sky": "clear"}

agent = needle.Needle(tools=[get_weather])
result = agent.run("What’s the weather like in Lagos right now?")
print(result["results"])

# → [{'city': 'Lagos', 'temp_c': 27, 'sky': 'clear'}]

```

The `@needle.tool` decorator registers the Python function, and the grammar-constrained decoder ensures the model emits JSON that matches the schema.

### 2. Structured Data Extraction

You can extract typed objects from raw text by passing a Pydantic model to `needle.extract`. Under the hood, this uses the same grammar engine that powers tool-calling.

```python
from pydantic import BaseModel
import needle

class Invoice(BaseModel):
    vendor: str
    total: float
    due_date: str

text = "Invoice from Acme Corp, $1,200.00, due 2026-09-01"
invoice = needle.extract(text, Invoice)
print(invoice.vendor, invoice.total)

# → Acme Corp 1200.0

```

### 3. LoRA Fine-Tuning on Custom Tools

Users can adapt the frozen base model to proprietary tool sets using LoRA adapters. The training loop in [`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py) handles data generation, adapter training, and merging back into the final archive.

```bash

# Generate or augment data (optional)

needle generate-data --tools my_tools.json --num-samples 500 --output data.jsonl

# Fine-tune a LoRA adapter on the data

needle finetune data.jsonl --epochs 10 --lora-rank 16 --lora-alpha 32

# Export a tuned, single-file engine

needle build checkpoints/needle2.pkl --lora checkpoints/needle_lora.pkl --out my_needle.cact

```

The `needle build` step merges the LoRA adapter into the base weights and packages everything into one `.cact` file that is ready for local deployment.

## How Needle 2 Achieves a ~28 MiB Footprint

Several mechanisms work together to keep memory usage constant and small:

- **Sliding-window Engram memory** caps the context cache at roughly 256 tokens.
- **CQ2-bit weights** reduce the model storage to a fraction of standard 8-bit or 16-bit formats.
- **Hadamard-MLP and GQA** shrink activation and KV-cache sizes inside the SAN blocks.
- **Self-contained execution** in [`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py) avoids loading external frameworks or downloading assets at runtime.

As a result, the engine runs on CPUs, GPUs, or Apple Silicon without the memory ballooning as conversations grow.

## Summary

- The overall goal of the Needle 2 project is to deliver a **45-million-parameter, self-contained AI** in a single 14 MiB binary that requires no network access.
- Needle 2 supports **tool-calling**, **structured extraction**, and **confidence-gated inference** through a grammar-constrained decoder.
- Core components include the **Simple Attention Network (SAN)**, **Engram memory**, **CQ2-bit quantization**, and the **ConfidenceHead**, all implemented in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py).
- The `.cact` archive format bundles quantized weights, grammar rules, and optional LoRA adapters into one file produced by `needle build`.
- Memory usage stays near **28 MiB** regardless of conversation length, making it suitable for tiny edge devices.

## Frequently Asked Questions

### What is the overall goal of the Needle 2 project?

The overall goal of the Needle 2 project is to create a 45-million-parameter foundation model that fits into a single 14 MiB binary and runs inference locally on devices with about 28 MiB of RAM. It combines tool-calling, structured JSON extraction, and calibrated confidence scoring without requiring network access or separate model files.

### How does Needle 2 guarantee valid JSON tool outputs?

Needle 2 compiles user-provided Pydantic schemas into a byte-level grammar that is enforced during decoding in [`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py). This grammar-constrained generation prevents syntax errors by allowing only schema-compliant tokens, so every tool call produces valid JSON arguments.

### What hardware can run Needle 2?

According to the repository source code, Needle 2 runs on CPUs, GPUs, and Apple Silicon. Its ~28 MiB memory footprint and self-contained `.cact` archive make it suitable for microcontrollers, mobile phones, and other tiny devices that lack dedicated AI accelerators or reliable network connectivity.

### How small is the Needle 2 runtime footprint?

The runtime footprint is approximately **28 MiB** of RAM during inference. This bound is maintained by the Engram sliding-window memory (≈ 256 tokens), CQ2-bit weight decoding, and efficient SAN architecture, meaning memory usage does not grow with conversation length.