What Is the Overall Goal of the Needle 2 Project? Inside the 45M-Parameter Edge AI Engine

The overall goal of the Needle 2 project is to ship a single 14 MiB binary that bundles a 45-million-parameter foundation model, a lightweight inference engine, and grammar-constrained decoding so the entire system runs on devices with roughly 28 MiB of RAM without any network access.

Needle 2, maintained in the cactus-compute/needle repository, targets tool-calling, structured extraction, and confidence-gated inference on the smallest edge devices. Every design choice—from the custom 2-bit weight format to the sliding-window memory—supports the overall goal of the Needle 2 project: deliver a self-contained AI that generates schema-valid JSON and knows when to escalate uncertain answers.

The 14 MiB Self-Contained Archive

Unlike typical deployments that require separate weight files, runtime libraries, and network endpoints, Needle 2 compresses everything into one .cact archive. According to the repository source code, the needle build command—implemented in needle/model/finetune.py—merges quantized weights, the byte-level grammar, and the inference runtime into a single file. At inference time, needle/model/run.py loads this archive locally, which means no network traffic occurs during decoding.

This archive contains:

• Quantized model weights in a custom CQ2 2-bit format decoded on-the-fly.
• Tool schemas compiled into a byte-level grammar that forces valid JSON output.
• LoRA adapters (optional) that have been merged back into the base weights at export time.

Architectural Highlights

The model architecture lives primarily in needle/model/architecture.py and replaces standard transformer blocks with components optimized for minimal memory use.

Simple Attention Network (SAN)

The backbone of Needle 2 is the Simple Attention Network (SAN), defined in needle/model/architecture.py inside the SimpleAttentionNetwork class. It replaces the conventional feed-forward network with a Hadamard-MLP and uses Grouped-Query Attention (GQA) to reduce key-value cache size. These changes cut memory overhead while preserving the model’s ability to attend across tool descriptions and user prompts.

Engram Memory for Bounded Context

Instead of storing an ever-growing key-value history, Needle 2 uses an Engram memory system—also in needle/model/architecture.py via the Engram class and the engram_kv pipeline. This is a hashed n-gram key-value store that provides a sliding window of roughly 256 tokens. Because the window is bounded, the runtime footprint stays near 28 MiB regardless of conversation length.

CQ2-Bit Quantization Engine

Weights are stored in a custom CQ2 2-bit format and decoded on-the-fly during inference. The quantization helpers in needle/model/quantize.py—invoked through maybe_quant_* utilities—keep the storage tiny and eliminate the need to decompress full float16 or float32 tensors into RAM. This is how Needle 2 maintains its sub-30 MiB footprint even on long sessions.

ConfidenceHead for Safe Automation

Needle 2 does not just generate text—it estimates how much it trusts each response. The ConfidenceHead class in needle/model/architecture.py outputs a calibrated confidence score for every answer. Callers can set a threshold and decide whether to accept the output or escalate to a human or larger model, making automated pipelines safer.

Three Primary Use Cases

The repository demonstrates three main patterns for using Needle 2 in production.

1. Tool-Calling with Valid JSON Output

Needle 2 reads tool descriptions through the @needle.tool decorator and decides which tool to invoke. The decoder in needle/model/run.py uses a byte-level grammar compiled from the function’s Pydantic schema, so the generated arguments are syntactically correct JSON.

import needle

@needle.tool
def get_weather(city: str):
    """Get the current weather for a city."""
    # In a real app you might call an API here.

    return {"city": city, "temp_c": 27, "sky": "clear"}

agent = needle.Needle(tools=[get_weather])
result = agent.run("What’s the weather like in Lagos right now?")
print(result["results"])

# → [{'city': 'Lagos', 'temp_c': 27, 'sky': 'clear'}]

The @needle.tool decorator registers the Python function, and the grammar-constrained decoder ensures the model emits JSON that matches the schema.

2. Structured Data Extraction

You can extract typed objects from raw text by passing a Pydantic model to needle.extract. Under the hood, this uses the same grammar engine that powers tool-calling.

from pydantic import BaseModel
import needle

class Invoice(BaseModel):
    vendor: str
    total: float
    due_date: str

text = "Invoice from Acme Corp, $1,200.00, due 2026-09-01"
invoice = needle.extract(text, Invoice)
print(invoice.vendor, invoice.total)

# → Acme Corp 1200.0

3. LoRA Fine-Tuning on Custom Tools

Users can adapt the frozen base model to proprietary tool sets using LoRA adapters. The training loop in needle/model/finetune.py handles data generation, adapter training, and merging back into the final archive.

# Generate or augment data (optional)

needle generate-data --tools my_tools.json --num-samples 500 --output data.jsonl

# Fine-tune a LoRA adapter on the data

needle finetune data.jsonl --epochs 10 --lora-rank 16 --lora-alpha 32

# Export a tuned, single-file engine

needle build checkpoints/needle2.pkl --lora checkpoints/needle_lora.pkl --out my_needle.cact

The needle build step merges the LoRA adapter into the base weights and packages everything into one .cact file that is ready for local deployment.

How Needle 2 Achieves a ~28 MiB Footprint

Several mechanisms work together to keep memory usage constant and small:

• Sliding-window Engram memory caps the context cache at roughly 256 tokens.
• CQ2-bit weights reduce the model storage to a fraction of standard 8-bit or 16-bit formats.
• Hadamard-MLP and GQA shrink activation and KV-cache sizes inside the SAN blocks.
• Self-contained execution in needle/model/run.py avoids loading external frameworks or downloading assets at runtime.

As a result, the engine runs on CPUs, GPUs, or Apple Silicon without the memory ballooning as conversations grow.

Summary

• The overall goal of the Needle 2 project is to deliver a 45-million-parameter, self-contained AI in a single 14 MiB binary that requires no network access.
• Needle 2 supports tool-calling, structured extraction, and confidence-gated inference through a grammar-constrained decoder.
• Core components include the Simple Attention Network (SAN), Engram memory, CQ2-bit quantization, and the ConfidenceHead, all implemented in needle/model/architecture.py.
• The .cact archive format bundles quantized weights, grammar rules, and optional LoRA adapters into one file produced by needle build.
• Memory usage stays near 28 MiB regardless of conversation length, making it suitable for tiny edge devices.

Frequently Asked Questions

What is the overall goal of the Needle 2 project?

The overall goal of the Needle 2 project is to create a 45-million-parameter foundation model that fits into a single 14 MiB binary and runs inference locally on devices with about 28 MiB of RAM. It combines tool-calling, structured JSON extraction, and calibrated confidence scoring without requiring network access or separate model files.

How does Needle 2 guarantee valid JSON tool outputs?

Needle 2 compiles user-provided Pydantic schemas into a byte-level grammar that is enforced during decoding in needle/model/run.py. This grammar-constrained generation prevents syntax errors by allowing only schema-compliant tokens, so every tool call produces valid JSON arguments.

What hardware can run Needle 2?

According to the repository source code, Needle 2 runs on CPUs, GPUs, and Apple Silicon. Its ~28 MiB memory footprint and self-contained .cact archive make it suitable for microcontrollers, mobile phones, and other tiny devices that lack dedicated AI accelerators or reliable network connectivity.

How small is the Needle 2 runtime footprint?

The runtime footprint is approximately 28 MiB of RAM during inference. This bound is maintained by the Engram sliding-window memory (≈ 256 tokens), CQ2-bit weight decoding, and efficient SAN architecture, meaning memory usage does not grow with conversation length.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}