What Is Needle 2? A Deep Dive into the 45M‑Parameter Edge AI Model

Needle 2 is a 45‑million‑parameter foundation model packaged as a single 14 MB binary engine that runs full inference sessions in approximately 28 MB of RAM, designed specifically for tool‑calling, device control, and structured extraction on resource‑constrained hardware.

Built by the team at cactus-compute/needle, this compact architecture challenges the assumption that effective AI assistants require cloud‑scale compute. Needle 2 delivers competitive performance against models 5–70× its size while maintaining a constant memory footprint ideal for mobile phones, micro‑servers, and embedded edge devices.

Core Architecture of Needle 2

The model’s efficiency stems from a Simple Attention Network (SAN)—a dense‑small‑model recipe that reimagines standard transformer components. Unlike conventional architectures that scale linearly with context length, Needle 2 employs specialized layers to bound memory usage while preserving reasoning capabilities.

Simple Attention Network (SAN)

At the heart of the system lies the SimpleAttentionNetwork class defined in needle/model/architecture.py (lines 78‑90). This class orchestrates the embedding layer, stacked transformer blocks, and an optional multi‑token prediction head. The SAN replaces standard feed‑forward networks with lightweight alternatives and integrates Grouped‑Query‑Attention (GQA) to reduce key‑value cache overhead during inference.

Hadamard MLP

Instead of traditional dense feed‑forward layers, Needle 2 uses a Hadamard‑based MLP implemented in the HadamardMLP class (lines 87‑103 of needle/model/architecture.py). This layer performs the feed‑forward step via a Walsh‑Hadamard transform, which operates in O(n log n) time complexity. The approach is weight‑free; the only learned parameters are three diagonal scaling vectors, drastically reducing the parameter count while maintaining representational capacity.

Engram Memory System

To handle long‑form conversations without ballooning RAM usage, Needle 2 incorporates an engram key‑value memory via the Engram class (lines 81‑107). This system stores recent n‑gram information in a fixed‑size hash table (slots) and injects cached context back into the attention stream. Combined with a 256‑token sliding window and pinned “KV sinks,” this guarantees a constant ~28 MB memory footprint regardless of conversation length.

Confidence Gating and Tool Retrieval

Every inference pass produces a calibrated confidence score through the ConfidenceHead (lines 63‑76), allowing applications to set automated thresholds for autonomous operation versus human oversight. For scenarios with large tool catalogs, a dedicated retrieval head selects the top‑5 relevant candidates per turn, constraining the decode grammar to only those tools and shrinking the search space dramatically.

Key Features of Needle 2

Self‑Contained Binary Distribution

All 45 million parameters are baked into a single 14 MB engine file (typically distributed as a .cact checkpoint). As documented in the project README, the system requires no external model downloads, network access, or dependency on cloud APIs, making it ideal for offline or air‑gapped deployments.

Structured Output Contract

Needle 2 operates on a simple I/O contract: free‑form text enters, and JSON‑structured data exits. The engine uses a byte‑level grammar compiled directly from user‑provided Pydantic schemas, guaranteeing syntactically valid tool calls and eliminating the need for post‑processing regex or JSON repair heuristics.

Bounded Memory Guarantees

The architecture enforces a strict 256‑token sliding window with constant memory usage. Whether processing a single turn or a hundred‑turn conversation, RAM consumption remains pinned near 28 MB—critical for microcontrollers and mobile environments where memory fragmentation can crash applications.

Competitive Performance

According to the source documentation, Needle 2 outperforms larger alternatives such as FunctionGemma 270 M, LFM 2.5 230 M, and Apple’s foundation models on tool‑calling benchmarks, despite utilizing 2‑bit quantization rather than standard 16‑bit floating point.

Using Needle 2 in Practice

Tool Calling with Python Decorators

The @needle.tool decorator introspects Python functions to automatically generate JSON schemas. During execution, the model selects, invokes, and returns results from the appropriate tool.

import needle

@needle.tool
def get_weather(city: str):
    """Get the current weather for a city."""
    # In production, this would query a weather API.

    return {"city": city, "temp_c": 27, "sky": "clear"}

agent = needle.Needle(tools=[get_weather])
result = agent.run("what's it like in Lagos right now?")["results"]
print(result)  # [{'city': 'Lagos', 'temp_c': 27, 'sky': 'clear'}]

The needle/cli.py file implements the runtime loop that handles this schema compilation and tool dispatch logic.

Structured Data Extraction

For extraction tasks, needle.extract() parses unstructured text into typed Pydantic models using the same SAN backbone but routing through a specialized extraction head.

from pydantic import BaseModel
import needle

class Invoice(BaseModel):
    vendor: str
    total: float
    due_date: str

text = "Invoice from Acme Corp, $1,200.00, due 2026-09-01"
invoice = needle.extract(text, Invoice)

print(invoice.vendor)  # -> Acme Corp

print(invoice.total)   # -> 1200.0

Fine‑Tuning Custom Checkpoints

The repository includes a JAX‑based LoRA training pipeline defined in needle/model/finetune.py. Users can generate synthetic training data, fine‑tune the base model, and export a tuned binary.

# Generate synthetic training data

export OPENROUTER_API_KEY=sk-...
needle generate-data --tools my_tools.json --num-samples 500 --output data.jsonl

# LoRA fine-tune

needle finetune data.jsonl --epochs 10 --lora-rank 16 --lora-alpha 32

# Export tuned engine

needle build checkpoints/needle2.pkl \
  --lora checkpoints/needle_lora.pkl \
  --out my_needle.cact

Loading Custom Weights

Tuned .cact files load directly into the inference engine without additional conversion.

import needle

agent = needle.Needle(weights="my_needle.cact", tools=[get_weather])
response = agent.run("dim the living room lights to 30%")
print(response["results"])

Summary

• Needle 2 is a 45 M‑parameter model shipping as a 14 MB binary that runs in ~28 MB RAM.
• It uses a Simple Attention Network with Hadamard MLPs and engram memory to achieve constant memory usage.
• Features include confidence‑gated outputs, tool retrieval heads, and 2‑bit quantization.
• The API supports tool calling, structured extraction, and LoRA fine‑tuning via the Python SDK or CLI.
• Source code for the architecture lives in needle/model/architecture.py, while user interaction is handled through needle/cli.py.

Frequently Asked Questions

How does Needle 2 maintain constant memory usage during long conversations?

Needle 2 implements a 256‑token sliding window combined with an engram memory system (defined in needle/model/architecture.py lines 81‑107). The engram stores recent n‑gram information in a fixed‑size hash table, while “KV sinks” pin critical key‑value entries. This design caps memory at approximately 28 MB regardless of conversation length, unlike standard transformers whose KV‑cache grows linearly with context.

What hardware can run Needle 2 effectively?

The model targets very small hardware including mobile phones, Raspberry Pi devices, and micro‑servers. The 14 MB binary footprint and 28 MB RAM requirement allow it to operate on devices with limited storage and memory, such as older smartphones or embedded Linux systems, without GPU acceleration.

Can I fine‑tune Needle 2 on domain‑specific tools?

Yes. The repository provides a complete fine‑tuning pipeline via needle finetune. You can generate synthetic data using the generate-data command, apply LoRA updates with configurable rank and alpha parameters, and export a custom .cact checkpoint. This fine‑tuned engine retains the original 14 MB size class while specializing on your specific tool schemas.

How does the confidence scoring work?

Every response includes a calibrated confidence score produced by the ConfidenceHead class (lines 63‑76 in needle/model/architecture.py). This allows developers to set programmatic thresholds—for example, rejecting outputs below 0.95 confidence for automated financial transactions while accepting lower thresholds for casual queries.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}