Primary Use Cases for the Needle 2 Foundation Model: Edge Deployment and Structured Tool Calling

TLDR: Needle 2 is a 45M-parameter foundation model packaged as a 14MB binary that enables grammar-constrained tool calling, schema-validated data extraction, and rapid LoRA fine-tuning on devices with only 28MB of RAM.

Needle 2 implements a "Simple Attention Network" architecture that trades model scale for deterministic, offline-capable inference. According to the cactus-compute/needle repository, the entire engine is self-contained in a single .cact file, making it deployable to microcontrollers, air-gapped servers, and mobile phones without network dependencies.

Tool-Calling Agents on Constrained and Offline Devices

Needle 2 targets edge AI agents that must execute structured tool calls without cloud connectivity. The model restricts output to JSON-only grammar via constrained decoding, eliminating free-text hallucinations through built-in confidence gating. With a peak memory footprint of just 28MB and a 14MB binary size, the runtime performs inference entirely within the cache of tiny devices.

The implementation in needle/__init__.py exposes a Needle class that accepts Python functions decorated with @needle.tool, transforming them into JSON schemas via needle/agent/tools.py. As documented in doc/apis.md, the engine is fetched once and cached, meaning "inference itself never touches the network" after initial load.

When using pre-built domain surfaces, developers can import ready-made environments from needle/environments/*.py:

from needle.environments import smart_home

smart_home.agent.complete("dim the study lights to 30 percent")
smart_home.run_tests()      # sanity-check the environment suite

For custom agents, the decorator compiles function signatures into deterministic call formats:

import needle

@needle.tool
def get_weather(city: str):
    """Get the current weather for a city."""
    return {"city": city, "temp_c": 27, "sky": "clear"}

agent = needle.Needle(tools=[get_weather])
print(agent.run("what's it like in Lagos right now?")["results"])

# → [{'city': 'Lagos', 'temp_c': 27, 'sky': 'clear'}]

Structured Data Extraction via Single-Tool Calls

Needle 2 treats data extraction as a specialized case of tool calling rather than a separate pipeline. By declaring a Pydantic model as the sole available tool, the model reduces extraction to a single, grammar-constrained function call that guarantees well-typed outputs. This approach eliminates post-processing regex or validation logic typically required with unstructured generation.

As stated in doc/apis.md, "Extraction is not a separate mode … it is tool calling with one tool." The needle.extract function handles the schema wrapping automatically:

from pydantic import BaseModel
import needle

class Invoice(BaseModel):
    vendor: str
    total: float
    due_date: str

text = "Invoice from Acme Corp, $1,200.00, due 2026-09-01"
invoice = needle.extract(text, Invoice)
print(invoice.vendor, invoice.total)   # → Acme Corp 1200.0

Rapid Prototyping and Domain-Specific Fine-Tuning

The third primary use case involves customizing the model for specialized tool catalogs without increasing deployment size. Needle 2 supports LoRA-based fine-tuning where the adapted weights are stored as a separate .cact archive while the base engine remains unchanged. Both the base and tuned models run on the identical binary runtime, enabling on-device customization without recompilation.

According to README.md, "Needle fine‑tunes with LoRA … the tuned model is still a single .cact." This allows teams to distribute fine-tuned variants for specific domains—such as medical coding or industrial sensor interpretation—while keeping the 14MB footprint constant:

import needle

agent = needle.Needle(
    weights="my_needle.cact",   # a LoRA‑tuned .cact file

    tools=[...],                # your domain‑specific tools

)
agent.run("perform the task defined by my tools")

The needle/cli.py module provides command-line utilities (needle finetune, needle download) to manage these archives.

Simple Attention Network Architecture

Underlying these capabilities is the Simple Attention Network design implemented in needle/model/architecture.py. This architecture replaces traditional feed-forward layers with Hadamard-MLP blocks, employs grouped-query attention to reduce KV-cache size, and introduces an "engram" KV memory system that periodically hashes activations into low-bit slots.

These innovations allow Needle 2 to quantize down to 2-bit precision while preserving generation quality. The compact representation—45M parameters in 14MB—explains why the model achieves "5× to 70× smaller" size compared to alternatives while maintaining functional accuracy for tool-driven workflows.

Summary

• Offline tool execution: Needle 2 runs grammar-constrained agents on 28MB RAM without network calls, using JSON-only output modes defined in doc/apis.md.
• Type-safe extraction: Schema validation is intrinsic to the generation process via Pydantic integration in needle/__init__.py, not a post-processing layer.
• Zero-bloat customization: LoRA fine-tuning produces new .cact files without altering the base engine binary, enabling rapid domain adaptation via needle/cli.py.
• Extreme compression: The Hadamard-MLP architecture and 2-bit quantization in needle/model/architecture.py deliver foundation-model capabilities at embedded-system scale.

Frequently Asked Questions

What hardware can run Needle 2 inference?

Devices with approximately 28MB of available RAM can execute Needle 2 inference, including smartphones, microcontrollers, and air-gapped edge servers. The entire 45M-parameter model is self-contained in a 14MB binary that requires no external downloads during inference, as implemented in the caching logic of doc/apis.md.

How does Needle 2 guarantee valid JSON output during tool calling?

The model uses grammar-constrained decoding that restricts the output vocabulary to valid JSON tokens only, combined with confidence gating that prevents fallback to free text. This deterministic approach is handled by the core generation loop in needle/model/architecture.py, ensuring that calls to @needle.tool decorated functions receive parseable arguments.

Can I fine-tune Needle 2 without increasing the deployment size?

Yes. Needle 2 utilizes LoRA (Low-Rank Adaptation) fine-tuning, which stores adapter weights in a separate .cact archive while reusing the original 14MB base engine. The needle/cli.py utilities manage this process, allowing the same runtime binary to load either base or fine-tuned weights without recompilation or binary bloat.

What makes the Simple Attention Network different from standard transformers?

The architecture replaces feed-forward layers with Hadamard-MLP blocks, uses grouped-query attention to compress the KV cache, and implements an "engram" memory system that hashes key-value states into low-bit representations. These modifications—found in needle/model/architecture.py—enable aggressive 2-bit quantization while maintaining the model's ability to perform structured generation tasks.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}