Needle Advanced Features: A Deep Dive into the Production-Ready Edge AI Engine

Needle packs a 14 MB self‑contained binary, grammar‑constrained JSON output, confidence‑gated tool calls, and LoRA fine‑tuning into a single package designed for offline, tiny‑device deployment.

Needle is a foundation model engineered specifically for tool‑calling and structured extraction on resource‑constrained hardware. Unlike cloud‑dependent LLMs, Needle ships as a standalone .cact file that runs inference without network access. This article explores every advanced feature in the cactus-compute/needle codebase, citing exact source locations and providing runnable code examples you can use immediately.

---

Self‑Contained Binary Engine

Needle distributes as a 14 MB .cact file containing quantized weights, tokenizer, and inference runtime. No Docker images, no pip dependency hell, and no API keys after initial fetch.

import needle

# Load once, run forever offline

agent = needle.Needle(weights="needle2.cact", tools=[my_tool])
result = agent.run("Schedule a meeting tomorrow at 3pm")

The README describes this design at lines 3‑15. The engine binary caches under ~/.cache/cactus-needle/ and subsequent runs require zero network traffic, as documented in doc/apis.md‑165).

---

Grammar‑Constrained JSON Output

Needle never emits free‑form text. Every turn returns a strictly validated JSON object describing tool calls or extraction results. This eliminates parsing failures and injection vulnerabilities.

The contract is defined in doc/apis.md‑90):

{
  "confidence": 0.97,
  "tool_calls": [{"name": "send_email", "args": {"to": "user@example.com"}}],
  "results": [...]
}

This guarantee holds because the decode grammar compiles directly from your tool schemas.

---

Confidence‑Gated Responses

Each reply carries a calibrated confidence score. Calls below your threshold can trigger human review or fallback logic.

Implementation lives in needle/model/architecture.py at the ConfidenceHead (lines 62‑66). The runtime contract specifying thresholds appears in doc/apis.md‑155).

agent = needle.Needle(tools=[transfer_funds], confidence_threshold=0.85)

resp = agent.run("Pay @bob $500")
if resp["confidence"] < 0.85:
    queue_for_human_review(resp)  # Never auto‑execute uncertain calls

---

Tool Retrieval for Large Catalogues

Supply Needle with 200+ tools without blowing the context window. A contrastive embedding head in architecture.py‑49) scores tool relevance and materializes only the top‑5 matches into the decode grammar per turn.

agent = needle.Needle(
    tools="massive_catalog.json",      # 200+ tool schemas

    tool_index_path="catalog.embeds"   # Cached embeddings for fast reload

)

resp = agent.run("Dim the living room lights")  # Only lighting tools enter context

See doc/apis.md‑149) for the retrieval specification.

---

Bounded Memory with KV‑Budget Management

Needle guarantees ~28 MiB memory footprint regardless of conversation length. The trick: a 256‑token sliding window plus KV‑memory pins that lock tool schemas as immutable "sinks."

The logic resides in architecture.py‑113). Tool schemas get pinned; dynamic conversation tokens rotate through the sliding window. No OOM, ever.

---

Simple Attention Network (SAN) Architecture

Needle replaces standard transformer blocks with a Simple Attention Network featuring:

• Hadamard‑MLP instead of feed‑forward layers
• GQA‑style multi‑head attention for cache efficiency
• Engram key‑value memory for long‑range patterns
• Multi‑lane hyper‑connections for gradient flow

The core SimpleAttentionNetwork class lives in architecture.py‑86), with the Hadamard‑MLP at lines 87‑103.

---

Engram KV Memory for "Sticky" Recall

Traditional KV caches forget beyond the window. Needle's Engram memory (lines 81‑88 in architecture.py) uses learned n‑gram tables to preserve critical patterns—dates, names, IDs—without consuming KV budget.

This gives the model apparent long‑range memory while keeping the constant‑memory guarantee from the KV‑budget calculations at lines 99‑112.

---

2‑Bit CQ2 Quantization

Base checkpoints store in 2‑bit CQ2 format. Optional higher‑bit exports trade size for precision. The quantization utilities live in needle/model/quantize.py, invoked during the build pipeline.

Typical compression ratios:

• 2‑bit: 14 MB (default, mobile‑optimized)
• 4‑bit: ~28 MB (desktop/server)
• 8‑bit FP8: ~56 MB (maximum accuracy)

---

LoRA Fine‑Tuning Pipeline

Adapt Needle to your domain without retraining the full 1.2B parameters. The LoRA workflow:

1. Generate synthetic data (optional, requires OPENROUTER_API_KEY)
2. Train adapter on frozen base
3. Merge and export to single .cact

# Step 1: Generate training data

needle generate-data --tools my_tools.json --num-samples 500 -o data.jsonl

# Step 2: LoRA fine‑tune

needle finetune data.jsonl --epochs 10 --lora-rank 16 --lora-alpha 32

# Step 3: Build deployable artifact

needle build checkpoints/needle2.pkl \
        --lora checkpoints/needle_lora.pkl \
        --out my_domain_needle.cact

The pipeline is documented in doc/finetuning.md‑102); LoRA implementation lives in needle/model/finetune.py.

---

Extraction Mode for Structured Data

Treat any Pydantic model as a single "tool" and extract structured data from unstructured text. Same confidence gating, same grammar guarantees as tool calling.

from pydantic import BaseModel

class WeatherReport(BaseModel):
    location: str
    temperature_c: float
    conditions: str

text = "It's sunny and 23 degrees in Seattle today"
report = needle.extract(text, WeatherReport)

# → WeatherReport(location='Seattle', temperature_c=23.0, conditions='sunny')

See doc/apis.md‑135) for extraction semantics.

---

Declarative Schema Constraints with needle.Field

Define valid argument ranges directly in Python type hints. Constraints compile into the decode grammar, guaranteeing only valid outputs.

Implementation in needle/agent/tools.py‑33):

from typing import Annotated
from needle.agent.tools import Field

@needle.tool
def book_flight(
    passengers: Annotated[int, Field(ge=1, le=9)],
    origin: Annotated[str, Field(pattern=r"^[A-Z]{3}$")],  # IATA code

    departure_date: Annotated[str, Field(pattern=r"^\d{4}-\d{2}-\d{2}$")],
    max_price: Annotated[float, Field(gt=0)] = 5000.0,
):
    """Book a flight with validated constraints."""
    ...

Invalid arguments are syntactically impossible to generate.

---

System Facts Turn for Context Awareness

A dedicated, read‑only turn supplies environment context—date, locale, device state, battery level—without steering the model. The model uses this for relative time resolution ("tomorrow", "next Tuesday") without hallucinating assumptions.

Configured in doc/apis.md‑144):

agent = needle.Needle(
    tools=[schedule_meeting],
    system_facts={
        "current_date": "2024-01-15",
        "timezone": "America/Los_Angeles",
        "device_battery_pct": 67
    }
)

---

Offline‑First Deployment

Needle is engineered for air‑gapped environments. Fetch the engine binary once, transfer to target, run forever without network.

# Connected machine: download engine

needle fetch --out ./engine

# Transfer ./engine + .cact file to offline device

# Target device (no internet):

HF_HUB_OFFLINE=1 python -c "
import needle
agent = needle.Needle(weights='my_needle.cact', tools=[...])
print(agent.run('What is the weather?'))
"

Offline guarantees are specified in doc/apis.md‑165).

---

Cross‑Platform Engine Distribution

Pre‑compiled binaries available for Linux, macOS, Windows, and Musl. The CLI auto‑detects platform and downloads the appropriate runner from Hugging Face.

needle fetch  # Automatically selects: linux-x86_64, darwin-arm64, etc.

See README (L70‑74) and needle/cli.py for platform detection logic.

---

Summary

• Self‑contained: 14 MB .cact with no runtime dependencies
• Correct by construction: Grammar‑constrained JSON, confidence gating, schema validation via needle.Field
• Scalable: Tool retrieval handles 200+ tools, bounded memory stays at ~28 MiB
• Adaptable: LoRA fine‑tuning, extraction mode, system facts for domain customization
• Deployable anywhere: Offline‑first, cross‑platform, 2‑bit quantized

---

Frequently Asked Questions

What makes Needle different from other tool‑calling LLMs?

Needle runs entirely on‑device in a 14 MB binary with no API calls after initial setup. Its grammar‑constrained output guarantees valid JSON on every turn, while confidence scoring lets you gate uncertain decisions. Most cloud LLMs require network access and probabilistic parsing of free‑form text.

How does Needle handle long conversations without running out of memory?

A 256‑token sliding window with KV‑memory pins maintains constant ~28 MiB footprint. Tool schemas get pinned as immutable sinks; conversation tokens rotate through the sliding window. The Engram memory (learned n‑gram tables) preserves critical patterns beyond the window without consuming KV budget. See architecture.py‑113).

Can I fine‑tune Needle on my proprietary tools?

Yes. The LoRA pipeline in doc/finetuning.md lets you train lightweight adapters on frozen base weights. Synthetic data generation, adapter training, and merging into a single .cact file are all supported via CLI commands. The implementation lives in needle/model/finetune.py.

Does Needle work completely offline?

Yes. After running needle fetch once on a connected machine, both the engine binary and .cact weights transfer to air‑gapped devices. Set HF_HUB_OFFLINE=1 and inference runs with zero network access. This is guaranteed by design per doc/apis.md‑165).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}