# Needle Advanced Features: A Deep Dive into the Production-Ready Edge AI Engine

> Explore Needle's advanced features like grammar constrained JSON output, confidence gated tool calls, and LoRA fine tuning. Optimize edge AI with this production-ready engine.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: deep-dive
- Published: 2026-08-27

---

**Needle packs a 14 MB self‑contained binary, grammar‑constrained JSON output, confidence‑gated tool calls, and LoRA fine‑tuning into a single package designed for offline, tiny‑device deployment.**

Needle is a foundation model engineered specifically for **tool‑calling and structured extraction** on resource‑constrained hardware. Unlike cloud‑dependent LLMs, Needle ships as a standalone `.cact` file that runs inference without network access. This article explores every advanced feature in the `cactus-compute/needle` codebase, citing exact source locations and providing runnable code examples you can use immediately.

---

## Self‑Contained Binary Engine

Needle distributes as a **14 MB `.cact` file** containing quantized weights, tokenizer, and inference runtime. No Docker images, no pip dependency hell, and no API keys after initial fetch.

```python
import needle

# Load once, run forever offline

agent = needle.Needle(weights="needle2.cact", tools=[my_tool])
result = agent.run("Schedule a meeting tomorrow at 3pm")

```

The README describes this design at lines 3‑15. The engine binary caches under `~/.cache/cactus-needle/` and subsequent runs require zero network traffic, as documented in [`doc/apis.md`](https://github.com/cactus-compute/needle/blob/main/doc/apis.md#L156)‑165).

---

## Grammar‑Constrained JSON Output

Needle never emits free‑form text. Every turn returns a **strictly validated JSON object** describing tool calls or extraction results. This eliminates parsing failures and injection vulnerabilities.

The contract is defined in [`doc/apis.md`](https://github.com/cactus-compute/needle/blob/main/doc/apis.md#L82)‑90):

```json
{
  "confidence": 0.97,
  "tool_calls": [{"name": "send_email", "args": {"to": "user@example.com"}}],
  "results": [...]
}

```

This guarantee holds because the decode grammar compiles directly from your tool schemas.

---

## Confidence‑Gated Responses

Each reply carries a **calibrated confidence score**. Calls below your threshold can trigger human review or fallback logic.

Implementation lives in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) at the `ConfidenceHead` (lines 62‑66). The runtime contract specifying thresholds appears in [`doc/apis.md`](https://github.com/cactus-compute/needle/blob/main/doc/apis.md#L150)‑155).

```python
agent = needle.Needle(tools=[transfer_funds], confidence_threshold=0.85)

resp = agent.run("Pay @bob $500")
if resp["confidence"] < 0.85:
    queue_for_human_review(resp)  # Never auto‑execute uncertain calls

```

---

## Tool Retrieval for Large Catalogues

Supply Needle with **200+ tools** without blowing the context window. A contrastive embedding head in [`architecture.py`](https://github.com/cactus-compute/needle/blob/main/architecture.py#L43)‑49) scores tool relevance and materializes only the **top‑5 matches** into the decode grammar per turn.

```python
agent = needle.Needle(
    tools="massive_catalog.json",      # 200+ tool schemas

    tool_index_path="catalog.embeds"   # Cached embeddings for fast reload

)

resp = agent.run("Dim the living room lights")  # Only lighting tools enter context

```

See [`doc/apis.md`](https://github.com/cactus-compute/needle/blob/main/doc/apis.md#L146)‑149) for the retrieval specification.

---

## Bounded Memory with KV‑Budget Management

Needle guarantees **~28 MiB memory footprint** regardless of conversation length. The trick: a **256‑token sliding window** plus KV‑memory pins that lock tool schemas as immutable "sinks."

The logic resides in [`architecture.py`](https://github.com/cactus-compute/needle/blob/main/architecture.py#L99)‑113). Tool schemas get pinned; dynamic conversation tokens rotate through the sliding window. No OOM, ever.

---

## Simple Attention Network (SAN) Architecture

Needle replaces standard transformer blocks with a **Simple Attention Network** featuring:

- **Hadamard‑MLP** instead of feed‑forward layers
- **GQA‑style multi‑head attention** for cache efficiency
- **Engram key‑value memory** for long‑range patterns
- **Multi‑lane hyper‑connections** for gradient flow

The core `SimpleAttentionNetwork` class lives in [`architecture.py`](https://github.com/cactus-compute/needle/blob/main/architecture.py#L78)‑86), with the Hadamard‑MLP at lines 87‑103.

---

## Engram KV Memory for "Sticky" Recall

Traditional KV caches forget beyond the window. Needle's **Engram memory** (lines 81‑88 in [`architecture.py`](https://github.com/cactus-compute/needle/blob/main/architecture.py)) uses **learned n‑gram tables** to preserve critical patterns—dates, names, IDs—without consuming KV budget.

This gives the model apparent long‑range memory while keeping the constant‑memory guarantee from the KV‑budget calculations at lines 99‑112.

---

## 2‑Bit CQ2 Quantization

Base checkpoints store in **2‑bit CQ2 format**. Optional higher‑bit exports trade size for precision. The quantization utilities live in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py), invoked during the build pipeline.

Typical compression ratios:

- **2‑bit**: 14 MB (default, mobile‑optimized)
- **4‑bit**: ~28 MB (desktop/server)
- **8‑bit FP8**: ~56 MB (maximum accuracy)

---

## LoRA Fine‑Tuning Pipeline

Adapt Needle to your domain without retraining the full 1.2B parameters. The LoRA workflow:

1. **Generate synthetic data** (optional, requires `OPENROUTER_API_KEY`)
2. **Train adapter** on frozen base
3. **Merge and export** to single `.cact`

```bash

# Step 1: Generate training data

needle generate-data --tools my_tools.json --num-samples 500 -o data.jsonl

# Step 2: LoRA fine‑tune

needle finetune data.jsonl --epochs 10 --lora-rank 16 --lora-alpha 32

# Step 3: Build deployable artifact

needle build checkpoints/needle2.pkl \
        --lora checkpoints/needle_lora.pkl \
        --out my_domain_needle.cact

```

The pipeline is documented in [`doc/finetuning.md`](https://github.com/cactus-compute/needle/blob/main/doc/finetuning.md#L77)‑102); LoRA implementation lives in [`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py).

---

## Extraction Mode for Structured Data

Treat any **Pydantic model as a single "tool"** and extract structured data from unstructured text. Same confidence gating, same grammar guarantees as tool calling.

```python
from pydantic import BaseModel

class WeatherReport(BaseModel):
    location: str
    temperature_c: float
    conditions: str

text = "It's sunny and 23 degrees in Seattle today"
report = needle.extract(text, WeatherReport)

# → WeatherReport(location='Seattle', temperature_c=23.0, conditions='sunny')

```

See [`doc/apis.md`](https://github.com/cactus-compute/needle/blob/main/doc/apis.md#L110)‑135) for extraction semantics.

---

## Declarative Schema Constraints with `needle.Field`

Define valid argument ranges directly in Python type hints. Constraints compile into the decode grammar, **guaranteeing only valid outputs**.

Implementation in [`needle/agent/tools.py`](https://github.com/cactus-compute/needle/blob/main/needle/agent/tools.py#L18)‑33):

```python
from typing import Annotated
from needle.agent.tools import Field

@needle.tool
def book_flight(
    passengers: Annotated[int, Field(ge=1, le=9)],
    origin: Annotated[str, Field(pattern=r"^[A-Z]{3}$")],  # IATA code

    departure_date: Annotated[str, Field(pattern=r"^\d{4}-\d{2}-\d{2}$")],
    max_price: Annotated[float, Field(gt=0)] = 5000.0,
):
    """Book a flight with validated constraints."""
    ...

```

Invalid arguments are **syntactically impossible** to generate.

---

## System Facts Turn for Context Awareness

A **dedicated, read‑only turn** supplies environment context—date, locale, device state, battery level—without steering the model. The model uses this for **relative time resolution** ("tomorrow", "next Tuesday") without hallucinating assumptions.

Configured in [`doc/apis.md`](https://github.com/cactus-compute/needle/blob/main/doc/apis.md#L136)‑144):

```python
agent = needle.Needle(
    tools=[schedule_meeting],
    system_facts={
        "current_date": "2024-01-15",
        "timezone": "America/Los_Angeles",
        "device_battery_pct": 67
    }
)

```

---

## Offline‑First Deployment

Needle is engineered for **air‑gapped environments**. Fetch the engine binary once, transfer to target, run forever without network.

```bash

# Connected machine: download engine

needle fetch --out ./engine

# Transfer ./engine + .cact file to offline device

# Target device (no internet):

HF_HUB_OFFLINE=1 python -c "
import needle
agent = needle.Needle(weights='my_needle.cact', tools=[...])
print(agent.run('What is the weather?'))
"

```

Offline guarantees are specified in [`doc/apis.md`](https://github.com/cactus-compute/needle/blob/main/doc/apis.md#L156)‑165).

---

## Cross‑Platform Engine Distribution

Pre‑compiled binaries available for **Linux, macOS, Windows, and Musl**. The CLI auto‑detects platform and downloads the appropriate runner from Hugging Face.

```bash
needle fetch  # Automatically selects: linux-x86_64, darwin-arm64, etc.

```

See README (L70‑74) and [`needle/cli.py`](https://github.com/cactus-compute/needle/blob/main/needle/cli.py) for platform detection logic.

---

## Summary

- **Self‑contained**: 14 MB `.cact` with no runtime dependencies
- **Correct by construction**: Grammar‑constrained JSON, confidence gating, schema validation via `needle.Field`
- **Scalable**: Tool retrieval handles 200+ tools, bounded memory stays at ~28 MiB
- **Adaptable**: LoRA fine‑tuning, extraction mode, system facts for domain customization
- **Deployable anywhere**: Offline‑first, cross‑platform, 2‑bit quantized

---

## Frequently Asked Questions

### What makes Needle different from other tool‑calling LLMs?

Needle runs **entirely on‑device** in a 14 MB binary with no API calls after initial setup. Its grammar‑constrained output guarantees valid JSON on every turn, while confidence scoring lets you gate uncertain decisions. Most cloud LLMs require network access and probabilistic parsing of free‑form text.

### How does Needle handle long conversations without running out of memory?

A **256‑token sliding window** with **KV‑memory pins** maintains constant ~28 MiB footprint. Tool schemas get pinned as immutable sinks; conversation tokens rotate through the sliding window. The **Engram memory** (learned n‑gram tables) preserves critical patterns beyond the window without consuming KV budget. See [`architecture.py`](https://github.com/cactus-compute/needle/blob/main/architecture.py#L99)‑113).

### Can I fine‑tune Needle on my proprietary tools?

Yes. The **LoRA pipeline** in [`doc/finetuning.md`](https://github.com/cactus-compute/needle/blob/main/doc/finetuning.md) lets you train lightweight adapters on frozen base weights. Synthetic data generation, adapter training, and merging into a single `.cact` file are all supported via CLI commands. The implementation lives in [`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py).

### Does Needle work completely offline?

Yes. After running `needle fetch` once on a connected machine, both the engine binary and `.cact` weights transfer to air‑gapped devices. Set `HF_HUB_OFFLINE=1` and inference runs with zero network access. This is guaranteed by design per [`doc/apis.md`](https://github.com/cactus-compute/needle/blob/main/doc/apis.md#L156)‑165).