# What Is Cactus Compute Needle? A 45M-Parameter Edge AI Model for Tool Calling

> Discover Cactus Compute Needle, a 45M parameter edge AI model for tool calling. This lightweight, 14MB binary runs ultra-fast inference on tiny devices with minimal RAM and no network needed.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: getting-started
- Published: 2026-08-27

---

**Cactus Compute Needle is a 45 million parameter "Simple Attention Network" designed for ultra-lightweight tool-calling and structured extraction on tiny devices, packaged as a single 14 MB binary that runs full inference in ~28 MB RAM with no network access required.**

Needle represents a new class of edge-first language models. According to the `cactus-compute/needle` source code, the entire neural engine ships as a `.cact` file that performs complete inference sessions without cloud dependencies. This makes it ideal for privacy-sensitive applications, offline environments, and resource-constrained hardware.

## Core Architecture of Cactus Compute Needle

Needle's neural engine abandons conventional transformer components in favor of memory-efficient alternatives. The architecture is implemented in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) and centers on several key innovations.

### Simple Attention Network (SAN) Components

- **`TransformerConfig`** – Stores hyperparameters including `d_model=768`, `num_heads=12`, and engram memory settings. Defined at line 58 in [`architecture.py`](https://github.com/cactus-compute/needle/blob/main/architecture.py).

- **`ZCRMSNorm`** – A lightweight RMS normalization layer that adds per-channel scale parameters without the memory overhead of batch normalization.

- **`MultiHeadAttention`** – Standard Q-K-V attention with optional Flash Attention on GPU. Outputs pass through a learned scalar gate before propagation.

- **`Stack`** – Uses `flax.linen.scan` to iterate `Block` layers across model depth, with configurable `scan_unroll` for memory-time tradeoffs.

### Hadamard MLP: Replacing Dense Feed-Forward Layers

The most distinctive architectural choice in Needle is the **`HadamardMLP`** (line 87, [`architecture.py`](https://github.com/cactus-compute/needle/blob/main/architecture.py)). Instead of dense weight matrices, this layer employs:

- A fixed Walsh-Hadamard transform matrix (`_walsh_matrix`)
- Three learned diagonal vectors: `d1`, `d2`, `d3`

This yields **O(n log n)** compute complexity without any dense weight matrices, dramatically reducing parameter count and memory bandwidth.

### Engram: Keyed-Value Memory for Context Retrieval

Needle's **`Engram`** (line 81, [`architecture.py`](https://github.com/cactus-compute/needle/blob/main/architecture.py)) provides a compressed keyed-value memory:

- Stores hashed n-gram embeddings in a small fixed number of slots
- Uses a learned mask for fast retrieval
- Applies a convolutional "tap" to mix recent values into current context

This mechanism extends effective context length without linear KV cache growth.

## Memory-Efficient Design: Hard KV Budgeting

Needle enforces strict memory constraints through adaptive KV-window management:

- **Hard limit:** ~11 MiB (`KV_BUDGET_BYTES`)
- **Minimum window:** 160 tokens (`KV_WINDOW_MIN`)
- **Adaptive sizing:** `kv_budget_window` and `effective_kv_window` functions (lines 104-119 and 144-149, [`architecture.py`](https://github.com/cactus-compute/needle/blob/main/architecture.py)) compute window size from model dimensions

The window is capped by both the minimum and user-specified `max_seq_len`, guaranteeing bounded memory regardless of conversation length.

## Confidence-Gated Outputs

Every Needle response includes a calibrated confidence score from `ConfidenceHead` (line 64, [`architecture.py`](https://github.com/cactus-compute/needle/blob/main/architecture.py)):

- Pools hidden states across the sequence
- Projects to a single logit
- Returns float-32 confidence for downstream thresholding

This enables applications to reject low-confidence tool calls before execution.

## Structured Tool-Calling API

Needle implements a **text-in / JSON-out** contract through Python decorators.

### Registering Tools with `@needle.tool`

```python
import needle

@needle.tool
def get_weather(city: str):
    """Get the current weather for a city."""
    return {"city": city, "temp_c": 27, "sky": "clear"}

agent = needle.Needle(tools=[get_weather])
out = agent.run("What's the weather like in Lagos right now?")
print(out["results"])

# → [{'city': 'Lagos', 'temp_c': 27, 'sky': 'clear'}]

```

The decorator in [`needle/agent/tools.py`](https://github.com/cactus-compute/needle/blob/main/needle/agent/tools.py) introspects function signatures and docstrings. Needle uses this metadata to decide *when* to call tools and *how* to populate arguments. Results appear under the `results` key in the response dictionary.

## Structured Data Extraction

Beyond tool calling, Needle extracts typed objects via Pydantic schemas:

```python
from pydantic import BaseModel
import needle

class Invoice(BaseModel):
    vendor: str
    total: float
    due_date: str

text = "Invoice from Acme Corp, $1,200.00, due 2026-09-01"
invoice = needle.extract(text, Invoice)
print(invoice.vendor, invoice.total)   # → Acme Corp 1200.0

```

The `needle.extract` function runs in specialized extraction mode, parsing unstructured text against the provided schema.

## Fine-Tuning and Custom Model Building

Needle supports LoRA fine-tuning through its CLI interface in [`needle/cli.py`](https://github.com/cactus-compute/needle/blob/main/needle/cli.py):

```bash

# 1️⃣ Generate synthetic training data (optional)

export OPENROUTER_API_KEY=sk-...
needle generate-data --tools my_tools.json --num-samples 500 --output data.jsonl

# 2️⃣ Apply LoRA fine-tuning

needle finetune data.jsonl --epochs 10

# 3️⃣ Build a tuned binary

needle build checkpoints/needle2.pkl --lora checkpoints/needle_lora.pkl --out my_needle.cact

```

The `needle build` command fuses the frozen base weights with LoRA adaptations into a single deployable `.cact` file.

Loading custom weights requires no recompilation:

```python
import needle

agent = needle.Needle(weights="my_needle.cact", tools=[get_weather])
response = agent.run("Dim the living room lights to 30%")
print(response["results"])

```

## Key Source Files in Cactus Compute Needle

| File | Purpose |
|------|---------|
| [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) | Core SAN implementation: attention, Engram, HadamardMLP, KV budgeting |
| [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) | Public API entry point (`Needle` class, decorators, `extract`, `run`) |
| [`needle/cli.py`](https://github.com/cactus-compute/needle/blob/main/needle/cli.py) | Command-line interface for running, fine-tuning, and building models |
| [`needle/agent/tools.py`](https://github.com/cactus-compute/needle/blob/main/needle/agent/tools.py) | Tool registration and introspection utilities |
| [`needle/agent/fetch.py`](https://github.com/cactus-compute/needle/blob/main/needle/agent/fetch.py) | Remote model download from Hugging Face with local caching |
| `tests/` | Comprehensive test suite covering inference, LoRA, data generation, CLI |

## Summary

- **Cactus Compute Needle** is a 45M-parameter edge AI model for tool calling and structured extraction, shipping as a 14 MB binary with ~28 MB RAM usage.
- **HadamardMLP** replaces dense feed-forward layers with O(n log n) Walsh-Hadamard transforms, eliminating dense weight matrices.
- **Engram memory** provides compressed keyed-value retrieval without linear cache growth.
- **Hard KV budget** (~11 MiB) guarantees bounded memory footprint regardless of sequence length.
- **Confidence gating** via `ConfidenceHead` enables threshold-based rejection of uncertain outputs.
- **Python decorator API** (`@needle.tool`) enables zero-configuration tool registration from function signatures.
- **LoRA fine-tuning pipeline** produces custom `.cact` binaries without model recompilation.

## Frequently Asked Questions

### How does Needle achieve such small memory usage compared to other language models?

Needle's memory efficiency stems from three architectural decisions: the **HadamardMLP** eliminates dense feed-forward matrices in favor of fixed transforms with learned diagonals; the **Engram** compresses context into hashed n-gram slots rather than full KV caches; and a **hard KV budget** (11 MiB) with adaptive window sizing enforces strict upper bounds. These are implemented in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) lines 87-119.

### Can Needle run completely offline after initial installation?

Yes. The [`needle/agent/fetch.py`](https://github.com/cactus-compute/needle/blob/main/needle/agent/fetch.py) module downloads the base `.cact` binary from Hugging Face on first use, then caches it locally. All inference runs entirely on-device with no network calls. Custom fine-tuned models built via `needle build` are fully self-contained.

### What types of tools work best with Needle's `@needle.tool` decorator?

Needle excels with deterministic, stateless functions that have clear type annotations. The decorator in [`needle/agent/tools.py`](https://github.com/cactus-compute/needle/blob/main/needle/agent/tools.py) introspects argument types and docstrings to guide the model's calling decisions. Tools with side effects (database writes, hardware control) work well; the confidence head allows applications to gate execution when the model is uncertain.

### Does Needle support GPU acceleration?

Yes. The `MultiHeadAttention` implementation in [`architecture.py`](https://github.com/cactus-compute/needle/blob/main/architecture.py) includes optional Flash Attention paths for compatible GPUs. However, the model is optimized for CPU inference on edge devices, with the HadamardMLP's O(n log n) complexity specifically designed to reduce memory-bandwidth pressure on resource-constrained hardware.