What Is Cactus Compute Needle? A 45M-Parameter Edge AI Model for Tool Calling

Cactus Compute Needle is a 45 million parameter "Simple Attention Network" designed for ultra-lightweight tool-calling and structured extraction on tiny devices, packaged as a single 14 MB binary that runs full inference in ~28 MB RAM with no network access required.

Needle represents a new class of edge-first language models. According to the cactus-compute/needle source code, the entire neural engine ships as a .cact file that performs complete inference sessions without cloud dependencies. This makes it ideal for privacy-sensitive applications, offline environments, and resource-constrained hardware.

Core Architecture of Cactus Compute Needle

Needle's neural engine abandons conventional transformer components in favor of memory-efficient alternatives. The architecture is implemented in needle/model/architecture.py and centers on several key innovations.

Simple Attention Network (SAN) Components

• TransformerConfig – Stores hyperparameters including d_model=768, num_heads=12, and engram memory settings. Defined at line 58 in architecture.py.

• ZCRMSNorm – A lightweight RMS normalization layer that adds per-channel scale parameters without the memory overhead of batch normalization.

• MultiHeadAttention – Standard Q-K-V attention with optional Flash Attention on GPU. Outputs pass through a learned scalar gate before propagation.

• Stack – Uses flax.linen.scan to iterate Block layers across model depth, with configurable scan_unroll for memory-time tradeoffs.

Hadamard MLP: Replacing Dense Feed-Forward Layers

The most distinctive architectural choice in Needle is the HadamardMLP (line 87, architecture.py). Instead of dense weight matrices, this layer employs:

• A fixed Walsh-Hadamard transform matrix (_walsh_matrix)
• Three learned diagonal vectors: d1, d2, d3

This yields O(n log n) compute complexity without any dense weight matrices, dramatically reducing parameter count and memory bandwidth.

Engram: Keyed-Value Memory for Context Retrieval

Needle's Engram (line 81, architecture.py) provides a compressed keyed-value memory:

• Stores hashed n-gram embeddings in a small fixed number of slots
• Uses a learned mask for fast retrieval
• Applies a convolutional "tap" to mix recent values into current context

This mechanism extends effective context length without linear KV cache growth.

Memory-Efficient Design: Hard KV Budgeting

Needle enforces strict memory constraints through adaptive KV-window management:

• Hard limit: ~11 MiB (KV_BUDGET_BYTES)
• Minimum window: 160 tokens (KV_WINDOW_MIN)
• Adaptive sizing: kv_budget_window and effective_kv_window functions (lines 104-119 and 144-149, architecture.py) compute window size from model dimensions

The window is capped by both the minimum and user-specified max_seq_len, guaranteeing bounded memory regardless of conversation length.

Confidence-Gated Outputs

Every Needle response includes a calibrated confidence score from ConfidenceHead (line 64, architecture.py):

• Pools hidden states across the sequence
• Projects to a single logit
• Returns float-32 confidence for downstream thresholding

This enables applications to reject low-confidence tool calls before execution.

Structured Tool-Calling API

Needle implements a text-in / JSON-out contract through Python decorators.

Registering Tools with @needle.tool

import needle

@needle.tool
def get_weather(city: str):
    """Get the current weather for a city."""
    return {"city": city, "temp_c": 27, "sky": "clear"}

agent = needle.Needle(tools=[get_weather])
out = agent.run("What's the weather like in Lagos right now?")
print(out["results"])

# → [{'city': 'Lagos', 'temp_c': 27, 'sky': 'clear'}]

The decorator in needle/agent/tools.py introspects function signatures and docstrings. Needle uses this metadata to decide when to call tools and how to populate arguments. Results appear under the results key in the response dictionary.

Structured Data Extraction

Beyond tool calling, Needle extracts typed objects via Pydantic schemas:

from pydantic import BaseModel
import needle

class Invoice(BaseModel):
    vendor: str
    total: float
    due_date: str

text = "Invoice from Acme Corp, $1,200.00, due 2026-09-01"
invoice = needle.extract(text, Invoice)
print(invoice.vendor, invoice.total)   # → Acme Corp 1200.0

The needle.extract function runs in specialized extraction mode, parsing unstructured text against the provided schema.

Fine-Tuning and Custom Model Building

Needle supports LoRA fine-tuning through its CLI interface in needle/cli.py:

# 1️⃣ Generate synthetic training data (optional)

export OPENROUTER_API_KEY=sk-...
needle generate-data --tools my_tools.json --num-samples 500 --output data.jsonl

# 2️⃣ Apply LoRA fine-tuning

needle finetune data.jsonl --epochs 10

# 3️⃣ Build a tuned binary

needle build checkpoints/needle2.pkl --lora checkpoints/needle_lora.pkl --out my_needle.cact

The needle build command fuses the frozen base weights with LoRA adaptations into a single deployable .cact file.

Loading custom weights requires no recompilation:

import needle

agent = needle.Needle(weights="my_needle.cact", tools=[get_weather])
response = agent.run("Dim the living room lights to 30%")
print(response["results"])

Key Source Files in Cactus Compute Needle

File	Purpose
needle/model/architecture.py	Core SAN implementation: attention, Engram, HadamardMLP, KV budgeting
needle/__init__.py	Public API entry point (Needle class, decorators, extract, run)
needle/cli.py	Command-line interface for running, fine-tuning, and building models
needle/agent/tools.py	Tool registration and introspection utilities
needle/agent/fetch.py	Remote model download from Hugging Face with local caching
tests/	Comprehensive test suite covering inference, LoRA, data generation, CLI

Summary

• Cactus Compute Needle is a 45M-parameter edge AI model for tool calling and structured extraction, shipping as a 14 MB binary with ~28 MB RAM usage.
• HadamardMLP replaces dense feed-forward layers with O(n log n) Walsh-Hadamard transforms, eliminating dense weight matrices.
• Engram memory provides compressed keyed-value retrieval without linear cache growth.
• Hard KV budget (~11 MiB) guarantees bounded memory footprint regardless of sequence length.
• Confidence gating via ConfidenceHead enables threshold-based rejection of uncertain outputs.
• Python decorator API (@needle.tool) enables zero-configuration tool registration from function signatures.
• LoRA fine-tuning pipeline produces custom .cact binaries without model recompilation.

Frequently Asked Questions

How does Needle achieve such small memory usage compared to other language models?

Needle's memory efficiency stems from three architectural decisions: the HadamardMLP eliminates dense feed-forward matrices in favor of fixed transforms with learned diagonals; the Engram compresses context into hashed n-gram slots rather than full KV caches; and a hard KV budget (11 MiB) with adaptive window sizing enforces strict upper bounds. These are implemented in needle/model/architecture.py lines 87-119.

Can Needle run completely offline after initial installation?

Yes. The needle/agent/fetch.py module downloads the base .cact binary from Hugging Face on first use, then caches it locally. All inference runs entirely on-device with no network calls. Custom fine-tuned models built via needle build are fully self-contained.

What types of tools work best with Needle's @needle.tool decorator?

Needle excels with deterministic, stateless functions that have clear type annotations. The decorator in needle/agent/tools.py introspects argument types and docstrings to guide the model's calling decisions. Tools with side effects (database writes, hardware control) work well; the confidence head allows applications to gate execution when the model is uncertain.

Does Needle support GPU acceleration?

Yes. The MultiHeadAttention implementation in architecture.py includes optional Flash Attention paths for compatible GPUs. However, the model is optimized for CPU inference on edge devices, with the HadamardMLP's O(n log n) complexity specifically designed to reduce memory-bandwidth pressure on resource-constrained hardware.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}