What Is the Needle 2 Language Model? A 45M Parameter On-Device Foundation Model

Needle 2 is a 45 million-parameter, on-device foundation model designed for tool-calling, device control, and structured data extraction, shipped as a 14 MB binary that runs in approximately 28 MB of RAM.

Needle 2 represents a new class of edge-optimized transformers developed by Cactus Compute that enables autonomous AI agents without cloud dependency. Unlike traditional large language models requiring substantial GPU resources, this open-source architecture fits entirely on consumer devices such as smartphones, wearables, and embedded edge servers.

Core Architecture: The Simple Attention Network

At the heart of the Needle 2 language model lies the SimpleAttentionNetwork class implemented in needle/model/architecture.py. This backbone replaces conventional transformer components with specialized efficiency layers that reduce computational overhead while maintaining reasoning capabilities.

HadamardMLP and Walsh-Hadamard Transforms

The HadamardMLP class eliminates traditional dense matrix operations by computing H·x using pre-computed Walsh-Hadamard transforms. This matrix-free approach drastically reduces both parameter count and FLOPs compared to standard feed-forward networks. The implementation appears in needle/model/architecture.py alongside the attention mechanisms.

Grouped-Query Attention with Flash Support

Attention is handled by the MultiHeadAttention class, which automatically utilizes flash-attention when GPU backends are available. On CPU-only devices, it falls back to a classic implementation featuring optional KV-compression to minimize memory bandwidth. The design employs Grouped-Query Attention (GQA) to share key and value heads across query heads, further reducing cache size.

Engram Memory for Extended Context

The Engram class provides a learned key-value cache that stores compressed token histories via hash-based tables. This system enables Needle 2 to maintain longer context windows without the linear memory growth typical of standard transformers, keeping the working set within the ≈28 MB RAM budget.

Quantization and Native Engine

Model weights are stored in CQ2-bit format (Cactus Quants) and lazily loaded by the inference engine. The quantization logic resides in needle/model/quantize.py, handling both weight storage and optional activation quantization to minimize data movement during inference.

The high-performance runtime is a compiled binary (approximately 14 MB) auto-downloaded from Hugging Face on first use. In needle/__init__.py, the _lib() function manages engine initialization via ctypes, binding the Python API to the native code and loading the CQ2-bit weights into memory.

Tool-Calling API and Python Interface

The public interface centers on the Needle class defined in needle/__init__.py. This thin wrapper accepts tool definitions—either Python functions decorated with @tool, Pydantic models, or raw JSON schemas—and orchestrates the inference loop.

When run() is invoked, the engine tokenizes input, executes a forward pass through SimpleAttentionNetwork, and emits JSON-encoded tool calls. The Python wrapper resolves these calls, executes the corresponding functions, and feeds results back into the model via the complete method until the model returns a final response or reaches the configured max_steps limit.

Structured Data Extraction

Beyond agentic tool use, the extract() function enables zero-shot structured extraction. By passing text and a Pydantic model, developers can parse unstructured content into typed objects without fine-tuning, leveraging the model's built-in JSON generation capabilities.

Practical Implementation Examples

The following examples demonstrate common usage patterns with the Needle 2 language model.

Basic Tool-Calling Agent

from needle import Needle, tool

@tool
def get_weather(city: str) -> str:
    """Return a short weather forecast for *city*."""
    return f"The weather in {city} is sunny."

# Instantiate the model (downloads binary on first use)

agent = Needle(tools=[get_weather])

# Run query - model automatically calls get_weather

response = agent.run("What will the weather be like in Paris tomorrow?")
print(response["results"][0])   # → "The weather in Paris is sunny."

print(response["text"])         # Final natural-language answer

Structured Extraction with Pydantic

from pydantic import BaseModel
from needle import extract

class FlightInfo(BaseModel):
    airline: str
    flight_number: str
    departure: str
    arrival: str

text = "I booked flight AA123 from JFK to LAX tomorrow morning."
info = extract(text, FlightInfo)
print(info)  # → FlightInfo(airline='AA', flight_number='123', ...)

Loading Fine-Tuned Checkpoints

# Load a LoRA-fine-tuned checkpoint (.cact archive)

agent = Needle(tools=[get_weather], weights="my_finetuned.cact")
response = agent.run("Is it raining in Berlin?")

Command-Line Interface

The package includes a CLI entry point in needle/cli.py supporting direct inference and fine-tuning workflows:

# Run inference from command line

needle run --prompt "What is the capital of France?"

# Fine-tune on custom data (produces .cact archive)

needle finetune --data training.jsonl --output my_model.cact

Summary

• Needle 2 is a 45M-parameter transformer optimized for on-device inference, requiring only ~28 MB RAM at runtime.
• The Simple Attention Network in needle/model/architecture.py uses HadamardMLP, GQA, and Engram memory to achieve efficiency.
• CQ2-bit quantization in needle/model/quantize.py compresses weights to enable the 14 MB binary distribution.
• The Python API via needle/__init__.py provides seamless tool-calling with the Needle class and @tool decorator.
• Native engine binding through ctypes delivers high-performance inference while maintaining a clean Python interface.

Frequently Asked Questions

What makes Needle 2 different from other compact language models?

Needle 2 distinguishes itself through the HadamardMLP architecture using Walsh-Hadamard transforms instead of dense matrices, the Engram memory system for compressed KV caching, and the CQ2-bit quantization format. These components—implemented in needle/model/architecture.py and needle/model/quantize.py—allow the model to operate within severe memory constraints while still supporting tool-calling and structured extraction.

Can Needle 2 run on smartphones and IoT devices?

Yes. The compiled binary is approximately 14 MB and a full inference session requires roughly 28 MB of RAM. According to the implementation in needle/__init__.py, the engine loads lazily and utilizes CPU-optimized paths when GPUs are unavailable, making it suitable for phones, wearables, and edge servers without dedicated AI accelerators.

How does the tool-calling mechanism work?

The model generates JSON-encoded tool calls during inference. The Needle class in needle/__init__.py parses these calls, looks up the corresponding Python function registered via the @tool decorator, executes it, and passes the result back to the model through the complete method. This loop continues until the model produces a final response or exceeds the max_steps threshold.

What is the CQ2-bit quantization format?

CQ2-bit (Cactus Quants) is a proprietary 2-bit weight quantization scheme defined in needle/model/quantize.py. It stores model weights in a highly compressed format that the native engine decompresses on-the-fly during inference, reducing storage requirements and memory bandwidth while maintaining model accuracy for tool-calling tasks.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}