How to Set Up Needle 2 for Local Inference on a Mobile Device

Needle 2 runs on mobile devices using a single ~14 MB native binary (libneedle.so) that executes complete inference sessions in ~28 MB of RAM, requiring only the NEEDLE_LIB_PATH environment variable to locate the engine and a .cact weight file to operate fully offline.

Needle 2 from the cactus-compute/needle repository is engineered for "tiny" hardware, enabling local inference on smartphones and tablets without network connectivity. This guide walks through deploying the 45M-parameter transformer using the CQ2-bit quantized engine, from fetching the binary to executing tool-calling agents on-device.

Mobile Architecture and Requirements

The Needle 2 inference stack consists of self-contained components designed for ARM64 mobile processors. Once transferred to the device, the system operates entirely offline.

Core Model Components

• Simple Attention Network: A 45M-parameter transformer defined in needle/model/architecture.py, utilizing Hadamard-MLP layers, Grouped Query Attention (GQA), and a 256-token KV "engram" memory buffer for context retention.
• CQ2-bit Quantization: The Cactus Quants compression format implemented in needle/model/quantize.py that reduces the model footprint to approximately 14 MB, enabling the entire weight file to load into mobile RAM.
• Native Engine (libneedle.so): A C-level runtime compiled for specific platforms (e.g., manylinux2014_aarch64 for Android) that handles token decoding, grammar constraints, and forward passes without Python-side JIT compilation. The engine loading logic resides in needle/model/run.py.
• Python Wrapper: The needle.Needle class and needle.tool decorator exposed in needle/__init__.py, which marshal Python calls into JSON payloads for the engine.

Resource Footprint

Needle 2 requires approximately 28 MB of RAM per inference session and stores the engine binary in roughly 14 MB of disk space. After the initial transfer of libneedle.so and the .cact weight file to the device, the system makes zero network requests during operation.

Installing the Engine on Mobile Devices

Deploying Needle 2 requires transferring the native binary from a development machine to the target device.

1. Install the package on a development machine to access the fetch utilities:

   pip install cactus-needle

2. Fetch the platform-specific binary using the correct platform tag (e.g., manylinux2014_aarch64 for Android ARM64):

   needle fetch --platform-tag manylinux2014_aarch64

   This downloads libneedle.so to the cache directory (typically ~/.cache/cactus-needle/<version>/). For alternative platforms like iOS or macOS ARM64, use needle download <platform> as documented in doc/apis.md.

3. Copy the binary to the mobile device:

   • Android: Push to /data/local/tmp/ or /sdcard/needle/:

     adb push ~/.cache/cactus-needle/*/libneedle.so /sdcard/needle/
     adb shell chmod +x /sdcard/needle/libneedle.so

   • iOS: Embed libneedle.so in the Xcode app bundle, or copy to the Documents folder for Pythonista.

4. Transfer a .cact weight file to the device. Use base weights from needle download or a LoRA-tuned file created with needle build. Place it in an accessible location such as /sdcard/needle/my_needle.cact.

5. Set environment variables before running Python:

   export NEEDLE_LIB_PATH=/sdcard/needle/libneedle.so
   export HF_HUB_OFFLINE=1

Running Local Inference on Mobile

With the engine and weights in place, execute inference using a minimal Python script compatible with Termux, Pythonista, or embedded interpreters.

Basic Inference Implementation

The following script demonstrates loading the binary, registering a tool, and running a constrained generation loop:

import os
import needle

# Point to the native engine binary

os.environ["NEEDLE_LIB_PATH"] = "/sdcard/needle/libneedle.so"

# Load quantized weights (base model or LoRA-tuned)

weights_path = "/sdcard/needle/my_needle.cact"

# Define a mobile-accessible tool

@needle.tool
def send_sms(number: str, body: str):
    """Send an SMS to a phone number."""
    # Implementation would call platform SMS API

    return {"sent_to": number, "body": body}

# Initialize agent

agent = needle.Needle(weights=weights_path, tools=[send_sms])

# Execute query

response = agent.run("Text my friend that I'm running late, 5 pm")
print("Result:", response["results"])
print("Metadata:", response)  # Contains confidence, peak_ram_mb, tokens/sec

Under the hood, needle.Needle loads libneedle.so from the specified path, tokenizes the input, executes a grammar-constrained forward pass through the Simple Attention Network, and handles tool execution via the Python wrapper defined in needle/model/run.py.

Using Pre-built Environments

For wearable-specific actions, import the predefined environment:

from needle.environments import wearable

agent = wearable.agent  # Auto-initialized with tools like reply_to_notification

print(agent.run("Start a running workout"))

The wearable toolset is implemented in needle/environments/wearable.py and provides offline-ready actions for smartwatch deployments.

Summary

• Needle 2 requires only a ~14 MB native binary (libneedle.so) and a .cact weight file to run on mobile devices.
• Set the NEEDLE_LIB_PATH environment variable to the location of libneedle.so before importing the library.
• The 45M-parameter Simple Attention Network with CQ2-bit quantization operates in ~28 MB RAM, suitable for phones and wearables.
• Use needle fetch --platform-tag to download the correct architecture-specific engine during development, then transfer it to the target device.
• Inference executes entirely offline after setup, with no network dependencies during the agent.run() or agent.complete() calls.

Frequently Asked Questions

Does Needle 2 require internet connectivity on the mobile device?

No. Once the libneedle.so engine and .cact weight file are transferred to the device, Needle 2 operates in a fully offline mode. Set HF_HUB_OFFLINE=1 to block any accidental download attempts during initialization.

What is the minimum RAM required for Needle 2 mobile inference?

Needle 2 requires approximately 28 MB of RAM per inference session. The CQ2-bit quantized model weights (Cactus Quants) compress the 45M-parameter Simple Attention Network to roughly 14 MB on disk, allowing the entire model to reside in mobile memory without swapping.

Can I use fine-tuned LoRA models with Needle 2 on mobile?

Yes. Use the needle build command on your development machine to create a tuned .cact file containing LoRA adapters, then transfer this file alongside the base engine to the mobile device. Load it via the weights parameter in needle.Needle().

Which platforms are supported for mobile deployment?

The repository provides pre-built binaries for ARM64 Linux (Android via manylinux2014_aarch64), macOS ARM64 (iOS development), and standard x86_64. Use needle fetch --platform-tag to retrieve the specific shared library for your target architecture, or consult doc/apis.md for the full list of supported platform tags.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}