# How to Set Up Needle 2 for Local Inference on a Mobile Device

> Learn how to set up Needle 2 for local inference on your mobile device. Run complete inference sessions offline with a small binary and minimal RAM. Get started now.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: how-to-guide
- Published: 2026-08-30

---

**Needle 2 runs on mobile devices using a single ~14 MB native binary (`libneedle.so`) that executes complete inference sessions in ~28 MB of RAM, requiring only the `NEEDLE_LIB_PATH` environment variable to locate the engine and a `.cact` weight file to operate fully offline.**

Needle 2 from the cactus-compute/needle repository is engineered for "tiny" hardware, enabling local inference on smartphones and tablets without network connectivity. This guide walks through deploying the 45M-parameter transformer using the CQ2-bit quantized engine, from fetching the binary to executing tool-calling agents on-device.

## Mobile Architecture and Requirements

The Needle 2 inference stack consists of self-contained components designed for ARM64 mobile processors. Once transferred to the device, the system operates entirely offline.

### Core Model Components

- **Simple Attention Network**: A 45M-parameter transformer defined in [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py), utilizing Hadamard-MLP layers, Grouped Query Attention (GQA), and a 256-token KV "engram" memory buffer for context retention.
- **CQ2-bit Quantization**: The Cactus Quants compression format implemented in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py) that reduces the model footprint to approximately 14 MB, enabling the entire weight file to load into mobile RAM.
- **Native Engine (`libneedle.so`)**: A C-level runtime compiled for specific platforms (e.g., `manylinux2014_aarch64` for Android) that handles token decoding, grammar constraints, and forward passes without Python-side JIT compilation. The engine loading logic resides in [`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py).
- **Python Wrapper**: The `needle.Needle` class and `needle.tool` decorator exposed in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py), which marshal Python calls into JSON payloads for the engine.

### Resource Footprint

Needle 2 requires approximately **28 MB of RAM** per inference session and stores the engine binary in roughly **14 MB** of disk space. After the initial transfer of `libneedle.so` and the `.cact` weight file to the device, the system makes zero network requests during operation.

## Installing the Engine on Mobile Devices

Deploying Needle 2 requires transferring the native binary from a development machine to the target device.

1. **Install the package on a development machine** to access the fetch utilities:

   ```bash
   pip install cactus-needle
   ```

2. **Fetch the platform-specific binary** using the correct platform tag (e.g., `manylinux2014_aarch64` for Android ARM64):

   ```bash
   needle fetch --platform-tag manylinux2014_aarch64
   ```

   This downloads `libneedle.so` to the cache directory (typically `~/.cache/cactus-needle/<version>/`). For alternative platforms like iOS or macOS ARM64, use `needle download <platform>` as documented in [`doc/apis.md`](https://github.com/cactus-compute/needle/blob/main/doc/apis.md).

3. **Copy the binary to the mobile device**:
   - **Android**: Push to `/data/local/tmp/` or `/sdcard/needle/`:

     ```bash
     adb push ~/.cache/cactus-needle/*/libneedle.so /sdcard/needle/
     adb shell chmod +x /sdcard/needle/libneedle.so
     ```

   - **iOS**: Embed `libneedle.so` in the Xcode app bundle, or copy to the Documents folder for Pythonista.

4. **Transfer a `.cact` weight file** to the device. Use base weights from `needle download` or a LoRA-tuned file created with `needle build`. Place it in an accessible location such as `/sdcard/needle/my_needle.cact`.

5. **Set environment variables** before running Python:

   ```bash
   export NEEDLE_LIB_PATH=/sdcard/needle/libneedle.so
   export HF_HUB_OFFLINE=1
   ```

## Running Local Inference on Mobile

With the engine and weights in place, execute inference using a minimal Python script compatible with Termux, Pythonista, or embedded interpreters.

### Basic Inference Implementation

The following script demonstrates loading the binary, registering a tool, and running a constrained generation loop:

```python
import os
import needle

# Point to the native engine binary

os.environ["NEEDLE_LIB_PATH"] = "/sdcard/needle/libneedle.so"

# Load quantized weights (base model or LoRA-tuned)

weights_path = "/sdcard/needle/my_needle.cact"

# Define a mobile-accessible tool

@needle.tool
def send_sms(number: str, body: str):
    """Send an SMS to a phone number."""
    # Implementation would call platform SMS API

    return {"sent_to": number, "body": body}

# Initialize agent

agent = needle.Needle(weights=weights_path, tools=[send_sms])

# Execute query

response = agent.run("Text my friend that I'm running late, 5 pm")
print("Result:", response["results"])
print("Metadata:", response)  # Contains confidence, peak_ram_mb, tokens/sec

```

Under the hood, `needle.Needle` loads `libneedle.so` from the specified path, tokenizes the input, executes a grammar-constrained forward pass through the Simple Attention Network, and handles tool execution via the Python wrapper defined in [`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py).

### Using Pre-built Environments

For wearable-specific actions, import the predefined environment:

```python
from needle.environments import wearable

agent = wearable.agent  # Auto-initialized with tools like reply_to_notification

print(agent.run("Start a running workout"))

```

The wearable toolset is implemented in [`needle/environments/wearable.py`](https://github.com/cactus-compute/needle/blob/main/needle/environments/wearable.py) and provides offline-ready actions for smartwatch deployments.

## Summary

- **Needle 2** requires only a ~14 MB native binary (`libneedle.so`) and a `.cact` weight file to run on mobile devices.
- Set the `NEEDLE_LIB_PATH` environment variable to the location of `libneedle.so` before importing the library.
- The 45M-parameter Simple Attention Network with CQ2-bit quantization operates in ~28 MB RAM, suitable for phones and wearables.
- Use `needle fetch --platform-tag` to download the correct architecture-specific engine during development, then transfer it to the target device.
- Inference executes entirely offline after setup, with no network dependencies during the `agent.run()` or `agent.complete()` calls.

## Frequently Asked Questions

### Does Needle 2 require internet connectivity on the mobile device?

No. Once the `libneedle.so` engine and `.cact` weight file are transferred to the device, Needle 2 operates in a fully offline mode. Set `HF_HUB_OFFLINE=1` to block any accidental download attempts during initialization.

### What is the minimum RAM required for Needle 2 mobile inference?

Needle 2 requires approximately **28 MB of RAM** per inference session. The CQ2-bit quantized model weights (Cactus Quants) compress the 45M-parameter Simple Attention Network to roughly 14 MB on disk, allowing the entire model to reside in mobile memory without swapping.

### Can I use fine-tuned LoRA models with Needle 2 on mobile?

Yes. Use the `needle build` command on your development machine to create a tuned `.cact` file containing LoRA adapters, then transfer this file alongside the base engine to the mobile device. Load it via the `weights` parameter in `needle.Needle()`.

### Which platforms are supported for mobile deployment?

The repository provides pre-built binaries for ARM64 Linux (Android via `manylinux2014_aarch64`), macOS ARM64 (iOS development), and standard x86_64. Use `needle fetch --platform-tag` to retrieve the specific shared library for your target architecture, or consult [`doc/apis.md`](https://github.com/cactus-compute/needle/blob/main/doc/apis.md) for the full list of supported platform tags.