How to Run Needle 2 Model Offline on Embedded Devices: Complete Deployment Guide

Needle 2 runs offline on embedded devices by fetching the native engine library (libneedle.so/.dylib/.dll) and model weights (.cact checkpoint) in advance, then executing inference with environment variables like NEEDLE_LIB_PATH and HF_HUB_OFFLINE=1 to eliminate network dependencies.

Needle 2 provides a lightweight inference engine specifically designed for air-gapped and resource-constrained environments. According to the cactus-compute/needle source code, the framework ships as a single binary engine alongside optional checkpoint files, enabling you to run Needle 2 model offline on embedded devices without any runtime network connectivity.

Prerequisites for Offline Deployment

Before deploying to embedded hardware, you need both the native library and the model weights. The system consists of two primary components: the engine binary (libneedle.so, libneedle.dylib, or libneedle.dll) and the checkpoint file (.cact extension containing model weights). According to doc/apis.md, these files must be pre-positioned on the target filesystem before disconnecting from the network.

Three-Stage Offline Deployment Workflow

Stage 1: Fetch the Native Engine Library

The first step involves pulling the pre-compiled native library for your specific platform. In needle/agent/fetch.py, the _platform_tag() function determines the correct wheel tag for your architecture, while _lib_name_for() selects the appropriate library name.

Use the CLI command needle fetch to download the engine from the Hugging Face repository Cactus-Compute/needle2:

needle fetch

This places the library in the local cache at ~/.cache/cactus-needle/<engine-version>/. For cross-compilation scenarios, specify the platform tag explicitly:

needle fetch --platform-tag manylinux2014_aarch64

Stage 2: Download the Model Checkpoint

Next, retrieve the model weights using the download command implemented in needle/cli.py. This command calls either fetch.download_platform for standard engines or hf_hub_download for custom weights.

Download the base checkpoint for your platform:

needle download linux-x86_64

Or fetch a custom tuned file:

needle download <org>/<repo>/<file>.cact

The resulting .cact file can reside in the cache path or alongside your application binary.

Stage 3: Run Inference Without Network Activity

Once the engine and weights are local, execute inference using either the CLI or Python API. Set NEEDLE_LIB_PATH to point to the engine binary and optionally set HF_HUB_OFFLINE=1 to prevent any network attempts.

CLI approach:

needle run \
    --checkpoint path/to/needle2.cact \
    --query "What is the weather in Paris?" \
    --tools tools.json

Python approach:

import os
import needle

# Point to the pre-copied library binary

os.environ["NEEDLE_LIB_PATH"] = "/home/user/.cache/cactus-needle/2.0.3/libneedle.so"

# Optional: ensure complete offline mode

os.environ["HF_HUB_OFFLINE"] = "1"
os.environ["NEEDLE_TELEMETRY"] = "0"

# Load checkpoint - no network requests performed

agent = needle.Needle(weights="my_needle.cact", tools=[...])

# Execute inference

response = agent.run("Dim the living room lights to 30%")
print(response["results"])

Embedded Device Configuration Guide

When deploying to air-gapped embedded systems, copy the cached libneedle.* to a persistent location such as /usr/lib/needle/ or /opt/needle/. The engine consumes approximately 28 MiB of RAM, making it suitable for devices with as little as 64 MiB total memory.

Key environment variables for embedded deployment:

• NEEDLE_LIB_PATH: Absolute path to the native engine binary (e.g., /opt/needle/libneedle.so)
• HF_HUB_OFFLINE=1: Prevents the Hugging Face hub library from attempting network downloads
• NEEDLE_TELEMETRY=0: Disables telemetry reporting for completely silent operation

Platform-Specific Implementation Examples

Linux ARM64 Embedded Board

For ARM64 microcontrollers running Linux, fetch the specific platform build and execute:

# Pull engine binary for linux-arm64

needle fetch --platform-tag manylinux2014_aarch64

# Download base checkpoint

needle download linux-arm64

# Run with no external connectivity

needle run --checkpoint ~/.cache/cactus-needle/2.0.3/needle2.cact \
           --query "Tell me a joke"

Python Integration on Resource-Constrained Devices

For direct Python integration on embedded Linux systems:

import os
import needle

# Configure library path for embedded filesystem

os.environ["NEEDLE_LIB_PATH"] = "/opt/needle/libneedle.so"
os.environ["HF_HUB_OFFLINE"] = "1"

# Initialize with pre-downloaded weights

agent = needle.Needle(weights="/opt/needle/needle2.cact", tools=[])

# Single-step inference

print(agent.run("What is the temperature?"))

Summary

• Needle 2 operates as a single binary engine (libneedle.so/.dylib/.dll) paired with .cact checkpoint files, requiring no network access during inference.
• Use needle fetch to download platform-specific engines from needle/agent/fetch.py, specifying tags like manylinux2014_aarch64 for embedded targets.
• Download weights via needle download from needle/cli.py before disconnecting from the network.
• Set NEEDLE_LIB_PATH to the engine location and HF_HUB_OFFLINE=1 to guarantee offline operation on air-gapped devices.
• The engine footprint stays under 28 MiB RAM, suitable for embedded devices with >64 MiB memory.

Frequently Asked Questions

What files do I need to transfer to an air-gapped device to run Needle 2?

You need two components: the native engine library (libneedle.so, libneedle.dylib, or libneedle.dll) and the model checkpoint file (.cact extension). According to the source code in needle/agent/fetch.py, the engine handles platform-specific tagging automatically when you use needle fetch on a connected system first.

How do I prevent Needle 2 from attempting network connections at runtime?

Set the environment variable HF_HUB_OFFLINE=1 before importing the needle module. Additionally, set NEEDLE_TELEMETRY=0 to disable telemetry. As implemented in the source codebase, these flags force the library to operate exclusively with local files and cache directories.

What is the minimum RAM requirement for running Needle 2 on embedded hardware?

The engine requires approximately 28 MiB of RAM during inference. The documentation in doc/apis.md recommends devices with at least 64 MiB total memory to accommodate the engine, weights, and operating system overhead.

Can I use custom-tuned model weights instead of the base checkpoint?

Yes. Use the command needle download <org>/<repo>/<file>.cact to fetch custom weights from Hugging Face Hub. The download logic in needle/cli.py supports arbitrary repository paths, allowing you to deploy fine-tuned models to offline embedded devices using the same deployment workflow as base models.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}