How to Deploy Needle 2 on Embedded Systems: Complete Guide for Air-Gapped Devices

Needle 2 deploys to embedded systems via a three-step offline workflow: cache the ~14 MiB native inference engine locally, transfer a self-contained .cact model archive to the target device, and execute inference entirely in-process using either the Python API or the compiled native runner, requiring only 28 MiB of RAM with zero ongoing network connectivity.

Deploying large language models to resource-constrained hardware traditionally requires complex orchestration and persistent cloud connectivity. The cactus-compute/needle repository eliminates these constraints by packaging the entire inference runtime into a single binary that operates in ultra-low-memory environments. This guide explains how to deploy Needle 2 on embedded systems using the official offline deployment strategy documented in doc/apis.md.

System Requirements and Resource Constraints

Memory and Storage Specifications

Needle 2 is optimized for microcontrollers and single-board computers with severe resource limits. The native engine binary occupies approximately 14 MiB of storage, while a full inference session executes within roughly 28 MiB of RAM. These specifications make the runtime suitable for devices such as the Raspberry Pi Zero W, Cortex-M microcontrollers, and WebAssembly hosts.

Network Architecture

The deployment model assumes air-gapped or intermittently connected devices. According to the "Offline devices" section in doc/apis.md (lines 159-166), the engine never contacts the network after initial installation, making it ideal for isolated industrial controllers or remote sensors.

The Three-Step Offline Deployment Workflow

The deployment strategy implemented in needle/agent/fetch.py consists of three distinct phases that separate network-dependent preparation from offline execution.

Step 1 – Pre-Fetch the Native Engine

On a machine with internet access, download the platform-specific inference engine using the CLI. The needle fetch command in needle/cli.py resolves the appropriate binary for your target architecture.

# Download the engine for Linux ARM64 (e.g., Raspberry Pi 4)

needle fetch --platform-tag manylinux2014_aarch64 --out /tmp/needle_engine

This command caches the shared object file (e.g., libneedle.so) and prints the full path, typically within ~/.cache/cactus-needle/v2/.

Step 2 – Transfer the Model Archive

Obtain a .cact archive—either from Hugging Face, a private registry, or built locally using needle build via needle/model/export.py. This file is completely self-contained; no additional weight files or external dependencies are required. Transfer both the engine binary and the model archive to the embedded device:

scp /tmp/needle_engine/libneedle.so user@device:/opt/needle/
scp my_needle.cact user@device:/opt/needle/

Step 3 – Execute Inference

Run the model using either the Python API or the native standalone runner, depending on whether a Python interpreter is available on the target.

Deployment Methods by Target Environment

Python Runtime Deployment

For devices running a full Linux distribution with Python support, set the NEEDLE2_LIB_PATH environment variable to bypass automatic engine detection and load the pre-cached binary directly.

import os
import needle

# Point to the pre-downloaded engine

os.environ["NEEDLE2_LIB_PATH"] = "/opt/needle/libneedle.so"

# Load the self-contained archive

agent = needle.Needle(weights="/opt/needle/my_needle.cact")

# Execute tool-calling inference

result = agent.run("What is the temperature in the kitchen?")
print(result["results"])

The needle.Needle class handles in-process execution without spawning external processes, keeping the memory footprint within the 28 MiB constraint.

Native Runner Deployment (No Python)

For bare-metal or minimal Linux environments lacking a Python interpreter, use the compiled native runner included with the engine distribution:

# Make the runner executable

chmod +x /opt/needle/needle

# Execute via stdin/stdout interface

echo "What is the temperature in the kitchen?" | /opt/needle/needle \
    --weights /opt/needle/my_needle.cact

The runner reads prompts from standard input and returns JSON responses via standard output, enabling integration with shell scripts or C applications.

WebAssembly and Microcontroller Targets

For WebAssembly hosts or constrained microcontrollers, compile the engine using the wasm or wasm-component platform tags. The resulting .wasm module exports a needle_run function that WASI runtimes (e.g., Wasmtime, WasmEdge) can invoke directly. When NEEDLE2_LIB_PATH points to a needle.component.wasm file, the Python API automatically selects the WASI implementation.

Building Custom Model Archives

When deploying fine-tuned models, use the export functionality in needle/model/export.py to merge LoRA adapters into a single distributable archive:

# Build a .cact archive from base weights and LoRA adapters

needle build --base-model llama-3-8b --lora-adapter ./custom_lora --output kitchen_assistant.cact

This produces the self-contained .cact file referenced in the deployment steps above.

Summary

• Ultra-low resource footprint: The Needle 2 engine requires only ~14 MiB storage and ~28 MiB RAM, enabling deployment on microcontrollers and single-board computers.
• True offline operation: Once cached via needle fetch, the engine in libneedle.so requires no network connectivity, as documented in doc/apis.md.
• Flexible execution modes: Deploy using the Python API (needle.Needle) for rich integrations, or the native runner for Python-free environments.
• Self-contained archives: The .cact format encapsulates base models and LoRA fine-tuning weights without external dependencies, handled by needle/model/export.py.
• Cross-platform support: Platform tags like manylinux2014_aarch64 and wasm-component in needle/agent/fetch.py ensure compatibility across ARM64 Linux, WebAssembly, and embedded bare-metal targets.

Frequently Asked Questions

How much RAM is required to run Needle 2 on an embedded device?

Needle 2 executes a full inference session in approximately 28 MiB of RAM. The entire engine binary occupies roughly 14 MiB of storage, making it suitable for microcontrollers and single-board computers with severe memory constraints.

Can Needle 2 operate on devices with no internet connection?

Yes. The architecture specifically supports air-gapped deployment. Pre-fetch the native engine using needle fetch on a connected machine, transfer the libneedle.so binary and .cact archive to the offline device, and execute inference without any network calls. The "Offline devices" section in doc/apis.md (lines 159-166) explicitly documents this workflow.

What is the difference between the Python API and the native runner?

The Python API (needle.Needle) provides programmatic access with support for tool calling and structured outputs, loading the engine via NEEDLE2_LIB_PATH. The native runner is a compiled binary that requires no Python interpreter, communicating via stdin/stdout JSON interfaces. Both modes use the same underlying libneedle.so engine and maintain identical memory footprints.

How do I deploy Needle 2 to a WebAssembly host?

Use needle fetch --platform-tag wasm or wasm-component to generate a .wasm module. Transfer this module along with your .cact archive to the target device. Invoke the exported needle_run function using a WASI-compliant runtime such as Wasmtime or WasmEdge. The Python package can also consume the WASM component directly when NEEDLE2_LIB_PATH points to the .wasm file.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}