How to Deploy Needle Offline and Use Local Weights: Complete Air-Gapped Guide

Needle supports fully offline deployment by caching the platform-specific engine binary (libneedle.so, libneedle.dylib, or libneedle.dll) alongside a self-contained .cact weight archive, allowing you to run inference on air-gapped machines without internet access.

Needle by Cactus Compute is a lightweight inference engine distributed as a tiny binary library rather than a traditional Python package. Because the engine is completely weight-agnostic and stores all model parameters in a single portable archive, you can easily deploy Needle offline and use local weights on secure or isolated networks after performing a one-time fetch on an internet-connected device.

Understanding Needle's Offline Architecture

Needle's strict separation between the runtime engine and model weights makes air-gapped deployment straightforward. The system consists of two discrete components that can be transferred independently to target machines.

The Engine Binary

The core runtime ships as a native shared library—libneedle.so on Linux, libneedle.dylib on macOS, or libneedle.dll on Windows. This binary handles all inference operations and is platform-specific, but contains no model parameters, keeping it compact and reusable across different weight files.

The .cact Weight Archive

Model parameters are stored in a single .cact archive, which contains everything the engine needs to run a specific model or fine-tuned checkpoint. According to the source code comments in needle/model/export.py, this format is self-contained and consumed directly by the engine during the binding phase without external dependencies.

Fetching Components on a Connected Machine

Before moving to an offline environment, you must acquire both the engine and weights on a machine with internet access.

Downloading the Engine with needle fetch

The needle fetch command downloads the pre-compiled engine for your current platform. Internally, this invokes fetch_library() in needle/agent/fetch.py, which pulls a wheel from the Hugging Face repository, extracts the shared library, and caches it at ~/.cache/cactus-needle/<engine-version>/.

needle fetch

This stores the platform-appropriate binary (e.g., libneedle.so) in the cache directory for later transfer to the offline system.

Acquiring Model Weights

You can obtain weights using the needle download command to fetch a .cact file from Hugging Face (e.g., Cactus-Compute/needle2/model.cact), or generate your own through fine-tuning followed by needle build.

# Download pre-trained weights

needle download Cactus-Compute/needle2/model.cact --output ./my_model.cact

# Or after fine-tuning locally

needle finetune --data ./training.jsonl
needle build --checkpoint ./checkpoints/final --output ./my_model.cact

Installing Needle on Offline Systems

Once you have the files on a connected machine, transfer them to your air-gapped device using secure media or sneaker-net.

Deploying the Shared Library

Copy the engine binary from the cache directory to your target machine. You can place it in the standard cache location ~/.cache/cactus-needle/<engine-version>/ or any directory accessible to your application.

# On the source machine

cp ~/.cache/cactus-needle/2.0.3/libneedle.so /mnt/offline-transfer/

# On the target offline machine

mkdir -p ~/.cache/cactus-needle/2.0.3/
cp /mnt/offline-transfer/libneedle.so ~/.cache/cactus-needle/2.0.3/

Configuring NEEDLE_LIB_PATH

If you place the library in a non-standard location, set the NEEDLE_LIB_PATH environment variable. The runtime checks this variable first when resolving the library path, as implemented in _library_path() within needle/__init__.py.

export NEEDLE_LIB_PATH=/opt/cactus/libneedle.so

Loading Local Weights for Inference

With the engine in place, you can load any .cact file locally without network calls.

Python API Usage

When constructing a Needle agent, pass the path to your local .cact file via the weights parameter. The _bind() method in needle/__init__.py (lines 70-90) handles loading the weight blob once per process.

import needle

# Create agent with local weights

agent = needle.Needle(weights="/path/to/my_needle.cact")

# Run inference

response = agent.run("What is the weather in Paris?")
print("Answer:", response["answer"])
print("Tool results:", response.get("results", []))

Subsequent agents created without a weights argument will reuse the already-loaded model in the same process.

CLI Usage

The command-line interface accepts the --weights flag to specify local archives, as defined in the CLI implementation at needle/cli.py (lines 81-84).

# Run a single query

needle run --checkpoint dummy --query "Tell me a joke" --weights ./my_needle.cact

# Start the playground server

needle playground --weights ./my_needle.cact --port 7860

Complete Offline Workflow Example

Follow this pattern to deploy Needle in secure environments:

1. On the connected machine: Run needle fetch to cache the engine, then needle download (or needle build) to create a .cact file.
2. Transfer files: Move libneedle.so (or .dylib/.dll) and the .cact archive to the offline machine via secure media.
3. Configure environment: Optionally set NEEDLE_LIB_PATH if not using the default cache location.
4. Execute inference: Use the Python API or CLI commands pointing to your local weight file.

# Offline machine setup

export NEEDLE_LIB_PATH=/opt/needle/libneedle.so

# Verify offline operation

python -c "import needle; agent=needle.Needle(weights='./model.cact'); print(agent.run('Hello')['answer'])"

Summary

• Needle separates the inference engine from model weights, enabling true offline deployment.
• The engine binary (libneedle.so, libneedle.dylib, or libneedle.dll) is fetched via fetch_library() in needle/agent/fetch.py and can be cached for transfer to air-gapped systems.
• Model parameters reside in self-contained .cact archives created by needle build or downloaded via needle download.
• Set NEEDLE_LIB_PATH to specify non-standard library locations, handled by _library_path() in needle/__init__.py.
• Load local weights via the weights parameter in Python or --weights flag in CLI, with binding logic in _bind() at needle/__init__.py.

Frequently Asked Questions

Can I deploy Needle on a completely air-gapped server?

Yes. Because the Needle engine is weight-agnostic and all model parameters are contained in the .cact archive, you can transfer both files to an offline machine after fetching them on a connected device. No internet connectivity is required for inference once the files are in place.

What files do I need to transfer to the offline machine?

You need two files: the platform-specific engine binary (libneedle.so for Linux, libneedle.dylib for macOS, or libneedle.dll for Windows) and the .cact weight archive. Optionally, include your application code and any tool definitions required for your use case.

How does Needle locate the engine library without internet access?

The runtime first checks the NEEDLE_LIB_PATH environment variable, then falls back to the platform-specific cache directory (~/.cache/cactus-needle/<engine-version>/). This resolution logic resides in _library_path() within needle/__init__.py, requiring no network calls to locate the library.

Can I use multiple different weight files offline?

Yes. Since the engine loads weights dynamically, you can instantiate multiple Needle agents with different weights parameters, each pointing to a different .cact file. The engine only loads the weight blob once per unique path, making it efficient to switch between models on the same offline host.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}