How to Run the Needle 2 Model Offline: Complete Guide to Local Inference

To run Needle 2 offline, pre-download the native library using fetch.fetch_library(), set the NEEDLE2_LIB_PATH environment variable to the local binary path, and instantiate the Needle class with local .cact weights to bypass all network calls.

The Needle 2 engine from the cactus-compute/needle repository operates as a native library (libneedle.*) loaded via Python's ctypes, requiring specific setup for air-gapped environments. When you understand how the wrapper locates and caches the native binary in needle/__init__.py, you can configure completely offline inference using local weight files without triggering downloads from Hugging Face.

How the Needle 2 Library Loading Works

The Python wrapper dynamically loads the native engine through a three-stage resolution process defined in needle/__init__.py.

First, the wrapper detects the model generation (2 or 3) by reading the first four bytes of the provided .cact weight file via the _weight_generation helper. If no weights are specified, it defaults to generation 2.

Next, the _library_path() function (lines 43-71) searches for the appropriate libneedle binary in this order:

1. The directory specified by the NEEDLE2_LIB_PATH or NEEDLE_LIB_PATH environment variable
2. The package installation directory
3. The user cache at ~/.cache/cactus-needle/v{gen}/{version}/
4. Remote download from the Hugging Face repository Cactus-Compute/needle2 (defined in ENGINE_REPOS in needle/agent/fetch.py lines 6-10)

Finally, the _lib() method creates a ctypes.CDLL instance and binds the C functions needle_init, needle_complete, and others required for inference.

Prerequisites for Offline Execution

Before running Needle 2 without internet access, ensure you have:

• The Python package installed: cactus-needle must be available in your environment
• The native library binary: Platform-specific libneedle.so (Linux), libneedle.dylib (macOS), or libneedle.dll (Windows)
• Compatible weights (optional): A local .cact file for fine-tuned inference; otherwise the engine runs with base generation 2 parameters

Step-by-Step Guide to Running Needle 2 Offline

Step 1: Install the Python Package

Install the wrapper once while you have network connectivity. This provides the Python interface and library fetching utilities.

pip install cactus-needle

Step 2: Download the Native Library Locally

Use the fetch utility in needle/agent/fetch.py to download the correct binary for your platform and store it in a directory you control.

from needle.agent import fetch
import os

# Choose a destination folder for the library

dest_dir = "./needle_lib"

# Download the generation 2 library for current OS/CPU

fetch.fetch_library(generation=2, dest_dir=dest_dir)

# Verify the file exists

lib_path = os.path.join(dest_dir, "libneedle.so")  # or .dylib / .dll

print(f"Library downloaded to: {lib_path}")

This pulls the wheel from the Hugging Face repo Cactus-Compute/needle2 and writes it to your specified location without requiring manual browser downloads.

Step 3: Configure the Library Path Environment Variable

Point the wrapper to your cached binary using an environment variable. Use NEEDLE2_LIB_PATH (the newer name) or the legacy NEEDLE_LIB_PATH for backward compatibility.

# Bash / zsh - export before running Python

export NEEDLE2_LIB_PATH=$(realpath ./needle_lib/libneedle.so)

# For legacy support

export NEEDLE_LIB_PATH=$NEEDLE2_LIB_PATH

Setting this variable forces _library_path() to skip the cache check and remote download logic entirely.

Step 4: Verify Offline Loading

Confirm that the library loads without network activity by instantiating the Needle class in a fresh Python session.

import os
os.environ["NEEDLE2_LIB_PATH"] = "/absolute/path/to/needle_lib/libneedle.so"

from needle import Needle

# No weights argument defaults to generation 2

agent = Needle(tools="[]")
print("Loaded library generation:", agent._generation)

If the path is correct, this executes without attempting to download files to ~/.cache/cactus-needle/.

Step 5: Run Inference with Local Weights

Provide a local .cact file path to perform inference with fine-tuned parameters while remaining offline.

weights_path = "./my_finetuned_model.cact"
agent = Needle(tools="[]", weights=weights_path)

result = agent.complete("Explain quantum computing.", max_new_tokens=128)
print(result)

The wrapper reads the generation from the weight file header and loads the already-cached library for inference.

Complete Offline Usage Examples

Example 1: Pre-download and run base model

from needle.agent import fetch
import os

# One-time setup: download library

LIB_DIR = "./needle_lib"
fetch.fetch_library(generation=2, dest_dir=LIB_DIR)

# Configure environment

os.environ["NEEDLE2_LIB_PATH"] = f"{os.path.abspath(LIB_DIR)}/libneedle.so"

# Offline usage

from needle import Needle
agent = Needle(tools=[])
response = agent.complete("Tell me a joke.", max_new_tokens=32)
print(response)

Example 2: Offline inference with custom weights

import os

# Must set before importing Needle

os.environ["NEEDLE2_LIB_PATH"] = "/opt/needle/libneedle.so"

from needle import Needle

# Load local .cact archive (contains generation identifier in header)

weights = "/data/models/my_model.cact"
agent = Needle(tools=[], weights=weights)

result = agent.run("Schedule a meeting tomorrow at 10am.", max_steps=4)
print(result["results"])

Key Source Files and Implementation Details

Understanding these implementation files helps debug offline deployment issues:

• needle/__init__.py: Contains the Needle class, _library_path() (lines 43-71), and _lib() functions that handle ctypes loading and generation detection from .cact files.
• needle/agent/fetch.py: Implements fetch_library() and defines ENGINE_REPOS (lines 6-10) mapping generations to Hugging Face repositories like Cactus-Compute/needle2.
• needle/_worker.py: Manages the fine-tuning worker process that communicates with the native engine when custom weights are provided.
• tests/test_inference.py: Reference implementation showing patterns for Needle.complete() and Needle.run() methods.

The default cache location follows the pattern ~/.cache/cactus-needle/v2/<version>/libneedle.* when downloads occur, but the NEEDLE2_LIB_PATH override bypasses this entirely.

Summary

• Needle 2 requires the native libneedle binary loaded via ctypes, which typically downloads from Hugging Face on first use.
• Use fetch.fetch_library() in needle/agent/fetch.py to pre-download the binary for your platform while online.
• Set the NEEDLE2_LIB_PATH environment variable to the absolute path of libneedle.so, .dylib, or .dll to force offline mode.
• The wrapper detects model generation from the first four bytes of .cact weight files, ensuring compatibility between your binary and weights.
• With the library cached locally and the environment variable set, all subsequent instantiation of the Needle class operates without network connectivity.

Frequently Asked Questions

Where is the Needle 2 library cached after the first download?

When allowed to download automatically, the library is stored in the user cache directory at ~/.cache/cactus-needle/v{gen}/{version}/libneedle.*, where {gen} is the model generation (2 or 3) and {version} is the package version. However, using the NEEDLE2_LIB_PATH environment variable overrides this location and skips cache lookups entirely.

Can I use the legacy NEEDLE_LIB_PATH environment variable?

Yes. The wrapper checks for NEEDLE2_LIB_PATH first (the newer naming convention), but falls back to NEEDLE_LIB_PATH for backward compatibility. Setting either variable to the absolute path of your pre-downloaded binary will prevent network downloads.

What file format should offline weights use?

Needle 2 requires weights in the .cact format, which is a custom archive containing model parameters and a four-byte header indicating the generation. When you provide a local path to a .cact file via the weights parameter, the wrapper reads the generation identifier from the first four bytes to ensure compatibility with the loaded libneedle binary.

Does offline mode support both Needle 2 and Needle 3 generations?

Yes. The offline setup works for any generation supported by your cached library. When using the fetch.fetch_library() utility, specify generation=2 or generation=3 to download the appropriate binary. The Needle class automatically detects the required generation from the weight file header or defaults to generation 2 when no weights are provided, loading the correct cached binary accordingly.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}