# How to Run the Needle 2 Model Offline: Complete Guide to Local Inference

> Run Needle 2 model offline with this complete guide. Download the native library, set the environment variable, and use local weights for seamless local inference.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: how-to-guide
- Published: 2026-09-05

---

**To run Needle 2 offline, pre-download the native library using `fetch.fetch_library()`, set the `NEEDLE2_LIB_PATH` environment variable to the local binary path, and instantiate the `Needle` class with local `.cact` weights to bypass all network calls.**

The Needle 2 engine from the **cactus-compute/needle** repository operates as a native library (`libneedle.*`) loaded via Python's **ctypes**, requiring specific setup for air-gapped environments. When you understand how the wrapper locates and caches the native binary in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py), you can configure completely offline inference using local weight files without triggering downloads from Hugging Face.

## How the Needle 2 Library Loading Works

The Python wrapper dynamically loads the native engine through a three-stage resolution process defined in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py).

First, the wrapper detects the **model generation** (2 or 3) by reading the first four bytes of the provided `.cact` weight file via the `_weight_generation` helper. If no weights are specified, it defaults to generation 2.

Next, the `_library_path()` function (lines 43-71) searches for the appropriate `libneedle` binary in this order:

1. The directory specified by the `NEEDLE2_LIB_PATH` or `NEEDLE_LIB_PATH` environment variable
2. The package installation directory
3. The user cache at `~/.cache/cactus-needle/v{gen}/{version}/`
4. Remote download from the Hugging Face repository `Cactus-Compute/needle2` (defined in `ENGINE_REPOS` in [`needle/agent/fetch.py`](https://github.com/cactus-compute/needle/blob/main/needle/agent/fetch.py) lines 6-10)

Finally, the `_lib()` method creates a `ctypes.CDLL` instance and binds the C functions `needle_init`, `needle_complete`, and others required for inference.

## Prerequisites for Offline Execution

Before running Needle 2 without internet access, ensure you have:

- **The Python package installed**: `cactus-needle` must be available in your environment
- **The native library binary**: Platform-specific `libneedle.so` (Linux), `libneedle.dylib` (macOS), or `libneedle.dll` (Windows)
- **Compatible weights** (optional): A local `.cact` file for fine-tuned inference; otherwise the engine runs with base generation 2 parameters

## Step-by-Step Guide to Running Needle 2 Offline

### Step 1: Install the Python Package

Install the wrapper once while you have network connectivity. This provides the Python interface and library fetching utilities.

```bash
pip install cactus-needle

```

### Step 2: Download the Native Library Locally

Use the fetch utility in [`needle/agent/fetch.py`](https://github.com/cactus-compute/needle/blob/main/needle/agent/fetch.py) to download the correct binary for your platform and store it in a directory you control.

```python
from needle.agent import fetch
import os

# Choose a destination folder for the library

dest_dir = "./needle_lib"

# Download the generation 2 library for current OS/CPU

fetch.fetch_library(generation=2, dest_dir=dest_dir)

# Verify the file exists

lib_path = os.path.join(dest_dir, "libneedle.so")  # or .dylib / .dll

print(f"Library downloaded to: {lib_path}")

```

This pulls the wheel from the Hugging Face repo `Cactus-Compute/needle2` and writes it to your specified location without requiring manual browser downloads.

### Step 3: Configure the Library Path Environment Variable

Point the wrapper to your cached binary using an environment variable. Use **NEEDLE2_LIB_PATH** (the newer name) or the legacy **NEEDLE_LIB_PATH** for backward compatibility.

```bash

# Bash / zsh - export before running Python

export NEEDLE2_LIB_PATH=$(realpath ./needle_lib/libneedle.so)

# For legacy support

export NEEDLE_LIB_PATH=$NEEDLE2_LIB_PATH

```

Setting this variable forces `_library_path()` to skip the cache check and remote download logic entirely.

### Step 4: Verify Offline Loading

Confirm that the library loads without network activity by instantiating the `Needle` class in a fresh Python session.

```python
import os
os.environ["NEEDLE2_LIB_PATH"] = "/absolute/path/to/needle_lib/libneedle.so"

from needle import Needle

# No weights argument defaults to generation 2

agent = Needle(tools="[]")
print("Loaded library generation:", agent._generation)

```

If the path is correct, this executes without attempting to download files to `~/.cache/cactus-needle/`.

### Step 5: Run Inference with Local Weights

Provide a local `.cact` file path to perform inference with fine-tuned parameters while remaining offline.

```python
weights_path = "./my_finetuned_model.cact"
agent = Needle(tools="[]", weights=weights_path)

result = agent.complete("Explain quantum computing.", max_new_tokens=128)
print(result)

```

The wrapper reads the generation from the weight file header and loads the already-cached library for inference.

## Complete Offline Usage Examples

**Example 1: Pre-download and run base model**

```python
from needle.agent import fetch
import os

# One-time setup: download library

LIB_DIR = "./needle_lib"
fetch.fetch_library(generation=2, dest_dir=LIB_DIR)

# Configure environment

os.environ["NEEDLE2_LIB_PATH"] = f"{os.path.abspath(LIB_DIR)}/libneedle.so"

# Offline usage

from needle import Needle
agent = Needle(tools=[])
response = agent.complete("Tell me a joke.", max_new_tokens=32)
print(response)

```

**Example 2: Offline inference with custom weights**

```python
import os

# Must set before importing Needle

os.environ["NEEDLE2_LIB_PATH"] = "/opt/needle/libneedle.so"

from needle import Needle

# Load local .cact archive (contains generation identifier in header)

weights = "/data/models/my_model.cact"
agent = Needle(tools=[], weights=weights)

result = agent.run("Schedule a meeting tomorrow at 10am.", max_steps=4)
print(result["results"])

```

## Key Source Files and Implementation Details

Understanding these implementation files helps debug offline deployment issues:

- **[`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py)**: Contains the `Needle` class, `_library_path()` (lines 43-71), and `_lib()` functions that handle **ctypes** loading and generation detection from `.cact` files.
- **[`needle/agent/fetch.py`](https://github.com/cactus-compute/needle/blob/main/needle/agent/fetch.py)**: Implements `fetch_library()` and defines `ENGINE_REPOS` (lines 6-10) mapping generations to Hugging Face repositories like `Cactus-Compute/needle2`.
- **[`needle/_worker.py`](https://github.com/cactus-compute/needle/blob/main/needle/_worker.py)**: Manages the fine-tuning worker process that communicates with the native engine when custom weights are provided.
- **[`tests/test_inference.py`](https://github.com/cactus-compute/needle/blob/main/tests/test_inference.py)**: Reference implementation showing patterns for `Needle.complete()` and `Needle.run()` methods.

The default cache location follows the pattern `~/.cache/cactus-needle/v2/<version>/libneedle.*` when downloads occur, but the `NEEDLE2_LIB_PATH` override bypasses this entirely.

## Summary

- **Needle 2** requires the native `libneedle` binary loaded via **ctypes**, which typically downloads from Hugging Face on first use.
- Use `fetch.fetch_library()` in [`needle/agent/fetch.py`](https://github.com/cactus-compute/needle/blob/main/needle/agent/fetch.py) to pre-download the binary for your platform while online.
- Set the **NEEDLE2_LIB_PATH** environment variable to the absolute path of `libneedle.so`, `.dylib`, or `.dll` to force offline mode.
- The wrapper detects model generation from the first four bytes of `.cact` weight files, ensuring compatibility between your binary and weights.
- With the library cached locally and the environment variable set, all subsequent instantiation of the `Needle` class operates without network connectivity.

## Frequently Asked Questions

### Where is the Needle 2 library cached after the first download?

When allowed to download automatically, the library is stored in the user cache directory at `~/.cache/cactus-needle/v{gen}/{version}/libneedle.*`, where `{gen}` is the model generation (2 or 3) and `{version}` is the package version. However, using the `NEEDLE2_LIB_PATH` environment variable overrides this location and skips cache lookups entirely.

### Can I use the legacy NEEDLE_LIB_PATH environment variable?

Yes. The wrapper checks for **NEEDLE2_LIB_PATH** first (the newer naming convention), but falls back to **NEEDLE_LIB_PATH** for backward compatibility. Setting either variable to the absolute path of your pre-downloaded binary will prevent network downloads.

### What file format should offline weights use?

Needle 2 requires weights in the **.cact** format, which is a custom archive containing model parameters and a four-byte header indicating the generation. When you provide a local path to a `.cact` file via the `weights` parameter, the wrapper reads the generation identifier from the first four bytes to ensure compatibility with the loaded `libneedle` binary.

### Does offline mode support both Needle 2 and Needle 3 generations?

Yes. The offline setup works for any generation supported by your cached library. When using the `fetch.fetch_library()` utility, specify `generation=2` or `generation=3` to download the appropriate binary. The `Needle` class automatically detects the required generation from the weight file header or defaults to generation 2 when no weights are provided, loading the correct cached binary accordingly.