# How the Needle Inference Engine Manages Model Weights: Inside the .cact Archive System

> Discover how the Needle inference engine manages model weights within a single .cact archive. Learn about automatic generation detection and zero-copy inference with a constant memory footprint.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: internals
- Published: 2026-09-12

---

**The Needle inference engine embeds all model weights directly inside a single binary `.cact` archive, automatically detects the engine generation from the first four bytes, and loads the appropriate native library via `ctypes` to provide zero-copy inference with a constant ~28 MiB memory footprint.**

The `cactus-compute/needle` repository implements a unique weight management strategy that eliminates external file dependencies by baking the entire model—including optional LoRA adapters—into one portable archive. Understanding how the Needle inference engine manages model weights reveals why it can deploy on tiny devices without requiring separate weight directories or runtime downloads.

## Embedding Weights in the .cact Archive Format

Unlike traditional frameworks that load weights from multiple `.bin` or `.safetensors` files, Needle uses a proprietary **`.cact`** archive format that bundles the base model and any fine-tuning parameters into a single executable binary. This design ensures that once the archive is built, the inference engine requires no additional filesystem access to retrieve parameters during runtime.

The archive supports both generation 2 and generation 3 engine binaries, with the version determined by a header tag embedded at the start of the file. When fine-tuning is required, the `FineTuneWorker` class (implemented in [`needle/_worker.py`](https://github.com/cactus-compute/needle/blob/main/needle/_worker.py)) merges LoRA adapters with the base weights before producing the final combined archive, maintaining the single-file execution model.

## Detecting Engine Generation from Weight Archives

Before loading any parameters, the engine must determine which native library version to use. The private helper **`_weight_generation`** in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) reads the first four bytes of the `.cact` file and maps the little-endian integer to a generation number using the `_CACT_GENERATIONS` lookup table.

According to the source code in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) (lines 29-41), the detection logic works as follows:

```python
def _weight_generation(path):
    with open(path, "rb") as handle:
        tag_bytes = handle.read(4)
    tag = int.from_bytes(tag_bytes, "little")
    return _CACT_GENERATIONS[tag]          # → 2 or 3

```

This generation number dictates which native engine binary (`libneedle` v2 or v3) the wrapper must load, ensuring backward compatibility while allowing the engine to evolve its weight format.

## Resolving and Loading the Native Engine Library

Once the generation is identified, the **`_library_path`** function (lines 43-73 in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py)) resolves the path to the appropriate native shared library. The resolution follows a strict hierarchy:

1. **Environment override**: Checks `NEEDLE2_LIB_PATH` or `NEEDLE3_LIB_PATH` for user-specified libraries
2. **Package directory**: Searches local filenames in the installed package
3. **Cache fallback**: Looks in `~/.cache/cactus-needle/v{generation}/{version}/`
4. **Dynamic download**: Fetches the correct binary via `fetch.fetch_library` in [`needle/agent/fetch.py`](https://github.com/cactus-compute/needle/blob/main/needle/agent/fetch.py) if not found locally

The implementation ensures that the engine can bootstrap itself on fresh installations while respecting system administrators' need to pin specific library versions:

```python
def _library_path(generation=2):
    override = os.environ.get(f"NEEDLE{generation}_LIB_PATH")
    # … try local filenames …

    if os.path.exists(local):
        return local
    # fallback to cached download

    version = fetch.engine_version(generation)
    cache = os.path.join(os.path.expanduser("~"),
                        ".cache", "cactus-needle", f"v{generation}", version)
    cached = os.path.join(cache, fetch._lib_name())
    if os.path.exists(cached):
        return cached
    # download the correct library

    return fetch.fetch_library(version, cache, generation=generation)

```

## Runtime Weight Loading and Memory Management

When instantiating the `Needle` class with a `weights` argument, the wrapper executes a three-phase initialization process defined in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) (lines 53-61 and 87-99):

1. **Generation detection**: Calls `_weight_generation(self._weights)` to identify the archive version
2. **Worker initialization**: Creates a `FineTuneWorker` if tuning data is present, otherwise proceeds to direct binding
3. **Native loading**: Invokes `needle_load` through the `ctypes` handle to memory-map the archive

The `needle_load` function exposes the weight archive directly to the native engine, eliminating Python-side copies. During inference calls (`complete`, `run`, `embed`), the engine references this already-loaded memory buffer, ensuring that the **RAM usage remains constant at approximately 28 MiB** regardless of input size.

```python
self._generation = _weight_generation(self._weights) if self._weights else 2
...
if self._weights:
    self._worker = FineTuneWorker(... )     # fine-tune path

else:
    self._bind()                           # plain engine path

```

## Practical Implementation Examples

The following patterns demonstrate how to leverage Needle's weight management in production code:

Load the default model with baked-in weights, allowing the engine to auto-fetch the correct native library:

```python
import needle

agent = needle.Needle()
print(agent.complete("What can you do?")["completion"])

```

Load a fine-tuned `.cact` archive exported from custom training, which automatically triggers generation detection:

```python
agent_ft = needle.Needle(weights="my_needle.cact", tools=[...])
print(agent_ft.run("Schedule a meeting tomorrow")["results"])

```

Inspect the generation of any archive file manually using the internal helper:

```python
from needle import _weight_generation
gen = _weight_generation("my_needle.cact")
print(f"The archive is for Needle generation {gen}")

```

## Summary

- **Single-archive design**: All base weights and LoRA adapters reside in one `.cact` binary, eliminating external dependencies during inference.
- **Generation tagging**: The first four bytes of the archive determine whether the engine loads v2 or v3 native libraries via the `_weight_generation` function.
- **Hierarchical library resolution**: The `_library_path` function searches environment variables, package directories, and `~/.cache/cactus-needle` before downloading missing binaries.
- **Zero-copy execution**: The `needle_load` function maps the archive directly into native memory via `ctypes`, maintaining a constant ~28 MiB footprint across all inference calls.

## Frequently Asked Questions

### What is the .cact file format in Needle?

The `.cact` format is a proprietary binary archive that encapsulates the entire model state—including base parameters and optional LoRA fine-tuning weights—into a single file. According to the `cactus-compute/needle` source code, the format prefixes the weight data with a 4-byte little-endian generation tag that identifies which engine binary (v2 or v3) can interpret the archive.

### How does Needle handle fine-tuned or LoRA weights?

When the `Needle` constructor receives a `weights` argument, it instantiates a `FineTuneWorker` from [`needle/_worker.py`](https://github.com/cactus-compute/needle/blob/main/needle/_worker.py) if fine-tuning data is detected. This worker merges the LoRA adapter with the base weights into a unified archive before the native `needle_load` function processes it, preserving the single-file execution model while supporting parameter-efficient fine-tuning.

### Where does Needle store the native engine libraries?

The `_library_path` function implements a tiered search strategy: it first checks the `NEEDLE{generation}_LIB_PATH` environment variables, then the package installation directory, followed by the user cache at `~/.cache/cactus-needle/v{generation}/{version}/`. If the library is absent, the system downloads the correct version via [`needle/agent/fetch.py`](https://github.com/cactus-compute/needle/blob/main/needle/agent/fetch.py) and stores it in that cache location for future runs.

### What is the memory footprint during inference?

The Needle inference engine maintains a constant memory footprint of approximately **28 MiB** regardless of prompt length or tool complexity. This stability comes from the zero-copy weight loading mechanism: once `needle_load` maps the `.cact` archive into memory, subsequent calls to `needle_complete` or `needle_embed` reference that same buffer without allocating additional weight storage.