How the Needle Inference Engine Manages Model Weights: Inside the .cact Archive System

The Needle inference engine embeds all model weights directly inside a single binary .cact archive, automatically detects the engine generation from the first four bytes, and loads the appropriate native library via ctypes to provide zero-copy inference with a constant ~28 MiB memory footprint.

The cactus-compute/needle repository implements a unique weight management strategy that eliminates external file dependencies by baking the entire model—including optional LoRA adapters—into one portable archive. Understanding how the Needle inference engine manages model weights reveals why it can deploy on tiny devices without requiring separate weight directories or runtime downloads.

Embedding Weights in the .cact Archive Format

Unlike traditional frameworks that load weights from multiple .bin or .safetensors files, Needle uses a proprietary .cact archive format that bundles the base model and any fine-tuning parameters into a single executable binary. This design ensures that once the archive is built, the inference engine requires no additional filesystem access to retrieve parameters during runtime.

The archive supports both generation 2 and generation 3 engine binaries, with the version determined by a header tag embedded at the start of the file. When fine-tuning is required, the FineTuneWorker class (implemented in needle/_worker.py) merges LoRA adapters with the base weights before producing the final combined archive, maintaining the single-file execution model.

Detecting Engine Generation from Weight Archives

Before loading any parameters, the engine must determine which native library version to use. The private helper _weight_generation in needle/__init__.py reads the first four bytes of the .cact file and maps the little-endian integer to a generation number using the _CACT_GENERATIONS lookup table.

According to the source code in needle/__init__.py (lines 29-41), the detection logic works as follows:

def _weight_generation(path):
    with open(path, "rb") as handle:
        tag_bytes = handle.read(4)
    tag = int.from_bytes(tag_bytes, "little")
    return _CACT_GENERATIONS[tag]          # → 2 or 3

This generation number dictates which native engine binary (libneedle v2 or v3) the wrapper must load, ensuring backward compatibility while allowing the engine to evolve its weight format.

Resolving and Loading the Native Engine Library

Once the generation is identified, the _library_path function (lines 43-73 in needle/__init__.py) resolves the path to the appropriate native shared library. The resolution follows a strict hierarchy:

1. Environment override: Checks NEEDLE2_LIB_PATH or NEEDLE3_LIB_PATH for user-specified libraries
2. Package directory: Searches local filenames in the installed package
3. Cache fallback: Looks in ~/.cache/cactus-needle/v{generation}/{version}/
4. Dynamic download: Fetches the correct binary via fetch.fetch_library in needle/agent/fetch.py if not found locally

The implementation ensures that the engine can bootstrap itself on fresh installations while respecting system administrators' need to pin specific library versions:

def _library_path(generation=2):
    override = os.environ.get(f"NEEDLE{generation}_LIB_PATH")
    # … try local filenames …

    if os.path.exists(local):
        return local
    # fallback to cached download

    version = fetch.engine_version(generation)
    cache = os.path.join(os.path.expanduser("~"),
                        ".cache", "cactus-needle", f"v{generation}", version)
    cached = os.path.join(cache, fetch._lib_name())
    if os.path.exists(cached):
        return cached
    # download the correct library

    return fetch.fetch_library(version, cache, generation=generation)

Runtime Weight Loading and Memory Management

When instantiating the Needle class with a weights argument, the wrapper executes a three-phase initialization process defined in needle/__init__.py (lines 53-61 and 87-99):

1. Generation detection: Calls _weight_generation(self._weights) to identify the archive version
2. Worker initialization: Creates a FineTuneWorker if tuning data is present, otherwise proceeds to direct binding
3. Native loading: Invokes needle_load through the ctypes handle to memory-map the archive

The needle_load function exposes the weight archive directly to the native engine, eliminating Python-side copies. During inference calls (complete, run, embed), the engine references this already-loaded memory buffer, ensuring that the RAM usage remains constant at approximately 28 MiB regardless of input size.

self._generation = _weight_generation(self._weights) if self._weights else 2
...
if self._weights:
    self._worker = FineTuneWorker(... )     # fine-tune path

else:
    self._bind()                           # plain engine path

Practical Implementation Examples

The following patterns demonstrate how to leverage Needle's weight management in production code:

Load the default model with baked-in weights, allowing the engine to auto-fetch the correct native library:

import needle

agent = needle.Needle()
print(agent.complete("What can you do?")["completion"])

Load a fine-tuned .cact archive exported from custom training, which automatically triggers generation detection:

agent_ft = needle.Needle(weights="my_needle.cact", tools=[...])
print(agent_ft.run("Schedule a meeting tomorrow")["results"])

Inspect the generation of any archive file manually using the internal helper:

from needle import _weight_generation
gen = _weight_generation("my_needle.cact")
print(f"The archive is for Needle generation {gen}")

Summary

• Single-archive design: All base weights and LoRA adapters reside in one .cact binary, eliminating external dependencies during inference.
• Generation tagging: The first four bytes of the archive determine whether the engine loads v2 or v3 native libraries via the _weight_generation function.
• Hierarchical library resolution: The _library_path function searches environment variables, package directories, and ~/.cache/cactus-needle before downloading missing binaries.
• Zero-copy execution: The needle_load function maps the archive directly into native memory via ctypes, maintaining a constant ~28 MiB footprint across all inference calls.

Frequently Asked Questions

What is the .cact file format in Needle?

The .cact format is a proprietary binary archive that encapsulates the entire model state—including base parameters and optional LoRA fine-tuning weights—into a single file. According to the cactus-compute/needle source code, the format prefixes the weight data with a 4-byte little-endian generation tag that identifies which engine binary (v2 or v3) can interpret the archive.

How does Needle handle fine-tuned or LoRA weights?

When the Needle constructor receives a weights argument, it instantiates a FineTuneWorker from needle/_worker.py if fine-tuning data is detected. This worker merges the LoRA adapter with the base weights into a unified archive before the native needle_load function processes it, preserving the single-file execution model while supporting parameter-efficient fine-tuning.

Where does Needle store the native engine libraries?

The _library_path function implements a tiered search strategy: it first checks the NEEDLE{generation}_LIB_PATH environment variables, then the package installation directory, followed by the user cache at ~/.cache/cactus-needle/v{generation}/{version}/. If the library is absent, the system downloads the correct version via needle/agent/fetch.py and stores it in that cache location for future runs.

What is the memory footprint during inference?

The Needle inference engine maintains a constant memory footprint of approximately 28 MiB regardless of prompt length or tool complexity. This stability comes from the zero-copy weight loading mechanism: once needle_load maps the .cact archive into memory, subsequent calls to needle_complete or needle_embed reference that same buffer without allocating additional weight storage.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}