# Needle 2 Engine Architecture for Loading Weights and Initialization

> Discover the Needle 2 engine architecture. Learn how it loads weights and initializes using a three-step sequence: native library resolution, weight blob loading, and engine initialization with system prompts and tool schemas.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: architecture
- Published: 2026-08-25

---

**Needle 2 uses a three-step engine initialization sequence: resolve the native shared library, load weight blobs via the `needle_load` C API, and initialize the engine with `needle_init` using system prompts and tool schemas.**

The [cactus-compute/needle](https://github.com/cactus-compute/needle) repository implements a **native inference engine** packaged as a platform-specific shared library. This architecture prioritizes runtime performance by executing tensor operations, quantization kernels, and KV-cache management in compiled C/C++ code rather than Python. Understanding how Needle 2 handles weight loading and engine initialization is essential for deploying custom models and debugging deployment issues.

---

## Three-Step Engine Initialization Sequence

The `Needle` class in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) orchestrates engine startup through a strict, ordered protocol. Each step must succeed before the next begins.

### Step 1: Resolve the Native Library

The engine binary location is determined through a cascading priority system:

- `NEEDLE_LIB_PATH` environment variable (user override)
- Bundled library within the installed package (`fetch._lib_name()`)
- User cache directory at `~/.cache/cactus-needle/<engine-version>`

If no binary exists at any location, `fetch.fetch_library` automatically downloads the correct build from the Hugging Face model hub.

```python
from needle.agent import fetch

# Force-specific version download

lib_path = fetch.fetch_library("v2.0.10", "/tmp/needle-lib")
print(lib_path)

```

The resolution logic spans lines 15-30 in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py).

### Step 2: Load Weight Blobs via needle_load

When a `Needle` instance receives a `weights` argument, the `.cact` archive is read into memory as a Python `bytes` object. This blob is passed to the engine's `needle_load` C-API function along with its byte length.

**Critical architectural constraint:** The engine accepts the weight blob **exactly once per process**. The compiled library holds this memory for the process lifetime—it cannot be unloaded, replaced, or shared between incompatible versions.

```python
from needle import Needle

# Load custom finetuned checkpoint

agent = Needle(weights="my_finetuned_model.cact")

# Subsequent agents reuse loaded weights

agent2 = Needle()  # OK—no weights argument needed

```

Attempting to create another agent with different weights raises `RuntimeError`. This one-time load pattern is implemented in lines 70-88 of [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py).

### Step 3: Initialize with needle_init

After weight loading (or skipping if using default weights), the engine initializes via `needle_init`. This C-API call receives three UTF-8 buffers:

1. **System prompt** – Base instructions for agent behavior
2. **Tool schema JSON** – Description of all callable functions
3. **Tool index file** (optional) – Additional tool routing metadata

Successful initialization registers the current `Needle` instance as the active engine. Lines 90-96 in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) handle this final step.

---

## Native Core vs. Python Wrapper Architecture

### Stateless Python Facade

The `Needle` class serves as a thin communication layer. It manages:

- Input/output buffer serialization
- Tool resolution and dispatch logic
- C-API call marshaling via `ctypes`

The actual model state—attention caches, embedding tables, generation buffers—resides entirely within the native engine.

### Runtime Execution Model

| Component | Location | Responsibility |
|-----------|----------|--------------|
| Tensor kernels | Native shared library | Matrix multiplication, attention, quantization |
| KV-cache management | Native shared library | Key-value storage and retrieval during generation |
| Tool calling | Python wrapper | JSON schema validation and function dispatch |
| Tokenization | Native shared library | Text → token conversion |

JAX/Flax is **not used at runtime**. These frameworks appear only during training and checkpoint export.

---

## Weight Format and Version Compatibility

The `.cact` file format is **tightly coupled to engine version**. Each archive contains:

- Serialized model parameters
- Quantization metadata
- Architecture configuration hashes

Loading a `.cact` file from an older Needle version fails with an explicit version-mismatch error. This prevents silent corruption from incompatible tensor layouts.

The `load_checkpoint` helper in [`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py) handles archive reading for CLI workflows, though direct `Needle(weights=...)` instantiation is preferred for programmatic use.

```python
from needle.model.run import load_checkpoint

# CLI-style checkpoint loading

params, config = load_checkpoint("model.cact")

```

---

## Complete Initialization Examples

### Default Weights with Automatic Download

```python
from needle import Needle

# No weights specified—downloads default checkpoint automatically

agent = Needle()
response = agent.complete("Write a haiku about clouds.")
print(response["text"])

```

### Custom Checkpoint with Pre-fetched Engine

```python
import pathlib
from needle.agent import fetch
from needle import Needle

# Ensure engine binary exists locally

fetch.fetch_library("v2.0.10")

# Load finetuned weights once

ckpt = pathlib.Path("experiments/run-42/final.cact")
agent = Needle(weights=str(ckpt))

# Reuse loaded weights across completions

result = agent.run("Summarize the quarterly report.")

```

### Engine Library Resolution Debugging

```python
import os
from needle import Needle

# Force specific library location

os.environ["NEEDLE_LIB_PATH"] = "/opt/needle/libneedle.so"

# Verify resolution without creating agent

from needle.__init__ import _resolve_library
path = _resolve_library()
print(f"Engine will load from: {path}")

```

---

## Source File Reference Map

| File | Key Function | Lines | Purpose |
|------|------------|-------|---------|
| [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) | Library resolution | 15-30 | `NEEDLE_LIB_PATH` → bundled → cache → download |
| [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) | Weight loading | 70-88 | `needle_load` C-API call with bytes blob |
| [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) | Engine initialization | 90-96 | `needle_init` with prompts and tool schemas |
| [`needle/agent/fetch.py`](https://github.com/cactus-compute/needle/blob/main/needle/agent/fetch.py) | `fetch_library()` | — | Hugging Face Hub download with version pinning |
| [`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py) | `load_checkpoint()` | — | `.cact` archive deserialization |
| [`needle/model/architecture.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/architecture.py) | Architecture definitions | — | Transformer config used by native engine |

---

## Summary

- **Three-step initialization:** resolve native library → load weight blob via `needle_load` → initialize engine via `needle_init`
- **One-time weight load:** Per-process constraint enforced by native engine; subsequent agents reuse loaded weights
- **Native execution:** Core inference runs in compiled C/C++ library, not Python or JAX/Flax
- **Version-locked format:** `.cact` archives are tied to specific engine versions for safety
- **Automatic fetching:** Missing engine binaries download from Hugging Face Hub via `fetch.fetch_library`

---

## Frequently Asked Questions

### How do I load multiple different models in the same Python process?

You cannot. Needle 2's native engine loads weights exactly once per process via `needle_load`. Attempting to create a second `Needle` instance with different weights raises `RuntimeError`. Use separate processes or containers for multi-model deployments.

### Where does Needle download the engine binary if it's not present?

The `fetch.fetch_library` function in [`needle/agent/fetch.py`](https://github.com/cactus-compute/needle/blob/main/needle/agent/fetch.py) retrieves the correct build from the Hugging Face model hub, caching it in `~/.cache/cactus-needle/<engine-version>`. Override this with the `NEEDLE_LIB_PATH` environment variable.

### Can I unload weights to free memory?

No. The native engine holds the weight blob in allocated memory for the process lifetime. There is no `needle_unload` API. To release memory, terminate the Python process and start fresh.

### What happens if my `.cact` file was created with an older Needle version?

The engine detects version mismatches during `needle_load` and raises an error with a clear message. You must re-export or re-download checkpoints matching your installed engine version.