Needle 2 Engine Architecture for Loading Weights and Initialization

Needle 2 uses a three-step engine initialization sequence: resolve the native shared library, load weight blobs via the needle_load C API, and initialize the engine with needle_init using system prompts and tool schemas.

The cactus-compute/needle repository implements a native inference engine packaged as a platform-specific shared library. This architecture prioritizes runtime performance by executing tensor operations, quantization kernels, and KV-cache management in compiled C/C++ code rather than Python. Understanding how Needle 2 handles weight loading and engine initialization is essential for deploying custom models and debugging deployment issues.

---

Three-Step Engine Initialization Sequence

The Needle class in needle/__init__.py orchestrates engine startup through a strict, ordered protocol. Each step must succeed before the next begins.

Step 1: Resolve the Native Library

The engine binary location is determined through a cascading priority system:

• NEEDLE_LIB_PATH environment variable (user override)
• Bundled library within the installed package (fetch._lib_name())
• User cache directory at ~/.cache/cactus-needle/<engine-version>

If no binary exists at any location, fetch.fetch_library automatically downloads the correct build from the Hugging Face model hub.

from needle.agent import fetch

# Force-specific version download

lib_path = fetch.fetch_library("v2.0.10", "/tmp/needle-lib")
print(lib_path)

The resolution logic spans lines 15-30 in needle/__init__.py.

Step 2: Load Weight Blobs via needle_load

When a Needle instance receives a weights argument, the .cact archive is read into memory as a Python bytes object. This blob is passed to the engine's needle_load C-API function along with its byte length.

Critical architectural constraint: The engine accepts the weight blob exactly once per process. The compiled library holds this memory for the process lifetime—it cannot be unloaded, replaced, or shared between incompatible versions.

from needle import Needle

# Load custom finetuned checkpoint

agent = Needle(weights="my_finetuned_model.cact")

# Subsequent agents reuse loaded weights

agent2 = Needle()  # OK—no weights argument needed

Attempting to create another agent with different weights raises RuntimeError. This one-time load pattern is implemented in lines 70-88 of needle/__init__.py.

Step 3: Initialize with needle_init

After weight loading (or skipping if using default weights), the engine initializes via needle_init. This C-API call receives three UTF-8 buffers:

1. System prompt – Base instructions for agent behavior
2. Tool schema JSON – Description of all callable functions
3. Tool index file (optional) – Additional tool routing metadata

Successful initialization registers the current Needle instance as the active engine. Lines 90-96 in needle/__init__.py handle this final step.

---

Native Core vs. Python Wrapper Architecture

Stateless Python Facade

The Needle class serves as a thin communication layer. It manages:

• Input/output buffer serialization
• Tool resolution and dispatch logic
• C-API call marshaling via ctypes

The actual model state—attention caches, embedding tables, generation buffers—resides entirely within the native engine.

Runtime Execution Model

Component	Location	Responsibility
Tensor kernels	Native shared library	Matrix multiplication, attention, quantization
KV-cache management	Native shared library	Key-value storage and retrieval during generation
Tool calling	Python wrapper	JSON schema validation and function dispatch
Tokenization	Native shared library	Text → token conversion

JAX/Flax is not used at runtime. These frameworks appear only during training and checkpoint export.

---

Weight Format and Version Compatibility

The .cact file format is tightly coupled to engine version. Each archive contains:

• Serialized model parameters
• Quantization metadata
• Architecture configuration hashes

Loading a .cact file from an older Needle version fails with an explicit version-mismatch error. This prevents silent corruption from incompatible tensor layouts.

The load_checkpoint helper in needle/model/run.py handles archive reading for CLI workflows, though direct Needle(weights=...) instantiation is preferred for programmatic use.

from needle.model.run import load_checkpoint

# CLI-style checkpoint loading

params, config = load_checkpoint("model.cact")

---

Complete Initialization Examples

Default Weights with Automatic Download

from needle import Needle

# No weights specified—downloads default checkpoint automatically

agent = Needle()
response = agent.complete("Write a haiku about clouds.")
print(response["text"])

Custom Checkpoint with Pre-fetched Engine

import pathlib
from needle.agent import fetch
from needle import Needle

# Ensure engine binary exists locally

fetch.fetch_library("v2.0.10")

# Load finetuned weights once

ckpt = pathlib.Path("experiments/run-42/final.cact")
agent = Needle(weights=str(ckpt))

# Reuse loaded weights across completions

result = agent.run("Summarize the quarterly report.")

Engine Library Resolution Debugging

import os
from needle import Needle

# Force specific library location

os.environ["NEEDLE_LIB_PATH"] = "/opt/needle/libneedle.so"

# Verify resolution without creating agent

from needle.__init__ import _resolve_library
path = _resolve_library()
print(f"Engine will load from: {path}")

---

Source File Reference Map

File	Key Function	Lines	Purpose
needle/__init__.py	Library resolution	15-30	NEEDLE_LIB_PATH → bundled → cache → download
needle/__init__.py	Weight loading	70-88	needle_load C-API call with bytes blob
needle/__init__.py	Engine initialization	90-96	needle_init with prompts and tool schemas
needle/agent/fetch.py	fetch_library()	—	Hugging Face Hub download with version pinning
needle/model/run.py	load_checkpoint()	—	.cact archive deserialization
needle/model/architecture.py	Architecture definitions	—	Transformer config used by native engine

---

Summary

• Three-step initialization: resolve native library → load weight blob via needle_load → initialize engine via needle_init
• One-time weight load: Per-process constraint enforced by native engine; subsequent agents reuse loaded weights
• Native execution: Core inference runs in compiled C/C++ library, not Python or JAX/Flax
• Version-locked format: .cact archives are tied to specific engine versions for safety
• Automatic fetching: Missing engine binaries download from Hugging Face Hub via fetch.fetch_library

---

Frequently Asked Questions

How do I load multiple different models in the same Python process?

You cannot. Needle 2's native engine loads weights exactly once per process via needle_load. Attempting to create a second Needle instance with different weights raises RuntimeError. Use separate processes or containers for multi-model deployments.

Where does Needle download the engine binary if it's not present?

The fetch.fetch_library function in needle/agent/fetch.py retrieves the correct build from the Hugging Face model hub, caching it in ~/.cache/cactus-needle/<engine-version>. Override this with the NEEDLE_LIB_PATH environment variable.

Can I unload weights to free memory?

No. The native engine holds the weight blob in allocated memory for the process lifetime. There is no needle_unload API. To release memory, terminate the Python process and start fresh.

What happens if my .cact file was created with an older Needle version?

The engine detects version mismatches during needle_load and raises an error with a clear message. You must re-export or re-download checkpoints matching your installed engine version.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}