How to Load Custom Tuned Weights into the Needle Engine

Load custom tuned weights into the Needle engine by passing a .cact checkpoint path to the Needle constructor, which lazily binds the binary blob to the native C library via the _bind() method and initializes the model for inference.

The Needle engine is a lightweight Python wrapper around a compiled C library optimized for high-performance inference. When deploying fine-tuned models, the process of loading custom tuned weights into the Needle engine handles the transition from a serialized checkpoint file to an active runtime capable of executing prompts and tool calls.

The Weight Loading Architecture

The engine follows a lazy-loading pattern that defers heavy operations until the first inference call. This design minimizes startup overhead while ensuring that custom weights are properly validated and bound to the underlying native functions.

Locating the Native Library

Before any weights can be loaded, the system must resolve the appropriate compiled library binary. The _library_path() function checks three sources in order: an environment variable override (CACTUS_NEEDLE_LIBRARY), a local copy adjacent to the package installation, or a cached copy under ~/.cache/cactus-needle/<engine-version>. If none exist, the system downloads the correct binary via fetch.fetch_library() as implemented in [needle/__init__.py](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py#L13-L28). The version constant ENGINE_VERSION defined in [needle/agent/fetch.py](https://github.com/cactus-compute/needle/blob/main/needle/agent/fetch.py) determines which binary artifact to retrieve.

Instantiating the Needle Class

The Needle constructor accepts a weights parameter that specifies the filesystem path to a .cact file. The constructor stores this path in self._weights and prepares the system prompt and tool schema, but it does not immediately load the binary data. It also allocates a reusable C buffer for the engine's response. This initialization logic resides in [needle/__init__.py](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py#L52-L66).

Binding and Loading Weights

The actual loading occurs inside the _bind() method, which is triggered automatically on the first call to any inference method. If a weights file was provided and differs from the currently active set, the method opens the file in binary mode, reads the entire blob into memory, and passes it to the C function needle_load. A non-zero return code raises a RuntimeError with detailed context; otherwise, the blob becomes the active weight set. This binding mechanism is defined in [needle/__init__.py](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py#L68-L82).

Engine Initialization

After successfully loading the weights (or re-using the already-loaded set), the engine initializes the model context via needle_init. This C function receives the system prompt, a JSON-encoded tool schema, and an optional tool-index path to configure the runtime environment. The initialization sequence appears in [needle/__init__.py](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py#L83-L86).

Running Inference

The complete() method forwards prompts to needle_complete, which writes results into the pre-allocated C buffer. The Python wrapper parses this buffer into a dictionary. Notably, when custom weights are loaded, the wrapper forces the confidence field to None because the confidence head is not fine-tuned in custom checkpoints. This inference handling is implemented in [needle/__init__.py](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py#L100-L117).

Practical Code Examples

The following examples demonstrate how to load and use custom tuned weights in different scenarios.

Loading a Local Checkpoint

from needle import Needle

# Path to a .cact file produced by `needle finetune` or `needle export`

weights_path = "my_tuned_model.cact"

# Instantiate with custom weights

agent = Needle(weights=weights_path)

# Run inference; weights are loaded lazily on this first call

response = agent.complete("Summarise the plot of *Moby-Dick* in three sentences.")
print(response["content"])

Using Custom Weights with Tools

from pydantic import BaseModel
from needle import Needle, tool, Field

class WeatherForecast(BaseModel):
    location: str = Field(description="City name")
    day: str = Field(description="Day of the week")

@tool
def get_weather(forecast: WeatherForecast):
    """Return a mock weather forecast."""
    return {"location": forecast.location, "day": forecast.day, "forecast": "sunny"}

# Load tuned weights and enable tool use

agent = Needle(tools=[get_weather], weights="tuned_weather_agent.cact")

# The model can invoke get_weather during multi-step reasoning

result = agent.run("What will the weather be like in Paris tomorrow?")
print(result["results"])

Command-Line Workflow

# Build a base model checkpoint

needle build --model facebook/opt-125m --out base_model.cact

# Fine-tune on custom dataset

needle finetune --model base_model.cact --data my_dataset.json --out tuned_model.cact

# Serve with custom weights

needle serve --weights tuned_model.cact --port 8080

Key Source Files and Implementation Details

File	Purpose	Key Implementation
[needle/__init__.py](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py)	Core wrapper defining Needle class and weight loading logic	_library_path() (L13-L28), __init__() (L52-L66), _bind() (L68-L82), complete() (L100-L117)
[needle/agent/fetch.py](https://github.com/cactus-compute/needle/blob/main/needle/agent/fetch.py)	Library acquisition and version management	fetch_library(), ENGINE_VERSION
[needle/model/finetune.py](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py)	Checkpoint conversion from HuggingFace format to .cact	Export and quantization logic
[needle/cli.py](https://github.com/cactus-compute/needle/blob/main/needle/cli.py)	User-facing commands for building and serving	build, finetune, serve command handlers
[needle/playground/server.py](https://github.com/cactus-compute/needle/blob/main/needle/playground/server.py)	HTTP server demonstrating runtime weight loading	Server initialization with weights parameter

Summary

• Custom weights are distributed as binary .cact files produced by the needle finetune or needle build commands.
• Lazy loading defers the expensive needle_load C call until the first inference request, minimizing startup latency.
• Path resolution checks environment variables, local directories, and the user cache before downloading the native engine library.
• Blob validation occurs in _bind(), where the Python wrapper validates the C return code and raises descriptive exceptions on failure.
• Confidence masking is automatically applied when custom weights are detected, setting the field to None to indicate the absence of calibrated confidence scores.

Frequently Asked Questions

What is a .cact file and how is it created?

A .cact file is a serialized binary checkpoint containing quantized model weights and configuration metadata specific to the Needle engine. You create one by running needle build to convert a HuggingFace model or needle finetune to further train an existing checkpoint on custom data before exporting.

When exactly are the weights loaded into memory?

Weights are loaded lazily during the first call to an inference method such as complete() or run(). The _bind() method checks if the requested weights differ from the currently loaded set; if so, it reads the binary blob and invokes the native needle_load function before proceeding with initialization.

Can I switch weights after creating a Needle instance?

Yes, you can load a different .cact file by creating a new Needle instance with a different weights argument. The engine does not support hot-swapping weights within a single instance; instantiate a new object to ensure a clean state and proper initialization of the native context.

Why is the confidence field None when using custom tuned weights?

The confidence scoring head is not included in fine-tuned checkpoints produced by the current toolchain. When the wrapper detects that custom weights are loaded (as opposed to base model weights), it explicitly sets the confidence field to None in the response dictionary to indicate that calibrated certainty estimates are unavailable.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}