How to Handle the Weights-Agnostic Engine for Custom Checkpoints in Needle

Needle's inference engine is deliberately weights-agnostic, allowing any .cact checkpoint to execute without rebuilding the compiled binary—but once loaded, weights cannot be unloaded, requiring careful process isolation when switching between base and tuned models.

The weights-agnostic engine in Needle is a core architectural decision that separates the compiled native library from the model weights. This design lets you swap between the base model and any fine-tuned checkpoint simply by loading a different .cact archive file. Understanding how to work within the engine's constraints—specifically its single-weight lifetime and load-once behavior—is essential for building reliable applications with custom checkpoints.

---

How the Weights-Agnostic Engine Works

Needle's engine achieves flexibility through a strict separation between the shared native library and the weight blob. The library is loaded once per process via _library_path() in needle/__init__.py, while weights are loaded on-demand when constructing a Needle agent.

Key architectural constraints from the source code:

• Single-weight lifetime: The engine cannot unload a checkpoint once loaded. Attempting to create a new Needle instance with different weights raises RuntimeError("cannot unload").
• Automatic reuse: Creating additional agents with the same checkpoint path skips re-loading—the engine reuses the already-loaded blob.
• Active weight tracking: The module-level _active_weights variable tracks which checkpoint is currently in memory, used by helper functions like extract() and run().

---

Loading Custom Checkpoints Step by Step

Step 1: Obtain or Build a .cact Checkpoint

Use the CLI to merge a LoRA adapter into a base model or download a pre-built checkpoint:

# Build a tuned checkpoint from base + LoRA

needle build base.pkl --lora tuned.pkl --out my_model.cact

# Or download from Hugging Face

needle download acme/needle2/my_model.cact --out ./weights

The build command (implemented in needle/cli.py lines 68-89) produces a single-file archive that the engine can directly ingest. The download command (lines 21-34) uses hf_hub_download() to cache files locally.

Step 2: Instantiate an Agent with Custom Weights

Pass the path to your .cact file via the weights argument:

import needle

agent = needle.Needle(
    weights="my_model.cact",
    tools=[my_tool]
)

This triggers Needle._bind() in needle/__init__.py, which validates the weight-loading constraints:

# From needle/__init__.py lines 70-90 (approximate)

if not self._weights and _active_weights:
    raise RuntimeError(f"{_active_weights} is loaded and the engine cannot unload it …")

if self._weights and _active_weights != self._weights:
    with open(self._weights, "rb") as handle:
        blob = handle.read()
    _active_blob = blob
    _lib().needle_load(blob, len(blob))
    _active_weights = self._weights

If no conflicting weights are loaded, the blob is read and passed to the native needle_load() function.

Step 3: Run Inference or Extract Structured Data

Once loaded, both run() and extract() automatically use the active checkpoint:

# Standard conversational response

response = agent.run("What's the weather in Paris?")

# Structured extraction without explicit weights

from pydantic import BaseModel

class Invoice(BaseModel):
    total: float
    vendor: str

invoice = needle.extract(raw_text, Invoice)

The extract() function defaults to _active_weights, so you don't need to pass the path again after the first load.

---

Sharing Checkpoints Across Multiple Agents

Creating multiple Needle instances with the same checkpoint path triggers zero additional loads. This is verified by test_tuned_agent_rebinds_without_reloading_its_own_weights in tests/test_weights.py:

agent_a = needle.Needle(weights="my_model.cact")
agent_b = needle.Needle(weights="my_model.cact")  # Reuses loaded weights

# Both agents point to the same underlying blob

print(needle._active_weights)  # → 'my_model.cact'

The test confirms this behavior by asserting engine.count("load") == 1 after multiple agent instantiations (see tests/test_weights.py lines 78-84).

---

Critical Constraint: No Mixed Weights in One Process

The engine cannot switch from tuned to base weights (or vice versa) within the same process. Attempting this raises RuntimeError as demonstrated in test_new_base_agent_refuses_once_tuned_weights_are_loaded:

# This sequence FAILS:

tuned = needle.Needle(weights="my_model.cact")
base = needle.Needle(tools=[])  # RuntimeError: cannot unload

# Solution: Use separate processes for different weight sets

Design pattern: Isolate each weight configuration in its own process or container. This aligns with the engine's "load-once" semantics and prevents runtime failures.

---

Debugging and Verification

Inspect the currently active checkpoint for troubleshooting:

print(needle._active_weights)  # Path to loaded .cact, or None

For the web UI playground, see needle/playground/server.py for an example of loading custom weights in a server context.

---

Summary

• The Needle engine is weights-agnostic: Any .cact file runs on the same compiled binary without rebuilding.
• Load-once constraint: Once a checkpoint is loaded, it cannot be unloaded—plan for process-per-weight-set isolation.
• Automatic deduplication: Multiple agents with identical weights paths share the same in-memory blob with no reload penalty.
• CLI integration: Use needle build to create tuned checkpoints and needle download to fetch remote archives.
• Helper functions inherit: extract() and run() default to _active_weights after initial loading.

---

Frequently Asked Questions

How do I switch from a fine-tuned checkpoint back to the base model?

You cannot switch checkpoints within the same process. Start a fresh Python process or container without specifying weights to load the base model, or explicitly pass your base .cact file in a new process. The engine raises RuntimeError("cannot unload") if you attempt to load different weights after a checkpoint is active.

Can I load multiple different checkpoints simultaneously?

No. Needle's native library maintains a single global weight blob. The test suite explicitly verifies this limitation in test_weights.py. Run separate processes if you need to serve multiple model variants.

What happens if I create two agents with the same weights path?

The second agent reuses the already-loaded weights without file I/O or native library calls. This is confirmed by test_tuned_agent_rebinds_without_reloading_its_own_weights, which asserts only one needle_load operation occurs.

How do I verify which checkpoint is currently loaded?

Check needle._active_weights—it contains the filesystem path to the loaded .cact archive, or None if no weights have been loaded yet.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}