How to Run a Fine-Tuned Needle Model: Complete Guide to LoRA Export and Inference

To run a fine-tuned Needle model, generate a training dataset, fine-tune the base checkpoint using LoRA adapters with needle finetune, merge the adapter into a self-contained .cact binary using needle build, and load it via needle.Needle(weights="file.cact") for inference.

The cactus-compute/needle repository provides a lightweight framework for fine-tuning small language models with tool-calling capabilities. Running a fine-tuned Needle model requires moving through three distinct phases—data preparation, adapter training, and binary export—before you can instantiate the runtime with your custom weights.

Generate Training Data

Before fine-tuning, you must produce a JSON-L file containing prompt-completion pairs that teach the model your specific tools or extraction patterns. You can write this manually or synthesize it using the built-in generator.

Synthetic Data Generation

The needle generate-data command contacts an LLM API (OpenRouter by default) to create training examples that conform to your tool schemas. The implementation resides in needle/model/finetune.py, specifically within the _openrouter and generate_examples functions.

export OPENROUTER_API_KEY=sk-...
needle generate-data --tools my_tools.json --num-samples 500 --output data.jsonl

If you already possess a partial dataset, augment it with additional synthetic examples:

needle generate-data --augment existing.jsonl --num-samples 300

Fine-Tune with LoRA

The finetune sub-command drives the training loop defined in needle/model/finetune.py. It freezes the base weights and injects trainable low-rank matrices into the projection layers.

Key Implementation Steps

• Loading: load_checkpoint in needle/model/run.py deserializes the base weights from checkpoints/needle2.pkl (default).
• Targeting: lora_target_paths scans the parameter tree for kernels named q_proj, k_proj, v_proj, and o_proj.
• Initialization: init_lora creates low-rank matrices A and B for each target.
• Merging: During forward passes, merge_lora computes scale * A·B and adds it to the frozen base weights on-the-fly.

Execute training with your desired rank and alpha:

needle finetune data.jsonl \
      --epochs 10 \
      --lora-rank 16 \
      --lora-alpha 32 \
      --batch-size 16 \
      --lr 1e-4

Upon completion, the CLI prints the adapter path and suggests the next build step:

adapter: checkpoints/needle_lora.pkl
next: needle build checkpoints/needle2.pkl --lora checkpoints/needle_lora.pkl

Build the Tuned Model

The build command (also in needle/model/finetune.py) merges the LoRA adapter back into the base checkpoint, quantizes the result, and emits a single-file .cact binary.

Build Process

1. Merge: Calls merge_lora to produce full-precision parameters.
2. Quantize: Applies the bit-width declared in the checkpoint (default 4-bit) or overrides via --bits.
3. Export: Invokes write_export from needle/model/export.py to serialize the binary.

needle build checkpoints/needle2.pkl \
      --lora checkpoints/needle_lora.pkl \
      --out my_needle.cact \
      --bits 4

The resulting my_needle.cact is self-contained and requires no external checkpoint files at runtime. Optionally upload to Hugging Face by setting NEEDLE_HF_REPO and adding --upload.

Run Inference with the Fine-Tuned Model

The inference engine loads the .cact file directly without referencing the original checkpoint. The high-level API is exposed in needle/__init__.py and implemented in needle/cli.py.

Basic Tool-Calling Example

import needle

@needle.tool
def set_lights(room: str, brightness: int):
    """Set the brightness of a room's lights."""
    return {"room": room, "brightness": brightness}

# Load the tuned model

agent = needle.Needle(
    weights="my_needle.cact",
    tools=[set_lights]
)

# Execute

response = agent.run("Dim the study lights to 30 percent")
print(response["results"])

# -> [{'room': 'study', 'brightness': 30}]

Structured Extraction

For extraction tasks, use the needle.extract helper with a Pydantic model:

from pydantic import BaseModel
import needle

class Invoice(BaseModel):
    vendor: str
    total: float
    due_date: str

invoice = needle.extract(
    "Invoice from Acme Corp, $1,200.00, due 2026-09-01",
    Invoice
)
print(invoice)

# Invoice(vendor='Acme Corp', total=1200.0, due_date='2026-09-01')

The runtime supports CPU, CUDA, and Metal backends. Download platform-specific binaries with needle download <platform> if needed.

Summary

• Data preparation: Create JSON-L manually or use needle generate-data (defined in needle/model/finetune.py) to synthesize training examples via OpenRouter.
• Fine-tuning: Run needle finetune to train LoRA adapters on projection layers using init_lora and merge_lora utilities.
• Export: Execute needle build to merge adapters, quantize weights, and write a portable .cact file via write_export in needle/model/export.py.
• Inference: Instantiate needle.Needle with the .cact path and call run() or extract() for tool-calling or structured outputs.

Frequently Asked Questions

What file format does a fine-tuned Needle model use?

Needle exports fine-tuned models as .cact files—self-contained binary archives that include the merged weights, quantization tables, and tokenizer metadata. According to the source code in needle/model/export.py, the write_export function serializes these binaries so the runtime can load them without separate checkpoint directories.

How do I specify which layers receive LoRA adapters?

The lora_target_paths function in needle/model/finetune.py automatically identifies target layers by scanning parameter names for q_proj, k_proj, v_proj, and o_proj substrings. You control the adapter capacity via --lora-rank (dimension of matrices A and B) and --lora-alpha (scaling factor), but you cannot manually exclude specific layers from the CLI.

Can I run inference on a LoRA adapter without building the .cact file?

No. The needle.Needle class expects a consolidated .cact binary. You must first run needle build to merge the LoRA adapter (produced by needle finetune) into the base checkpoint and quantize the result. The build command calls merge_lora to bake the low-rank updates into the frozen weights before export.

Does the fine-tuned model retain tool definitions from training?

No. Tool schemas are not embedded in the .cact weights. When you run needle.Needle(weights="model.cact", tools=[...]), you must explicitly pass the Python callables you want to expose. The model learns the syntax of tool calls during fine-tuning, but the runtime binds actual function implementations at load time via the tools parameter.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}