# How to Run a Fine-Tuned Needle Model: Complete Guide to LoRA Export and Inference

> Master running fine-tuned Needle models with this guide. Learn LoRA export and inference using needle finetune and needle build for efficient model deployment and quick inference.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: how-to-guide
- Published: 2026-08-28

---

**To run a fine-tuned Needle model, generate a training dataset, fine-tune the base checkpoint using LoRA adapters with `needle finetune`, merge the adapter into a self-contained `.cact` binary using `needle build`, and load it via `needle.Needle(weights="file.cact")` for inference.**

The `cactus-compute/needle` repository provides a lightweight framework for fine-tuning small language models with tool-calling capabilities. Running a fine-tuned Needle model requires moving through three distinct phases—data preparation, adapter training, and binary export—before you can instantiate the runtime with your custom weights.

## Generate Training Data

Before fine-tuning, you must produce a JSON-L file containing prompt-completion pairs that teach the model your specific tools or extraction patterns. You can write this manually or synthesize it using the built-in generator.

### Synthetic Data Generation

The `needle generate-data` command contacts an LLM API (OpenRouter by default) to create training examples that conform to your tool schemas. The implementation resides in [`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py), specifically within the `_openrouter` and `generate_examples` functions.

```bash
export OPENROUTER_API_KEY=sk-...
needle generate-data --tools my_tools.json --num-samples 500 --output data.jsonl

```

If you already possess a partial dataset, augment it with additional synthetic examples:

```bash
needle generate-data --augment existing.jsonl --num-samples 300

```

## Fine-Tune with LoRA

The `finetune` sub-command drives the training loop defined in [`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py). It freezes the base weights and injects trainable low-rank matrices into the projection layers.

### Key Implementation Steps

- **Loading**: `load_checkpoint` in [`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py) deserializes the base weights from `checkpoints/needle2.pkl` (default).
- **Targeting**: `lora_target_paths` scans the parameter tree for kernels named `q_proj`, `k_proj`, `v_proj`, and `o_proj`.
- **Initialization**: `init_lora` creates low-rank matrices **A** and **B** for each target.
- **Merging**: During forward passes, `merge_lora` computes `scale * A·B` and adds it to the frozen base weights on-the-fly.

 Execute training with your desired rank and alpha:

```bash
needle finetune data.jsonl \
      --epochs 10 \
      --lora-rank 16 \
      --lora-alpha 32 \
      --batch-size 16 \
      --lr 1e-4

```

Upon completion, the CLI prints the adapter path and suggests the next build step:

```

adapter: checkpoints/needle_lora.pkl
next: needle build checkpoints/needle2.pkl --lora checkpoints/needle_lora.pkl

```

## Build the Tuned Model

The `build` command (also in [`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py)) merges the LoRA adapter back into the base checkpoint, quantizes the result, and emits a single-file `.cact` binary.

### Build Process

1. **Merge**: Calls `merge_lora` to produce full-precision parameters.
2. **Quantize**: Applies the bit-width declared in the checkpoint (default 4-bit) or overrides via `--bits`.
3. **Export**: Invokes `write_export` from [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) to serialize the binary.

```bash
needle build checkpoints/needle2.pkl \
      --lora checkpoints/needle_lora.pkl \
      --out my_needle.cact \
      --bits 4

```

The resulting `my_needle.cact` is self-contained and requires no external checkpoint files at runtime. Optionally upload to Hugging Face by setting `NEEDLE_HF_REPO` and adding `--upload`.

## Run Inference with the Fine-Tuned Model

The inference engine loads the `.cact` file directly without referencing the original checkpoint. The high-level API is exposed in [`needle/__init__.py`](https://github.com/cactus-compute/needle/blob/main/needle/__init__.py) and implemented in [`needle/cli.py`](https://github.com/cactus-compute/needle/blob/main/needle/cli.py).

### Basic Tool-Calling Example

```python
import needle

@needle.tool
def set_lights(room: str, brightness: int):
    """Set the brightness of a room's lights."""
    return {"room": room, "brightness": brightness}

# Load the tuned model

agent = needle.Needle(
    weights="my_needle.cact",
    tools=[set_lights]
)

# Execute

response = agent.run("Dim the study lights to 30 percent")
print(response["results"])

# -> [{'room': 'study', 'brightness': 30}]

```

### Structured Extraction

For extraction tasks, use the `needle.extract` helper with a Pydantic model:

```python
from pydantic import BaseModel
import needle

class Invoice(BaseModel):
    vendor: str
    total: float
    due_date: str

invoice = needle.extract(
    "Invoice from Acme Corp, $1,200.00, due 2026-09-01",
    Invoice
)
print(invoice)

# Invoice(vendor='Acme Corp', total=1200.0, due_date='2026-09-01')

```

The runtime supports CPU, CUDA, and Metal backends. Download platform-specific binaries with `needle download <platform>` if needed.

## Summary

- **Data preparation**: Create JSON-L manually or use `needle generate-data` (defined in [`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py)) to synthesize training examples via OpenRouter.
- **Fine-tuning**: Run `needle finetune` to train LoRA adapters on projection layers using `init_lora` and `merge_lora` utilities.
- **Export**: Execute `needle build` to merge adapters, quantize weights, and write a portable `.cact` file via `write_export` in [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py).
- **Inference**: Instantiate `needle.Needle` with the `.cact` path and call `run()` or `extract()` for tool-calling or structured outputs.

## Frequently Asked Questions

### What file format does a fine-tuned Needle model use?

Needle exports fine-tuned models as **`.cact` files**—self-contained binary archives that include the merged weights, quantization tables, and tokenizer metadata. According to the source code in [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py), the `write_export` function serializes these binaries so the runtime can load them without separate checkpoint directories.

### How do I specify which layers receive LoRA adapters?

The `lora_target_paths` function in [`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py) automatically identifies target layers by scanning parameter names for `q_proj`, `k_proj`, `v_proj`, and `o_proj` substrings. You control the adapter capacity via `--lora-rank` (dimension of matrices **A** and **B**) and `--lora-alpha` (scaling factor), but you cannot manually exclude specific layers from the CLI.

### Can I run inference on a LoRA adapter without building the `.cact` file?

No. The `needle.Needle` class expects a consolidated `.cact` binary. You must first run `needle build` to merge the LoRA adapter (produced by `needle finetune`) into the base checkpoint and quantize the result. The `build` command calls `merge_lora` to bake the low-rank updates into the frozen weights before export.

### Does the fine-tuned model retain tool definitions from training?

No. Tool schemas are not embedded in the `.cact` weights. When you run `needle.Needle(weights="model.cact", tools=[...])`, you must explicitly pass the Python callables you want to expose. The model learns the *syntax* of tool calls during fine-tuning, but the runtime binds actual function implementations at load time via the `tools` parameter.