How to Fine-Tune Needle with LoRA and Build a Tuned Archive

To fine-tune Needle with LoRA, you generate a dataset, train a LoRA adapter using needle finetune, and export a merged .cact archive with needle export --lora for efficient inference.

Needle, the inference engine from the cactus-compute/needle repository, supports parameter-efficient fine-tuning through Low-Rank Adaptation (LoRA). This workflow allows you to specialize a frozen base model for specific tasks without touching the original weights, then bake the adaptations into a portable .cact archive. The entire process uses three CLI commands and keeps the tokenizer, confidence head, and runtime engine untouched.

Prepare a JSONL Dataset

Needle expects training data in JSON Lines format, where each line contains a "prompt" and a "completion" field.

You can generate synthetic training data directly through the CLI:

needle generate --count 1000 --output data.jsonl

Alternatively, create the file manually with your own examples. The data capture logic and schema validation reside in needle/environments/data_capture.py.

Fine-Tune with LoRA

The needle finetune command trains a LoRA adapter on top of a frozen base checkpoint. According to the source code in needle/model/finetune.py, the finetune_local function implements the training loop, while argument parsing lives in needle/cli.py.

Key parameters for the fine-tuning command:

• --lora-rank: Rank of the LoRA adapters (default: 16)
• --lora-alpha: Scaling factor for updates (default: 32.0)
• --checkpoint: Base model identifier or path (auto-downloads from Hugging Face if needed)
• --output: Destination for the trained adapter weights (default: adapter.pt)
• --epochs: Number of training epochs (default: 3)
• --batch-size: Mini-batch size (default: 8)

Example command to fine-tune Needle with LoRA:

needle finetune \
  --checkpoint mistralai/Mistral-7B-v0.1 \
  --data data.jsonl \
  --lora-rank 16 \
  --lora-alpha 32.0 \
  --epochs 3 \
  --output my_adapter.pt

During this phase, the base model weights remain frozen; only the low-rank adapter matrices are updated. Progress displays through a text progress bar, and the resulting .pt file contains only the LoRA parameters.

Build the Tuned .cact Archive

After training, merge the LoRA adapter into the base weights and export a unified archive. The export logic in needle/model/export.py handles the weight merging and serialization.

Run the export command with the --lora flag:

needle export \
  --checkpoint mistralai/Mistral-7B-v0.1 \
  --lora my_adapter.pt \
  --output tuned_model.cact

The resulting tuned_model.cact contains a single weight blob with the LoRA adaptations baked in. This eliminates runtime overhead during inference, as the engine loads one unified file rather than separate base and adapter weights.

Verify and Deploy the Archive

Test the tuned model using the run command to confirm the adapter behavior:

needle run \
  --weights tuned_model.cact \
  --prompt "Explain the benefits of LoRA fine-tuning."

If the output reflects your domain-specific training, deploy the archive by moving tuned_model.cact into any compatible Needle runtime, such as the Playground server (needle/playground/server.py). The archive requires no additional build steps and works in air-gapped environments because all weights are self-contained.

Summary

• Generate data using needle generate or provide a manual JSONL file with "prompt" and "completion" fields.
• Train the adapter with needle finetune, specifying --lora-rank and --lora-alpha (defaults are 16 and 32.0).
• Export the archive using needle export --lora <adapter> to merge weights into a .cact file.
• Deploy seamlessly by loading the .cact archive into any Needle inference runtime.

Frequently Asked Questions

What file contains the core LoRA training implementation?

The finetune_local function in needle/model/finetune.py contains the core training loop that freezes base weights and updates only the LoRA parameters. The CLI interface that exposes --lora-rank and --lora-alpha flags resides in needle/cli.py.

Does the base model get modified during LoRA fine-tuning?

No. The base checkpoint remains frozen throughout the process. Only the low-rank adapter matrices are trained and stored separately in a .pt file. The needle export command merges these adapters into the base weights only during the archive building phase.

What are the default LoRA hyperparameters in Needle?

The defaults are rank 16 (--lora-rank 16) and alpha 32.0 (--lora-alpha 32.0). These values provide a balance between adaptation capacity and memory efficiency for most fine-tuning tasks.

Can I use the tuned archive without the original base model?

Yes. The .cact archive produced by needle export --lora contains merged weights and is fully self-contained. You can load it directly with needle run --weights tuned_model.cact without requiring access to the original Hugging Face checkpoint or the separate adapter file.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}