How to Fine-Tune a Needle Model with LoRA: Complete CLI Guide

The command to fine-tune a Needle model with LoRA is needle finetune my_data.jsonl, which trains Low-Rank Adaptation adapters on your JSONL dataset while keeping the base model weights frozen.

Needle, the transformer library developed by cactus-compute, provides native support for parameter-efficient fine-tuning via the finetune sub-command. This implementation allows you to adapt large models to custom tasks without updating billions of base parameters, significantly reducing GPU memory requirements and storage costs.

The Fine-Tune Command Syntax

The basic syntax follows this pattern:

needle finetune <dataset.jsonl> [OPTIONS]

According to the argument parser in needle/cli.py (lines 31-50), the command supports the following key options:

• --checkpoint: Path to the base model checkpoint (default: checkpoints/needle2.pkl)
• --lora-rank: Dimension of the low-rank matrices, r (default: 16)
• --lora-alpha: Scaling parameter, α (default: 32)
• --epochs: Number of training epochs
• --batch-size: Training batch size
• --lr: Peak learning rate for the AdamW optimizer
• --out: Destination path for the saved adapter (default: needle_lora.pkl)

Inside the LoRA Implementation

The fine-tuning workflow is implemented in needle/model/finetune.py, which orchestrates the training pipeline through seven distinct stages:

Loading the Base Checkpoint

The process begins by loading the frozen base parameters. In needle/model/finetune.py (lines 106-108), the code resolves the checkpoint path:

base_path = args.checkpoint or DEFAULT_BASE

If you do not specify --checkpoint, Needle defaults to checkpoints/needle2.pkl.

Preparing the Training Data

Your JSONL file is processed through the _encode and load_jsonl functions (lines 215-252), which handle tokenization, sequence padding, and attention masking to create batched training examples compatible with the model's expected input format.

Targeting Attention Projection Matrices

The lora_target_paths function (lines 54-62) automatically scans the model's parameter tree to identify attention projection kernels. By default, it targets standard transformer attention components such as q_proj, k_proj, v_proj, and o_proj weight matrices.

Initializing Low-Rank Adapters

For each identified target matrix, the init_lora function (lines 66-83) initializes two small matrices: A (initialized with random values) and B (initialized to zero). These matrices have dimensions (d, r) and (r, d) respectively, where r is your configured --lora-rank. During training, only these low-rank matrices receive gradient updates.

Scaling and Training Loop

The effective adaptation strength is controlled by the scaling factor alpha / rank (lines 130-132). With default values of --lora-alpha 32 and --lora-rank 16, the scale factor is 2.0. The training loop uses Optax's AdamW optimizer with a cosine-decay learning rate schedule, strictly updating only the LoRA parameters while the original base weights remain frozen.

Saving the Adapter

Upon completion, the adapter state—including the A and B matrices for all targets and the computed scale factor—is serialized to the path specified by --out (lines 89-99). This produces a compact .pkl file containing only the trainable parameters, typically orders of magnitude smaller than the full model checkpoint.

Practical Code Examples

Basic Fine-Tuning

Run fine-tuning with default LoRA settings (rank 16, alpha 32):

needle finetune my_data.jsonl --checkpoint checkpoints/needle2.pkl

Advanced Configuration

Customize hyperparameters for larger rank and longer training:

needle finetune my_data.jsonl \
    --checkpoint checkpoints/needle2.pkl \
    --epochs 5 \
    --batch-size 32 \
    --lr 2e-4 \
    --lora-rank 32 \
    --lora-alpha 64 \
    --out tuned_adapter.pkl

Merging the Adapter Programmatically

To combine your trained adapter with the base model for inference:

from needle.model.finetune import merge_lora, load_checkpoint
import pickle
import jax.numpy as jnp

# Load base checkpoint

params, cfg = load_checkpoint("checkpoints/needle2.pkl")

# Load trained LoRA adapter

with open("tuned_adapter.pkl", "rb") as f:
    adapter = pickle.load(f)

# Reconstruct lora dictionary with proper key structure

lora = {
    tuple(k.split("/")): {
        "A": jnp.asarray(v["A"]), 
        "B": jnp.asarray(v["B"])
    }
    for k, v in adapter["lora"].items()
}

# Merge into base parameters

merged_params = merge_lora(params, lora, adapter["scale"])

# merged_params now contains the fine-tuned weights for inference

Key Source Files

• needle/model/finetune.py: Contains the complete LoRA implementation including lora_target_paths for matrix discovery, init_lora for adapter initialization, the Optax-based training loop, and merge_lora for combining adapters with base weights.
• needle/cli.py: Defines the finetune sub-command interface and CLI argument parsing (lines 31-50), routing execution to finetune_local.

Summary

• Use needle finetune <data.jsonl> to train LoRA adapters on custom datasets while preserving base model weights.
• Key hyperparameters --lora-rank (default 16) and --lora-alpha (default 32) control adapter capacity and scaling via the formula alpha / rank.
• Target matrices are automatically identified as attention projection layers (q_proj, k_proj, etc.) within the transformer blocks.
• Training efficiency is achieved through Optax AdamW with cosine decay, updating only the low-rank A and B matrices.
• Output artifacts are compact .pkl files that can be merged during the build step or loaded via merge_lora for inference.

Frequently Asked Questions

What is the exact command to fine-tune a Needle model with LoRA?

Run needle finetune my_data.jsonl followed by optional flags. According to needle/cli.py, this invokes the finetune_local function in needle/model/finetune.py, which handles the complete LoRA training pipeline from data loading to adapter serialization.

Which model layers does LoRA target in Needle?

The lora_target_paths function in needle/model/finetune.py (lines 54-62) automatically targets attention projection kernels including q_proj, k_proj, v_proj, and o_proj matrices. These weight matrices within the self-attention mechanism are identified by scanning the model's parameter tree for names matching the LoRA target list.

How does Needle calculate the LoRA scaling factor?

As implemented in needle/model/finetune.py (lines 130-132), the scaling factor is computed as alpha / rank using the values provided via --lora-alpha and --lora-rank. With the default values of 32 and 16 respectively, the adapter outputs are scaled by 2.0 before being added to the base model activations.

Can I merge the LoRA adapter back into the base checkpoint?

Yes. Use the merge_lora function available in needle/model/finetune.py to permanently combine the adapter weights with the base model parameters. Alternatively, you can include the adapter during the needle build step to produce a fully fine-tuned .cact archive ready for deployment.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}