How to Merge LoRA Adapters and Build a Tuned .cact File

To merge LoRA adapters and build a tuned .cact file, use finetune_local in needle/model/finetune.py to train a low-rank adapter on your dataset, then run build_main to merge the adapter weights into the base model and export the combined parameters to a quantized .cact binary using write_export.

The cactus-compute/needle repository implements a lightweight JAX-based inference engine that supports parameter-efficient fine-tuning through Low-Rank Adaptation (LoRA). When you merge LoRA adapters and build a tuned .cact file, you preserve the original base checkpoint while creating a standalone, compressed binary that contains both the model weights and fine-tuned knowledge, ready for immediate inference without additional dependencies.

Step 1 – Train a LoRA Adapter with finetune_local

The first phase involves creating a low-rank adapter that learns task-specific updates without modifying the base model parameters. In needle/model/finetune.py, the finetune_local function orchestrates this process by loading the base checkpoint, initializing LoRA matrices for target weight groups, and executing a JAX training loop.

The function targets attention projection layers—including q_proj, k_proj, v_proj, gate_proj, and out_proj—and constructs "A" and "B" matrices for each. According to the source code around lines 67–84, the adapter initialization creates these low-rank decomposition matrices with configurable rank and alpha scaling. The training loop then runs for the specified number of epochs, optimizing only the adapter parameters while keeping the base weights frozen.

To train an adapter, prepare a JSONL dataset and invoke the training pipeline:

from needle.model.finetune import finetune_local, argparse

parser = argparse.ArgumentParser()
parser.add_argument("--jsonl_path", required=True)
parser.add_argument("--checkpoint_dir", default="ckpt")
parser.add_argument("--lora_rank", type=int, default=8)
parser.add_argument("--lora_alpha", type=float, default=16)
parser.add_argument("--epochs", type=int, default=3)
parser.add_argument("--batch_size", type=int, default=32)
args = parser.parse_args([
    "--jsonl_path", "needle_data.jsonl",
    "--checkpoint_dir", "ckpt",
    "--lora_rank", "8",
    "--lora_alpha", "16",
    "--epochs", "3",
    "--batch_size", "32",
])

finetune_local(args)

This script loads the default base checkpoint from checkpoints/needle2.pkl (via load_checkpoint in needle/model/run.py), initializes LoRA adapters with rank 8 and alpha 16, and serializes the trained adapter to ckpt/needle_lora.pkl using pickle. The output .pkl file contains only the low-rank matrices, keeping the storage footprint minimal.

Step 2 – Merge the Adapter and Export to .cact

Once you have a trained adapter, the build_main function in finetune.py handles both merging and export. This function loads the base checkpoint and the adapter pickle file, applies the low-rank updates to reconstruct the fine-tuned weights, and writes the result to the compact .cact format.

The merging process occurs in three stages:

1. Load components: build_main restores the base weights from the checkpoint path and deserializes the LoRA adapter from the .pkl file (lines 12–18).
2. Apply LoRA updates: The merge_lora function combines the base weights with the scaled low-rank matrices (B×A×alpha/rank), producing updated parameters that incorporate the fine-tuned knowledge.
3. Quantize and export: write_export from needle/model/export.py serializes the merged weights, configuration, tokenizer, and KV-window metadata into a single binary, applying 4-bit quantization by default to minimize file size.

Execute the build process with the following configuration:

from needle.model.finetune import build_main, argparse

parser = argparse.ArgumentParser()
parser.add_argument("--checkpoint", default="checkpoints/needle2.pkl")
parser.add_argument("--lora", required=True)
parser.add_argument("--out", default="needle_finetuned.cact")
parser.add_argument("--bits", default="4")
args = parser.parse_args([
    "--checkpoint", "checkpoints/needle2.pkl",
    "--lora", "ckpt/needle_lora.pkl",
    "--out", "needle_finetuned.cact",
    "--bits", "4",
])

build_main(args)

The --bits parameter controls quantization precision; the default 4-bit format reduces storage requirements while maintaining inference quality. After completion, build_main reports the final file size and tensor count (lines 32–34), confirming that the .cact binary contains all necessary components for standalone execution.

Step 3 – Load the Tuned Binary for Inference

The resulting .cact file is self-contained and requires no external checkpoint or adapter files. The Needle class in the main package automatically restores the tokenizer, model weights, and KV-cache configuration from this binary.

To run inference with your fine-tuned model:

from needle import Needle

model = Needle(
    weights="needle_finetuned.cact",
    tools=[...]  # JSON schema definitions for tool use

)

result = model("What are the top-3 most recent issues in the repo?")
print(result)

Because the .cact format bundles the tokenizer and architecture configuration alongside the weights, the initialization process requires only the single binary path. This design eliminates dependency management issues and ensures reproducible inference across different environments.

Summary

• LoRA training uses finetune_local in needle/model/finetune.py to create low-rank adapter matrices for attention projections, storing the result in a pickle file.
• Merging occurs in build_main, which calls merge_lora to combine adapter updates with base weights before exporting.
• Export via write_export in needle/model/export.py produces a quantized .cact binary containing weights, tokenizer, and metadata.
• The final .cact file operates as a standalone artifact, loaded directly by the Needle class without requiring the original checkpoint or separate adapter files.

Frequently Asked Questions

What is the difference between a .pkl adapter and a .cact file?

The .pkl file contains only the LoRA adapter parameters—the low-rank A and B matrices trained during fine-tuning. It requires the base checkpoint to function and is used as an intermediate artifact. The .cact file is a merged, self-contained binary that includes the full model weights (base + merged adapter), tokenizer vocabulary, and configuration metadata, making it suitable for deployment and inference without external dependencies.

Can I merge multiple LoRA adapters into a single .cact file?

The current build_main implementation in needle/model/finetune.py is designed to merge one adapter at a time with the base weights. To combine multiple adapters, you would need to sequentially load and merge them before the final export, or modify the merge_lora logic to accept multiple adapter paths. Each merge operation produces a new weight set that could theoretically serve as the base for subsequent adapter merges.

What quantization options are available when building .cact files?

The write_export function supports configurable bit-width quantization via the --bits parameter in build_main. The default configuration uses 4-bit quantization to balance model size and inference accuracy. While the codebase implements 4-bit packing by default, the export architecture in needle/model/export.py can be extended to support alternative bit depths by modifying the quantization routine passed to write_export.

Do I need the original base checkpoint to run the merged .cact file?

No. Once you have merged the LoRA adapter and exported the .cact binary, the original base checkpoint and adapter .pkl are no longer required. The .cact format embeds all necessary tensors, tokenizer data, and KV-window specifications, allowing the Needle class to initialize directly from the single binary file for inference.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}