How to Perform LoRA Fine-Tuning with Cactus-Needle: A Complete Guide

The fastest way to fine-tune large language models with LoRA in Cactus-Needle is through the needle finetune CLI command or the Python API in needle/model/finetune.py, which handles adapter injection, frozen base weights, and efficient training in under 50 lines of code.

Cactus-Needle provides a modular, production-ready pipeline for LoRA fine-tuning that reduces GPU memory requirements by freezing original model parameters and training only low-rank adapter matrices. This guide walks through the complete workflow using the actual source implementation.

LoRA Fine-Tuning Architecture in Needle

The Cactus-Needle codebase separates concerns across specialized modules. Understanding this structure helps you customize training for your use case.

Core Pipeline Components

• needle/model/finetune.py — Orchestrates the entire LoRA workflow: adapter creation, training loop, checkpointing, and weight serialization
• needle/model/run.py — Loads base models via Hugging Face's AutoModelForCausalLM with optional quantization
• needle/model/quantize.py — Enables 8-bit and 4-bit quantization for memory-constrained environments
• needle/model/tokenizer.py — Prepares tokenized datasets for the training loop
• needle/model/export.py — Merges trained LoRA weights back into the base model for deployment
• needle/cli.py — Exposes the needle finetune command-line interface

Step 1: Load and Prepare the Base Model

The load_base_model function in needle/model/finetune.py wraps the generic loader from needle/model/run.py. It supports quantization through needle/model/quantize.py to fit large models on consumer GPUs.

from needle.model.finetune import load_base_model

model = load_base_model(
    repo_id="EleutherAI/gpt-neo-125M",
    quantize=False,  # Set True for 8-bit via quantize.py

)

When quantize=True, the loader applies compression algorithms from needle/model/quantize.py before LoRA adapter injection.

Step 2: Inject LoRA Adapters

The create_lora_modules function builds low-rank linear layers (LoRALinear) that wrap target weight matrices. By default, this targets the query and value projections where adaptation is most effective.

from needle.model.finetune import create_lora_modules

model = create_lora_modules(
    model,
    lora_rank=8,          # Lower rank = fewer parameters, faster training

    lora_alpha=16,        # Scaling factor for adapter outputs

    target_modules=["q_proj", "v_proj"],  # Attention projections to adapt

)

This step automatically freezes all original model parameters (requires_grad=False), leaving only the LoRA weights trainable.

Step 3: Prepare Your Dataset

The prepare_dataset utility handles tokenization through needle/model/tokenizer.py and returns a PyTorch DataLoader.

from needle.model.finetune import prepare_dataset

train_loader = prepare_dataset(
    tokenized_path="data/train_tokenized.jsonl",
    batch_size=4,
    max_length=512,
)

Input files should contain JSON lines with input_ids and attention_mask fields, or you can pass a raw Hugging Face datasets.Dataset object.

Step 4: Execute the Training Loop

The train_lora function implements gradient accumulation, learning rate scheduling, and telemetry logging via needle/_telemetry.py.

from needle.model.finetune import train_lora

train_lora(
    model=model,
    dataloader=train_loader,
    epochs=3,
    learning_rate=5e-5,
    gradient_accumulation_steps=4,  # Effective batch size = 4 × 4 = 16

)

Training metrics are emitted through needle/_telemetry.py for monitoring in Weights & Biases or TensorBoard.

Step 5: Save and Export LoRA Weights

After fine-tuning, persist adapter weights separately or merge them for inference-only deployment.

from needle.model.finetune import save_lora
from needle.model.export import merge_lora

# Save adapter weights only (small checkpoint)

save_lora(model, output_dir="lora_weights/")

# Or merge into base model for standalone inference

merged_model = merge_lora(model)

The merge_lora function in needle/model/export.py computes W_merged = W_base + (alpha/rank) * B * A for each adapted layer, eliminating inference overhead.

Complete Python API Example

from needle.model.finetune import (
    load_base_model,
    create_lora_modules,
    prepare_dataset,
    train_lora,
    save_lora,
)

# Load base model

model = load_base_model("EleutherAI/gpt-neo-125M", quantize=False)

# Attach LoRA adapters

model = create_lora_modules(
    model,
    lora_rank=8,
    lora_alpha=16,
    target_modules=["q_proj", "v_proj"],
)

# Prepare data

loader = prepare_dataset("data/train.jsonl", batch_size=4, max_length=512)

# Train

train_lora(model, loader, epochs=3, learning_rate=5e-5)

# Save

save_lora(model, "lora_weights/")

CLI Quick Start

The needle finetune command in needle/cli.py exposes all parameters through the FinetuneConfig dataclass:

needle finetune \
  --model EleutherAI/gpt-neo-125M \
  --dataset data/train_tokenized.jsonl \
  --lora-rank 8 \
  --lora-alpha 16 \
  --target-modules q_proj,v_proj \
  --epochs 3 \
  --lr 5e-5 \
  --batch-size 4 \
  --gradient-accumulation-steps 4 \
  --output-dir lora_weights/

Hyperparameter Guidelines for LoRA Fine-Tuning

Parameter	Typical Range	Effect
lora_rank	4–64	Higher rank = more capacity, more parameters
lora_alpha	2× rank	Scaling factor; higher values amplify adapter influence
target_modules	["q_proj", "v_proj"] or all linear layers	More targets = more trainable parameters
learning_rate	1e-4 to 1e-5	LoRA typically uses 2–10× higher LR than full fine-tuning
batch_size × gradient_accumulation_steps	16–128	Larger effective batches stabilize training

Summary

• needle/model/finetune.py provides the complete LoRA implementation: load_base_model, create_lora_modules, train_lora, and save_lora
• Base model weights remain frozen; only low-rank adapter matrices train, reducing memory by 10–100×
• Quantization via needle/model/quantize.py enables training on single consumer GPUs
• The needle finetune CLI maps directly to Python API parameters for reproducible workflows
• Use needle/model/export.py to merge adapters for inference or keep separate for multi-task serving

Frequently Asked Questions

What hardware is required for LoRA fine-tuning with Cactus-Needle?

With 8-bit quantization enabled in needle/model/quantize.py, you can fine-tune 7B parameter models on a single 24GB GPU. Without quantization, plan for approximately 2× the model size in VRAM for fp16 training. The LoRA adapter itself adds negligible memory overhead—typically under 100MB for rank-8 adapters.

How do I choose between LoRA rank values?

Rank 4–8 works for most style and format adaptation tasks. Rank 16–32 improves performance on knowledge-intensive tasks but increases trainable parameters linearly. The lora_alpha parameter in create_lora_modules should generally scale with rank (alpha = 2× rank is a stable default). Experiment with target_modules before increasing rank—all linear layers train more parameters than q/v projection only.

Can I resume training from a LoRA checkpoint?

Yes. The save_lora function in needle/model/finetune.py serializes adapter weights that can be reloaded with load_base_model followed by re-injecting saved adapters. Pass the checkpoint directory to --resume-from in the CLI, or load manually in Python and continue calling train_lora.

How do I deploy a LoRA-fine-tuned model for inference?

Two options exist in Cactus-Needle. Merged deployment: Use merge_lora from needle/model/export.py to bake adapters into base weights, producing a standard Hugging Face model with no inference overhead. Adapter-swapping: Keep base and adapter weights separate to serve multiple fine-tuned variants from one loaded base model, switching adapters between requests.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}