# How to Perform LoRA Fine-Tuning with Cactus-Needle: A Complete Guide

> Learn LoRA fine-tuning with Cactus-Needle. Quickly train LLMs using the `needle finetune` CLI or Python API for efficient adapter injection and frozen base weights. Get started in under 50 lines of code.

- Repository: [Cactus Compute, Inc./needle](https://github.com/cactus-compute/needle)
- Tags: how-to-guide
- Published: 2026-09-05

---

**The fastest way to fine-tune large language models with LoRA in Cactus-Needle is through the `needle finetune` CLI command or the Python API in [`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py), which handles adapter injection, frozen base weights, and efficient training in under 50 lines of code.**

Cactus-Needle provides a modular, production-ready pipeline for **LoRA fine-tuning** that reduces GPU memory requirements by freezing original model parameters and training only low-rank adapter matrices. This guide walks through the complete workflow using the actual source implementation.

## LoRA Fine-Tuning Architecture in Needle

The Cactus-Needle codebase separates concerns across specialized modules. Understanding this structure helps you customize training for your use case.

### Core Pipeline Components

- **[`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py)** — Orchestrates the entire LoRA workflow: adapter creation, training loop, checkpointing, and weight serialization
- **[`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py)** — Loads base models via Hugging Face's `AutoModelForCausalLM` with optional quantization
- **[`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py)** — Enables 8-bit and 4-bit quantization for memory-constrained environments
- **[`needle/model/tokenizer.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/tokenizer.py)** — Prepares tokenized datasets for the training loop
- **[`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py)** — Merges trained LoRA weights back into the base model for deployment
- **[`needle/cli.py`](https://github.com/cactus-compute/needle/blob/main/needle/cli.py)** — Exposes the `needle finetune` command-line interface

## Step 1: Load and Prepare the Base Model

The `load_base_model` function in [`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py) wraps the generic loader from [`needle/model/run.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/run.py). It supports quantization through [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py) to fit large models on consumer GPUs.

```python
from needle.model.finetune import load_base_model

model = load_base_model(
    repo_id="EleutherAI/gpt-neo-125M",
    quantize=False,  # Set True for 8-bit via quantize.py

)

```

When `quantize=True`, the loader applies compression algorithms from [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py) before LoRA adapter injection.

## Step 2: Inject LoRA Adapters

The `create_lora_modules` function builds **low-rank linear layers** (`LoRALinear`) that wrap target weight matrices. By default, this targets the query and value projections where adaptation is most effective.

```python
from needle.model.finetune import create_lora_modules

model = create_lora_modules(
    model,
    lora_rank=8,          # Lower rank = fewer parameters, faster training

    lora_alpha=16,        # Scaling factor for adapter outputs

    target_modules=["q_proj", "v_proj"],  # Attention projections to adapt

)

```

This step automatically freezes all original model parameters (`requires_grad=False`), leaving only the LoRA weights trainable.

## Step 3: Prepare Your Dataset

The `prepare_dataset` utility handles tokenization through [`needle/model/tokenizer.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/tokenizer.py) and returns a PyTorch `DataLoader`.

```python
from needle.model.finetune import prepare_dataset

train_loader = prepare_dataset(
    tokenized_path="data/train_tokenized.jsonl",
    batch_size=4,
    max_length=512,
)

```

Input files should contain JSON lines with `input_ids` and `attention_mask` fields, or you can pass a raw Hugging Face `datasets.Dataset` object.

## Step 4: Execute the Training Loop

The `train_lora` function implements gradient accumulation, learning rate scheduling, and telemetry logging via [`needle/_telemetry.py`](https://github.com/cactus-compute/needle/blob/main/needle/_telemetry.py).

```python
from needle.model.finetune import train_lora

train_lora(
    model=model,
    dataloader=train_loader,
    epochs=3,
    learning_rate=5e-5,
    gradient_accumulation_steps=4,  # Effective batch size = 4 × 4 = 16

)

```

Training metrics are emitted through [`needle/_telemetry.py`](https://github.com/cactus-compute/needle/blob/main/needle/_telemetry.py) for monitoring in Weights & Biases or TensorBoard.

## Step 5: Save and Export LoRA Weights

After fine-tuning, persist adapter weights separately or merge them for inference-only deployment.

```python
from needle.model.finetune import save_lora
from needle.model.export import merge_lora

# Save adapter weights only (small checkpoint)

save_lora(model, output_dir="lora_weights/")

# Or merge into base model for standalone inference

merged_model = merge_lora(model)

```

The `merge_lora` function in [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) computes `W_merged = W_base + (alpha/rank) * B * A` for each adapted layer, eliminating inference overhead.

## Complete Python API Example

```python
from needle.model.finetune import (
    load_base_model,
    create_lora_modules,
    prepare_dataset,
    train_lora,
    save_lora,
)

# Load base model

model = load_base_model("EleutherAI/gpt-neo-125M", quantize=False)

# Attach LoRA adapters

model = create_lora_modules(
    model,
    lora_rank=8,
    lora_alpha=16,
    target_modules=["q_proj", "v_proj"],
)

# Prepare data

loader = prepare_dataset("data/train.jsonl", batch_size=4, max_length=512)

# Train

train_lora(model, loader, epochs=3, learning_rate=5e-5)

# Save

save_lora(model, "lora_weights/")

```

## CLI Quick Start

The `needle finetune` command in [`needle/cli.py`](https://github.com/cactus-compute/needle/blob/main/needle/cli.py) exposes all parameters through the `FinetuneConfig` dataclass:

```bash
needle finetune \
  --model EleutherAI/gpt-neo-125M \
  --dataset data/train_tokenized.jsonl \
  --lora-rank 8 \
  --lora-alpha 16 \
  --target-modules q_proj,v_proj \
  --epochs 3 \
  --lr 5e-5 \
  --batch-size 4 \
  --gradient-accumulation-steps 4 \
  --output-dir lora_weights/

```

## Hyperparameter Guidelines for LoRA Fine-Tuning

| Parameter | Typical Range | Effect |
|-----------|-------------|--------|
| `lora_rank` | 4–64 | Higher rank = more capacity, more parameters |
| `lora_alpha` | 2× rank | Scaling factor; higher values amplify adapter influence |
| `target_modules` | `["q_proj", "v_proj"]` or all linear layers | More targets = more trainable parameters |
| `learning_rate` | 1e-4 to 1e-5 | LoRA typically uses 2–10× higher LR than full fine-tuning |
| `batch_size` × `gradient_accumulation_steps` | 16–128 | Larger effective batches stabilize training |

## Summary

- **[`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py)** provides the complete LoRA implementation: `load_base_model`, `create_lora_modules`, `train_lora`, and `save_lora`
- Base model weights remain frozen; only low-rank adapter matrices train, reducing memory by 10–100×
- Quantization via [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py) enables training on single consumer GPUs
- The `needle finetune` CLI maps directly to Python API parameters for reproducible workflows
- Use [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) to merge adapters for inference or keep separate for multi-task serving

## Frequently Asked Questions

### What hardware is required for LoRA fine-tuning with Cactus-Needle?

With 8-bit quantization enabled in [`needle/model/quantize.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/quantize.py), you can fine-tune 7B parameter models on a single 24GB GPU. Without quantization, plan for approximately 2× the model size in VRAM for fp16 training. The LoRA adapter itself adds negligible memory overhead—typically under 100MB for rank-8 adapters.

### How do I choose between LoRA rank values?

Rank 4–8 works for most style and format adaptation tasks. Rank 16–32 improves performance on knowledge-intensive tasks but increases trainable parameters linearly. The `lora_alpha` parameter in `create_lora_modules` should generally scale with rank (alpha = 2× rank is a stable default). Experiment with `target_modules` before increasing rank—all linear layers train more parameters than q/v projection only.

### Can I resume training from a LoRA checkpoint?

Yes. The `save_lora` function in [`needle/model/finetune.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/finetune.py) serializes adapter weights that can be reloaded with `load_base_model` followed by re-injecting saved adapters. Pass the checkpoint directory to `--resume-from` in the CLI, or load manually in Python and continue calling `train_lora`.

### How do I deploy a LoRA-fine-tuned model for inference?

Two options exist in Cactus-Needle. **Merged deployment:** Use `merge_lora` from [`needle/model/export.py`](https://github.com/cactus-compute/needle/blob/main/needle/model/export.py) to bake adapters into base weights, producing a standard Hugging Face model with no inference overhead. **Adapter-swapping:** Keep base and adapter weights separate to serve multiple fine-tuned variants from one loaded base model, switching adapters between requests.