How Fine-Tuning LLMs with LoRA and QLoRA Works: A Complete Implementation Guide

Fine-tuning LLMs with LoRA and QLoRA freezes the entire pretrained model and trains only small low-rank adapter matrices, with QLoRA adding 4-bit quantization to compress the frozen base weights and reduce memory usage by up to 4×.

Fine-tuning massive language models traditionally requires updating billions of parameters, making it computationally prohibitive for most practitioners. The ai-engineering-from-scratch repository by rohitg00 demonstrates a memory-efficient solution through Low-Rank Adaptation (LoRA) and its quantized variant (QLoRA), allowing you to adapt large models by training less than 1% of the total parameters while preserving full model quality.

The LoRA Method: Low-Rank Adaptation

LoRA solves the parameter efficiency problem by injecting trainable rank factorization matrices into specific layers while keeping the original pretrained weights frozen.

Mathematical Foundation

The core innovation replaces full weight updates with a low-rank decomposition. For a pretrained weight matrix $W$, LoRA adds the trainable decomposition $BA$ where $B$ and $A$ are low-rank matrices with rank $r$:

$$\text{output} = W x + \frac{\alpha}{r} B A x$$

Here, $r$ is the rank (typically 4-64), and $\alpha$ is a scaling factor that controls adaptation strength. The original matrix $W$ remains frozen with requires_grad=False, while only $A$ and $B$ receive gradient updates. This reduces trainable parameters from billions to millions.

Core Implementation Classes

In phases/11-llm-engineering/08-fine-tuning-lora/code/lora.py, the implementation centers on three key components:

• LoRALayer (lines 16-18): Defines the trainable matrices $A$ and $B$ and performs the scaled multiplication $\frac{\alpha}{r} B A x$ during the forward pass.
• LinearWithLoRA (lines 20-33): Wraps any standard nn.Linear layer, modifying the forward pass to compute linear(x) + lora(x), combining the frozen base output with the adapter correction.
• inject_lora (lines 35-52): Recursively walks the model architecture, identifies target nn.Linear modules by name, and replaces them with LinearWithLoRA instances on specified layers.

QLoRA: Quantized Low-Rank Adaptation

QLoRA extends LoRA by compressing the frozen base model to 4-bit precision using the NF4 (Normal Float 4) quantization scheme, further reducing memory footprint while maintaining training stability.

4-Bit NF4 Quantization Strategy

The quantization utilities in lora.py implement block-wise compression:

• quantize_to_nf4 (lines 83-104): Splits tensors into 64-element blocks, computes per-block scaling factors, and stores values as signed 4-bit integers.
• dequantize_from_nf4: Reconstructs full-precision values during the forward pass using the stored block scales.
• quantize_model (lines 107-119): Applies NF4 quantization to all frozen base weights while preserving LoRA adapters in full precision (FP16/FP32).

During training, only the adapter parameters receive gradients, while the quantized base weights are dequantized on-the-fly for forward passes. This achieves approximately 4× memory reduction compared to standard LoRA without requiring gradient checkpointing.

Implementation Workflow

According to the source code in rohitg00/ai-engineering-from-scratch, the complete fine-tuning pipeline follows five distinct stages:

1. Freeze the pretrained LLM and set all base parameters to requires_grad=False.
2. Inject LoRA adapters into target linear layers (typically attention and MLP projections) using inject_lora.
3. Quantize (optional for QLoRA) the frozen weights to 4-bit NF4 using quantize_model.
4. Train only the adapter parameters using train_lora (lines 122-152), which handles gradient accumulation and optimizer state management.
5. Merge adapters back into base weights for inference-only deployment using merge_lora_weights (lines 67-81), eliminating runtime overhead.

Practical Code Implementation

Below is a complete workflow demonstrating how to fine-tune LLMs with LoRA and QLoRA using the repository's utilities:

from phases.11_llm_engineering.08_fine_tuning_lora.code.lora import (
    create_demo_model, create_demo_data,
    inject_lora, train_lora, quantize_model,
    merge_lora_weights, save_lora_adapter, load_lora_adapter,
)

# Initialize model and synthetic training data

model = create_demo_model()  # Replace with actual LLM architecture

data = create_demo_data()

# Step 1: Inject LoRA adapters into specific layers (rank=8, alpha=16)

lora_layers = inject_lora(
    model, 
    target_modules=["0", "2"],  # Layer names to adapt

    rank=8, 
    alpha=16
)

# Step 2: (QLoRA) Quantize frozen base to 4-bit NF4

q_state = quantize_model(model)  # Returns dict of quantized tensors

# Step 3: Train only adapter parameters

train_losses = train_lora(model, data, epochs=10, lr=1e-3)

# Step 4: Merge adapters back for inference-only deployment

merge_lora_weights(model)

# Step 5: Save adapters for portability

import tempfile, os
tmp = tempfile.NamedTemporaryFile(suffix=".pt", delete=False)
save_lora_adapter(model, tmp.name)

# Load adapters into fresh model instance

new_model = create_demo_model()
load_lora_adapter(new_model, tmp.name)
os.unlink(tmp.name)

Merging and Deployment Strategies

After fine-tuning completes, you have two deployment options. Merging adds the low-rank matrices back into the base weights ($W_{\text{new}} = W + \frac{\alpha}{r}BA$), producing a standard model with no inference overhead. Alternatively, adapter checkpointing preserves the base model and loads only the small adapter files (typically 10-100MB), enabling efficient task switching by swapping adapters while keeping one quantized base model in memory.

The save_lora_adapter and load_lora_adapter functions serialize only the trainable matrices $A$ and $B$, making checkpoints highly portable across different hardware configurations.

Summary

• LoRA freezes the full pretrained model and trains only small low-rank matrices $A$ and $B$ for each target layer, reducing trainable parameters by 99%.
• QLoRA adds 4-bit NF4 quantization to the frozen base weights, achieving roughly 4× additional memory savings while maintaining adapter precision.
• The inject_lora function in lora.py (lines 35-52) recursively replaces target nn.Linear layers with LinearWithLoRA wrappers.
• Only adapter parameters receive gradients during train_lora (lines 122-152), while quantized weights are dequantized on-the-fly during forward passes.
• Use merge_lora_weights (lines 67-81) to combine adapters with base weights for production deployment, or preserve them separately for multi-task flexibility.

Frequently Asked Questions

What is the difference between LoRA and QLoRA?

LoRA freezes the base model and trains low-rank adapter matrices, reducing trainable parameters by roughly 99%. QLoRA applies the same adapter training but additionally compresses the frozen base weights to 4-bit precision using NF4 quantization, reducing memory usage by approximately 4× compared to standard LoRA while maintaining identical training dynamics and final model quality.

How do I choose the rank and alpha parameters for LoRA fine-tuning?

The rank ($r$) typically ranges from 4 to 64, with lower values (4-8) sufficient for simple tasks and higher values (32-64) for complex domain adaptation. The alpha ($\alpha$) scaling factor usually equals $2r$ (e.g., $\alpha=16$ when $r=8$) to maintain consistent learning rates across different ranks. The inject_lora function accepts these as parameters: rank=8, alpha=16.

Can I merge LoRA adapters back into the original model weights?

Yes, adapters can be permanently merged into the base weights using merge_lora_weights (lines 67-81 in lora.py), which computes $W_{\text{new}} = W + \frac{\alpha}{r}BA$ for each adapted layer. This produces a standard model with no runtime overhead, ideal for inference-only deployment where you want to eliminate adapter computation costs.

Why are only the LoRA matrices trained during fine-tuning?

The original base model weights are intentionally frozen (set to requires_grad=False in inject_lora, lines 35-52) because updating billions of parameters requires massive GPU memory and risks catastrophic forgetting. By training only the low-rank decomposition matrices $A$ and $B$ (typically 0.5-2% of model size), LoRA achieves parameter-efficient transfer learning while preserving the pretrained knowledge encoded in the frozen weights.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}