What is LoRA and How to Use It for Parameter-Efficient Fine-Tuning

LoRA (Low-Rank Adaptation) freezes original model weights and injects trainable low-rank matrices into selected layers, reducing trainable parameters by over 95% while maintaining task performance.

LoRA is a parameter-efficient fine-tuning technique that makes training large language models (LLMs) accessible on consumer hardware. Instead of updating billions of parameters, LoRA introduces small adapter matrices that capture task-specific knowledge while keeping the base model frozen. This article explains the mechanics of LoRA as implemented in the rohitg00/ai-engineering-from-scratch repository, covering the mathematical foundations and practical implementation details found in phases/11-llm-engineering/08-fine-tuning-lora/code/lora.py.

How LoRA Works: Low-Rank Adaptation Explained

Standard fine-tuning updates every weight in a model, but parameter-efficient fine-tuning with LoRA takes a different approach. It freezes the original weight matrix $W_0$ and approximates the weight update $\Delta W$ using a low-rank decomposition.

Low-Rank Matrix Decomposition

LoRA injects two small matrices, A and B, into selected linear layers. The forward pass becomes:

$$h = W_0x + \Delta Wx = W_0x + BAx$$

Where:

• Matrix A has dimensions $\mathbb{R}^{d \times r}$
• Matrix B has dimensions $\mathbb{R}^{r \times d}$
• Rank $r$ (typically 8 or 16) is far smaller than the full dimension $d$

This reduces the parameter count from $d^2$ to $2rd$, often representing less than 2% of the total parameters while capturing most of the task-specific signal.

The Scaling Factor (Alpha)

The LoRA contribution is multiplied by a scaling factor $\frac{\alpha}{r}$, where $\alpha$ is a hyperparameter typically set equal to $2r$. This allows you to control the magnitude of adaptation independently of the base learning rate. In phases/11-llm-engineering/08-fine-tuning-lora/code/lora.py, this scaling is applied during the forward pass to regulate how much the adapters influence the output.

Parameter Freezing and Adapter Injection

All original model parameters have requires_grad=False, guaranteeing that the base model stays unchanged. Only the low-rank matrices A and B receive gradients during backpropagation. This freezing mechanism ensures that the base LLM retains its general knowledge while the adapters learn task-specific transformations.

Implementing LoRA for Parameter-Efficient Fine-Tuning

The rohitg00/ai-engineering-from-scratch repository provides a complete implementation of LoRA injection, training, and deployment workflows.

Injecting LoRA Layers

To apply LoRA to a model, identify the target modules (typically linear layers) and inject the adapters using the provided utilities:

import torch.nn as nn
from phases.11_llm_engineering.08_fine_tuning_lora.code.lora import inject_lora

# 1️⃣ Build a demo model

def create_demo_model(d_model=256, hidden=512, n_classes=10):
    return nn.Sequential(
        nn.Linear(d_model, hidden), 
        nn.ReLU(),
        nn.Linear(hidden, hidden), 
        nn.ReLU(),
        nn.Linear(hidden, n_classes),
    )

model = create_demo_model()

# 2️⃣ Inject LoRA into selected linear layers (indices "0" and "2")

lora_layers = inject_lora(model, target_modules=["0", "2"], rank=8, alpha=16)
print("LoRA injected into:", list(lora_layers.keys()))

The inject_lora function replaces standard nn.Linear layers with LinearWithLoRA wrappers that contain the trainable A and B matrices.

Training Only the Adapters

During training, only the LoRA parameters require gradients. The repository provides a training utility that handles the forward pass and loss computation while keeping base weights frozen:

from phases.11_llm_engineering.08_fine_tuning_lora.code.lora import train_lora, create_demo_data

# Create dummy data: 500 samples, 256-dim input, 10-class targets

data = create_demo_data()

# Train for 10 epochs; only LoRA parameters are updated

losses = train_lora(model, data, epochs=10, lr=1e-3)
print(f"Loss ↓: {losses[0]:.4f} → {losses[-1]:.4f}")

Because you're updating only $2rd$ parameters instead of $d^2$, training requires significantly less GPU memory and completes faster than full fine-tuning.

Merging and Deploying Adapters

After training, you can merge the low-rank adapters back into the frozen weight matrix, yielding a single standard model without inference overhead. According to the implementation in code/lora.py#L66-L73, the merging operation computes $W_{new} = W_0 + \frac{\alpha}{r}BA$:

from phases.11_llm_engineering.08_fine_tuning_lora.code.lora import merge_lora_weights

# Merge LoRA weights back into the base model

merge_lora_weights(model)
print("After merge, model has no LoRA modules.")

This zero-cost deployment approach means the final model has the same architecture and latency as the original base model.

Saving and Loading Adapters

For multi-adapter serving scenarios, you can save just the LoRA weights (not the full model) and load them into any compatible base model:

from phases.11_llm_engineering.08_fine_tuning_lora.code.lora import (
    save_lora_adapter, load_lora_adapter,
)

# Save adapter weights

adapter_path = "my_lora_adapter.pt"
n_saved = save_lora_adapter(model, adapter_path)
print(f"Saved {n_saved} LoRA layers to {adapter_path}")

# Load into a fresh copy of the base model

base = create_demo_model()
load_lora_adapter(base, adapter_path)

This enables rapid experimentation with many task-specific versions of the same base model without storing multiple full-sized copies.

QLoRA: Quantized Parameter-Efficient Fine-Tuning

QLoRA extends the LoRA approach by quantizing the frozen base model to 4-bit (NF4 format) while keeping the LoRA adapters in fp16. This further shrinks memory requirements, allowing you to fine-tune models like Llama-7B on GPUs with as little as 6GB VRAM. The docs/en.md file in the repository explains that QLoRA uses double quantization and normalized float 4-bit format to preserve model quality while maximizing memory efficiency.

Why Use LoRA for Parameter-Efficient Fine-Tuning?

• Memory-friendly: Only the adapter matrices need GPU memory. A 7B model that would require >50GB VRAM can be fine-tuned on a 6GB GPU when LoRA is used.
• Fast training: Fewer trainable parameters mean fewer gradient updates and shorter training time compared to full fine-tuning.
• Zero-cost deployment: After merging, the model size is identical to the original, so serving incurs no extra latency.
• Multi-adapter support: Different LoRA adapters can be swapped in-place, enabling rapid experimentation with many task-specific versions of the same base model.

Summary

• LoRA enables parameter-efficient fine-tuning by freezing base weights and injecting trainable low-rank matrices A and B.
• The rank $r$ and scaling factor $\alpha$ control the trade-off between parameter count and adaptation capacity.
• Implementation in phases/11-llm-engineering/08-fine-tuning-lora/code/lora.py provides utilities for injection (inject_lora), training (train_lora), and merging (merge_lora_weights).
• QLoRA combines 4-bit quantization with LoRA to further reduce memory requirements for training large models.
• Adapters can be saved independently and loaded into base models, supporting efficient multi-task serving pipelines.

Frequently Asked Questions

What is the difference between LoRA and full fine-tuning?

Full fine-tuning updates every parameter in the model, requiring massive GPU memory and storage for each task. LoRA freezes the original weights and updates only small adapter matrices (typically <2% of parameters), making it feasible to fine-tune large language models on consumer hardware while achieving comparable task performance.

How do I choose the rank and alpha values for LoRA?

The rank ($r$) controls the expressiveness of the adaptation—common values are 8, 16, or 32, with higher ranks capturing more complex patterns but using more parameters. The alpha ($\alpha$) scaling factor is usually set to $2r$ to maintain consistent learning rates. According to the repository's docs/en.md, you should start with $r=8$ and $\alpha=16$ for most tasks, increasing rank only if the model underfits.

Can I use multiple LoRA adapters with the same base model?

Yes. Because LoRA adapters are small (often just megabytes), you can train dozens of task-specific adapters and swap them in-place at runtime. The load_lora_adapter function allows you to load different adapters into the same base model instance without reloading the full weights, enabling efficient multi-task serving architectures.

What is QLoRA and when should I use it?

QLoRA (Quantized Low-Rank Adaptation) quantizes the frozen base model to 4-bit precision while keeping LoRA adapters in fp16. You should use QLoRA when GPU memory is severely constrained—for example, when fine-tuning 7B or 13B models on single consumer GPUs. The 4-bit quantization reduces memory footprint by approximately 75% compared to standard LoRA, though it requires compatible quantization libraries.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}