# What is LoRA and How to Use It for Parameter-Efficient Fine-Tuning

> Discover LoRA for parameter-efficient fine-tuning. Learn how LoRA freezes model weights and injects low-rank matrices, cutting trainable parameters by over 95% with minimal performance loss.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: tutorial
- Published: 2026-07-31

---

**LoRA (Low-Rank Adaptation) freezes original model weights and injects trainable low-rank matrices into selected layers, reducing trainable parameters by over 95% while maintaining task performance.**

**LoRA** is a parameter-efficient fine-tuning technique that makes training large language models (LLMs) accessible on consumer hardware. Instead of updating billions of parameters, LoRA introduces small adapter matrices that capture task-specific knowledge while keeping the base model frozen. This article explains the mechanics of LoRA as implemented in the `rohitg00/ai-engineering-from-scratch` repository, covering the mathematical foundations and practical implementation details found in [`phases/11-llm-engineering/08-fine-tuning-lora/code/lora.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/11-llm-engineering/08-fine-tuning-lora/code/lora.py).

## How LoRA Works: Low-Rank Adaptation Explained

Standard fine-tuning updates every weight in a model, but **parameter-efficient fine-tuning** with LoRA takes a different approach. It freezes the original weight matrix $W_0$ and approximates the weight update $\Delta W$ using a low-rank decomposition.

### Low-Rank Matrix Decomposition

LoRA injects two small matrices, **A** and **B**, into selected linear layers. The forward pass becomes:

$$h = W_0x + \Delta Wx = W_0x + BAx$$

Where:
- Matrix **A** has dimensions $\mathbb{R}^{d \times r}$
- Matrix **B** has dimensions $\mathbb{R}^{r \times d}$
- **Rank** $r$ (typically 8 or 16) is far smaller than the full dimension $d$

This reduces the parameter count from $d^2$ to $2rd$, often representing less than 2% of the total parameters while capturing most of the task-specific signal.

### The Scaling Factor (Alpha)

The LoRA contribution is multiplied by a **scaling factor** $\frac{\alpha}{r}$, where $\alpha$ is a hyperparameter typically set equal to $2r$. This allows you to control the magnitude of adaptation independently of the base learning rate. In [`phases/11-llm-engineering/08-fine-tuning-lora/code/lora.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/11-llm-engineering/08-fine-tuning-lora/code/lora.py), this scaling is applied during the forward pass to regulate how much the adapters influence the output.

### Parameter Freezing and Adapter Injection

All original model parameters have `requires_grad=False`, guaranteeing that the base model stays unchanged. Only the low-rank matrices A and B receive gradients during backpropagation. This freezing mechanism ensures that the base LLM retains its general knowledge while the adapters learn task-specific transformations.

## Implementing LoRA for Parameter-Efficient Fine-Tuning

The `rohitg00/ai-engineering-from-scratch` repository provides a complete implementation of LoRA injection, training, and deployment workflows.

### Injecting LoRA Layers

To apply LoRA to a model, identify the target modules (typically linear layers) and inject the adapters using the provided utilities:

```python
import torch.nn as nn
from phases.11_llm_engineering.08_fine_tuning_lora.code.lora import inject_lora

# 1️⃣ Build a demo model

def create_demo_model(d_model=256, hidden=512, n_classes=10):
    return nn.Sequential(
        nn.Linear(d_model, hidden), 
        nn.ReLU(),
        nn.Linear(hidden, hidden), 
        nn.ReLU(),
        nn.Linear(hidden, n_classes),
    )

model = create_demo_model()

# 2️⃣ Inject LoRA into selected linear layers (indices "0" and "2")

lora_layers = inject_lora(model, target_modules=["0", "2"], rank=8, alpha=16)
print("LoRA injected into:", list(lora_layers.keys()))

```

The `inject_lora` function replaces standard `nn.Linear` layers with `LinearWithLoRA` wrappers that contain the trainable A and B matrices.

### Training Only the Adapters

During training, only the LoRA parameters require gradients. The repository provides a training utility that handles the forward pass and loss computation while keeping base weights frozen:

```python
from phases.11_llm_engineering.08_fine_tuning_lora.code.lora import train_lora, create_demo_data

# Create dummy data: 500 samples, 256-dim input, 10-class targets

data = create_demo_data()

# Train for 10 epochs; only LoRA parameters are updated

losses = train_lora(model, data, epochs=10, lr=1e-3)
print(f"Loss ↓: {losses[0]:.4f} → {losses[-1]:.4f}")

```

Because you're updating only $2rd$ parameters instead of $d^2$, training requires significantly less GPU memory and completes faster than full fine-tuning.

### Merging and Deploying Adapters

After training, you can merge the low-rank adapters back into the frozen weight matrix, yielding a single standard model without inference overhead. According to the implementation in `code/lora.py#L66-L73`, the merging operation computes $W_{new} = W_0 + \frac{\alpha}{r}BA$:

```python
from phases.11_llm_engineering.08_fine_tuning_lora.code.lora import merge_lora_weights

# Merge LoRA weights back into the base model

merge_lora_weights(model)
print("After merge, model has no LoRA modules.")

```

This **zero-cost deployment** approach means the final model has the same architecture and latency as the original base model.

### Saving and Loading Adapters

For multi-adapter serving scenarios, you can save just the LoRA weights (not the full model) and load them into any compatible base model:

```python
from phases.11_llm_engineering.08_fine_tuning_lora.code.lora import (
    save_lora_adapter, load_lora_adapter,
)

# Save adapter weights

adapter_path = "my_lora_adapter.pt"
n_saved = save_lora_adapter(model, adapter_path)
print(f"Saved {n_saved} LoRA layers to {adapter_path}")

# Load into a fresh copy of the base model

base = create_demo_model()
load_lora_adapter(base, adapter_path)

```

This enables rapid experimentation with many task-specific versions of the same base model without storing multiple full-sized copies.

## QLoRA: Quantized Parameter-Efficient Fine-Tuning

**QLoRA** extends the LoRA approach by quantizing the frozen base model to 4-bit (NF4 format) while keeping the LoRA adapters in fp16. This further shrinks memory requirements, allowing you to fine-tune models like Llama-7B on GPUs with as little as 6GB VRAM. The [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md) file in the repository explains that QLoRA uses double quantization and normalized float 4-bit format to preserve model quality while maximizing memory efficiency.

## Why Use LoRA for Parameter-Efficient Fine-Tuning?

- **Memory-friendly:** Only the adapter matrices need GPU memory. A 7B model that would require >50GB VRAM can be fine-tuned on a 6GB GPU when LoRA is used.
- **Fast training:** Fewer trainable parameters mean fewer gradient updates and shorter training time compared to full fine-tuning.
- **Zero-cost deployment:** After merging, the model size is identical to the original, so serving incurs no extra latency.
- **Multi-adapter support:** Different LoRA adapters can be swapped in-place, enabling rapid experimentation with many task-specific versions of the same base model.

## Summary

- **LoRA** enables parameter-efficient fine-tuning by freezing base weights and injecting trainable low-rank matrices **A** and **B**.
- The rank $r$ and scaling factor $\alpha$ control the trade-off between parameter count and adaptation capacity.
- Implementation in [`phases/11-llm-engineering/08-fine-tuning-lora/code/lora.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/11-llm-engineering/08-fine-tuning-lora/code/lora.py) provides utilities for injection (`inject_lora`), training (`train_lora`), and merging (`merge_lora_weights`).
- **QLoRA** combines 4-bit quantization with LoRA to further reduce memory requirements for training large models.
- Adapters can be saved independently and loaded into base models, supporting efficient multi-task serving pipelines.

## Frequently Asked Questions

### What is the difference between LoRA and full fine-tuning?

Full fine-tuning updates every parameter in the model, requiring massive GPU memory and storage for each task. LoRA freezes the original weights and updates only small adapter matrices (typically <2% of parameters), making it feasible to fine-tune large language models on consumer hardware while achieving comparable task performance.

### How do I choose the rank and alpha values for LoRA?

The **rank** ($r$) controls the expressiveness of the adaptation—common values are 8, 16, or 32, with higher ranks capturing more complex patterns but using more parameters. The **alpha** ($\alpha$) scaling factor is usually set to $2r$ to maintain consistent learning rates. According to the repository's [`docs/en.md`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/docs/en.md), you should start with $r=8$ and $\alpha=16$ for most tasks, increasing rank only if the model underfits.

### Can I use multiple LoRA adapters with the same base model?

Yes. Because LoRA adapters are small (often just megabytes), you can train dozens of task-specific adapters and swap them in-place at runtime. The `load_lora_adapter` function allows you to load different adapters into the same base model instance without reloading the full weights, enabling efficient multi-task serving architectures.

### What is QLoRA and when should I use it?

**QLoRA** (Quantized Low-Rank Adaptation) quantizes the frozen base model to 4-bit precision while keeping LoRA adapters in fp16. You should use QLoRA when GPU memory is severely constrained—for example, when fine-tuning 7B or 13B models on single consumer GPUs. The 4-bit quantization reduces memory footprint by approximately 75% compared to standard LoRA, though it requires compatible quantization libraries.