# How Fine-Tuning LLMs with LoRA and QLoRA Works: A Complete Implementation Guide

> Learn how fine-tuning LLMs with LoRA and QLoRA works. This guide explains training small adapter matrices and using QLoRA for efficient 4-bit quantization, saving memory.

- Repository: [Rohit Ghumare/ai-engineering-from-scratch](https://github.com/rohitg00/ai-engineering-from-scratch)
- Tags: how-to-guide
- Published: 2026-06-16

---

**Fine-tuning LLMs with LoRA and QLoRA freezes the entire pretrained model and trains only small low-rank adapter matrices, with QLoRA adding 4-bit quantization to compress the frozen base weights and reduce memory usage by up to 4×.**

Fine-tuning massive language models traditionally requires updating billions of parameters, making it computationally prohibitive for most practitioners. The `ai-engineering-from-scratch` repository by rohitg00 demonstrates a memory-efficient solution through Low-Rank Adaptation (LoRA) and its quantized variant (QLoRA), allowing you to adapt large models by training less than 1% of the total parameters while preserving full model quality.

## The LoRA Method: Low-Rank Adaptation

LoRA solves the parameter efficiency problem by injecting trainable rank factorization matrices into specific layers while keeping the original pretrained weights frozen.

### Mathematical Foundation

The core innovation replaces full weight updates with a low-rank decomposition. For a pretrained weight matrix $W$, LoRA adds the trainable decomposition $BA$ where $B$ and $A$ are low-rank matrices with rank $r$:

$$\text{output} = W x + \frac{\alpha}{r} B A x$$

Here, **$r$** is the rank (typically 4-64), and **$\alpha$** is a scaling factor that controls adaptation strength. The original matrix $W$ remains frozen with `requires_grad=False`, while only $A$ and $B$ receive gradient updates. This reduces trainable parameters from billions to millions.

### Core Implementation Classes

In [`phases/11-llm-engineering/08-fine-tuning-lora/code/lora.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/phases/11-llm-engineering/08-fine-tuning-lora/code/lora.py), the implementation centers on three key components:

- **`LoRALayer`** (lines 16-18): Defines the trainable matrices $A$ and $B$ and performs the scaled multiplication $\frac{\alpha}{r} B A x$ during the forward pass.
- **`LinearWithLoRA`** (lines 20-33): Wraps any standard `nn.Linear` layer, modifying the forward pass to compute `linear(x) + lora(x)`, combining the frozen base output with the adapter correction.
- **`inject_lora`** (lines 35-52): Recursively walks the model architecture, identifies target `nn.Linear` modules by name, and replaces them with `LinearWithLoRA` instances on specified layers.

## QLoRA: Quantized Low-Rank Adaptation

**QLoRA** extends LoRA by compressing the frozen base model to 4-bit precision using the **NF4 (Normal Float 4)** quantization scheme, further reducing memory footprint while maintaining training stability.

### 4-Bit NF4 Quantization Strategy

The quantization utilities in [`lora.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/lora.py) implement block-wise compression:

- **`quantize_to_nf4`** (lines 83-104): Splits tensors into 64-element blocks, computes per-block scaling factors, and stores values as signed 4-bit integers.
- **`dequantize_from_nf4`**: Reconstructs full-precision values during the forward pass using the stored block scales.
- **`quantize_model`** (lines 107-119): Applies NF4 quantization to all frozen base weights while preserving LoRA adapters in full precision (FP16/FP32).

During training, only the adapter parameters receive gradients, while the quantized base weights are dequantized on-the-fly for forward passes. This achieves approximately **4× memory reduction** compared to standard LoRA without requiring gradient checkpointing.

## Implementation Workflow

According to the source code in `rohitg00/ai-engineering-from-scratch`, the complete fine-tuning pipeline follows five distinct stages:

1. **Freeze** the pretrained LLM and set all base parameters to `requires_grad=False`.
2. **Inject** LoRA adapters into target linear layers (typically attention and MLP projections) using `inject_lora`.
3. **Quantize** (optional for QLoRA) the frozen weights to 4-bit NF4 using `quantize_model`.
4. **Train** only the adapter parameters using `train_lora` (lines 122-152), which handles gradient accumulation and optimizer state management.
5. **Merge** adapters back into base weights for inference-only deployment using `merge_lora_weights` (lines 67-81), eliminating runtime overhead.

## Practical Code Implementation

Below is a complete workflow demonstrating how to fine-tune LLMs with LoRA and QLoRA using the repository's utilities:

```python
from phases.11_llm_engineering.08_fine_tuning_lora.code.lora import (
    create_demo_model, create_demo_data,
    inject_lora, train_lora, quantize_model,
    merge_lora_weights, save_lora_adapter, load_lora_adapter,
)

# Initialize model and synthetic training data

model = create_demo_model()  # Replace with actual LLM architecture

data = create_demo_data()

# Step 1: Inject LoRA adapters into specific layers (rank=8, alpha=16)

lora_layers = inject_lora(
    model, 
    target_modules=["0", "2"],  # Layer names to adapt

    rank=8, 
    alpha=16
)

# Step 2: (QLoRA) Quantize frozen base to 4-bit NF4

q_state = quantize_model(model)  # Returns dict of quantized tensors

# Step 3: Train only adapter parameters

train_losses = train_lora(model, data, epochs=10, lr=1e-3)

# Step 4: Merge adapters back for inference-only deployment

merge_lora_weights(model)

# Step 5: Save adapters for portability

import tempfile, os
tmp = tempfile.NamedTemporaryFile(suffix=".pt", delete=False)
save_lora_adapter(model, tmp.name)

# Load adapters into fresh model instance

new_model = create_demo_model()
load_lora_adapter(new_model, tmp.name)
os.unlink(tmp.name)

```

## Merging and Deployment Strategies

After fine-tuning completes, you have two deployment options. **Merging** adds the low-rank matrices back into the base weights ($W_{\text{new}} = W + \frac{\alpha}{r}BA$), producing a standard model with no inference overhead. Alternatively, **adapter checkpointing** preserves the base model and loads only the small adapter files (typically 10-100MB), enabling efficient task switching by swapping adapters while keeping one quantized base model in memory.

The `save_lora_adapter` and `load_lora_adapter` functions serialize only the trainable matrices $A$ and $B$, making checkpoints highly portable across different hardware configurations.

## Summary

- **LoRA** freezes the full pretrained model and trains only small low-rank matrices $A$ and $B$ for each target layer, reducing trainable parameters by 99%.
- **QLoRA** adds 4-bit NF4 quantization to the frozen base weights, achieving roughly 4× additional memory savings while maintaining adapter precision.
- The `inject_lora` function in [`lora.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/lora.py) (lines 35-52) recursively replaces target `nn.Linear` layers with `LinearWithLoRA` wrappers.
- Only adapter parameters receive gradients during `train_lora` (lines 122-152), while quantized weights are dequantized on-the-fly during forward passes.
- Use `merge_lora_weights` (lines 67-81) to combine adapters with base weights for production deployment, or preserve them separately for multi-task flexibility.

## Frequently Asked Questions

### What is the difference between LoRA and QLoRA?

LoRA freezes the base model and trains low-rank adapter matrices, reducing trainable parameters by roughly 99%. QLoRA applies the same adapter training but additionally compresses the frozen base weights to 4-bit precision using NF4 quantization, reducing memory usage by approximately 4× compared to standard LoRA while maintaining identical training dynamics and final model quality.

### How do I choose the rank and alpha parameters for LoRA fine-tuning?

The **rank** ($r$) typically ranges from 4 to 64, with lower values (4-8) sufficient for simple tasks and higher values (32-64) for complex domain adaptation. The **alpha** ($\alpha$) scaling factor usually equals $2r$ (e.g., $\alpha=16$ when $r=8$) to maintain consistent learning rates across different ranks. The `inject_lora` function accepts these as parameters: `rank=8, alpha=16`.

### Can I merge LoRA adapters back into the original model weights?

Yes, adapters can be permanently merged into the base weights using `merge_lora_weights` (lines 67-81 in [`lora.py`](https://github.com/rohitg00/ai-engineering-from-scratch/blob/main/lora.py)), which computes $W_{\text{new}} = W + \frac{\alpha}{r}BA$ for each adapted layer. This produces a standard model with no runtime overhead, ideal for inference-only deployment where you want to eliminate adapter computation costs.

### Why are only the LoRA matrices trained during fine-tuning?

The original base model weights are intentionally frozen (set to `requires_grad=False` in `inject_lora`, lines 35-52) because updating billions of parameters requires massive GPU memory and risks catastrophic forgetting. By training only the low-rank decomposition matrices $A$ and $B$ (typically 0.5-2% of model size), LoRA achieves parameter-efficient transfer learning while preserving the pretrained knowledge encoded in the frozen weights.