# What Are Parameter-Efficient Fine-Tuning Techniques? A Complete Guide

> Discover parameter-efficient fine-tuning PEFT techniques to adapt large models by updating just a small fraction of parameters. Fine-tune on a single GPU, saving resources.

- Repository: [Chip Huyen/aie-book](https://github.com/chiphuyen/aie-book)
- Tags: deep-dive
- Published: 2026-04-24

---

**Parameter-efficient fine-tuning (PEFT) techniques are methods that adapt large foundation models by updating only a tiny fraction of parameters—typically less than 1%—enabling fine-tuning on a single GPU while preserving pretrained knowledge.**

In the `chiphuyen/aie-book` repository, these techniques are highlighted as essential tools for AI engineers working with large language models (LLMs). Instead of modifying billions of frozen base weights, PEFT introduces lightweight trainable components that steer model behavior with minimal computational overhead.

## Why Parameter-Efficient Fine-Tuning Matters

Full-model fine-tuning scales linearly with parameter count, quickly exceeding the memory capacity of commodity hardware. When you update every weight in a multi-billion parameter model, optimizer states alone can consume hundreds of gigabytes of GPU memory.

PEFT solves this by freezing the massive pretrained backbone and introducing a small "side-network" of trainable parameters. This approach reduces training time, lowers data requirements, and produces modular adapters that can be swapped at inference time. According to the repository's curated resources, this pattern makes fine-tuning feasible on single-GPU setups and even edge devices.

## Core Parameter-Efficient Fine-Tuning Techniques

### Adapter Layers

**Adapter layers** insert tiny bottleneck MLPs (down-project → up-project) after each transformer block. The original weights remain frozen, while only the adapter parameters are updated.

- **Parameter overhead**: < 1% of total parameters
- **Architecture**: Bottleneck design with reduction factors (commonly 16:1) compresses and expands activations
- **Benefit**: Task-specific adapters can be hot-swapped without reloading the base model

### LoRA (Low-Rank Adaptation)

**LoRA** adds rank-*r* matrices to selected weight matrices following the formula ΔW = A·B. The base model stays frozen; only the low-rank matrices *A* and *B* are trained.

- **Parameter overhead**: 0.1%–0.5% (varies by rank *r*)
- **Target modules**: Typically applied to query (`q_proj`) and value (`v_proj`) projection matrices
- **Rank**: Common values are 8 or 16, with alpha scaling factors

The repository references a concise overview of LoRA benefits in [`chapter-summaries.md`](https://github.com/chiphuyen/aie-book/blob/main/chapter-summaries.md) (lines 60–62).

### QLoRA

**QLoRA** combines LoRA with 4-bit quantization of the base model. This dramatically cuts GPU memory usage while maintaining the training dynamics of standard LoRA.

- **Parameter overhead**: Same as LoRA (0.1%–0.5%) plus quantization gains
- **Memory savings**: Base model stored in 4-bit precision; only LoRA weights in full precision
- **Use case**: Enables fine-tuning 70B+ parameter models on consumer GPUs

### Prefix-Tuning

**Prefix-tuning** prepends learnable "virtual tokens" to the input sequence. The model's attention mechanism attends to these tokens, which steer output generation without modifying the underlying weights.

- **Parameter overhead**: ~0.1% (just the prefix embeddings)
- **Configuration**: Typically 20–100 virtual tokens
- **Task type**: Effective for text generation tasks (causal language modeling)

### Prompt-Tuning

**Prompt-tuning** is similar to prefix-tuning but learns a short prompt embedding concatenated once per task. It represents the most parameter-efficient approach.

- **Parameter overhead**: < 0.01%
- **Architecture**: Soft prompts (embedding vectors) prepended to input tokens
- **Trade-off**: Requires more tuning steps than prefix-tuning for convergence

### BitFit

**BitFit** fine-tunes only the bias terms of every linear layer while keeping all weight matrices frozen.

- **Parameter overhead**: < 0.1%
- **Implementation**: Modify only `bias` parameters in `Linear` layers
- **Performance**: Surprisingly competitive with more complex methods on certain tasks

### IA³ (Infused Adapter-by-Additive-Attention)

**IA³** applies scale-only modifications (vectors) to the attention output and feed-forward output, keeping base weights unchanged.

- **Parameter overhead**: ~0.2%
- **Mechanism**: Learnable scaling vectors rather than additive adaptations
- **Advantage**: Avoids inference latency by merging scales into frozen weights post-training

## Implementation Guide: Code Examples from the PEFT Library

The following examples use the Hugging Face `peft` library, referenced in [`resources.md`](https://github.com/chiphuyen/aie-book/blob/main/resources.md) (lines 268–270) as the de-facto standard for PEFT implementation.

### LoRA on a Causal Language Model

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig, get_peft_model

model_name = "meta-llama/Meta-Llama-3-8B"
tokenizer = AutoTokenizer.from_pretrained(model_name)

# Load the frozen base model

base_model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="auto",
    torch_dtype="auto"
)

# LoRA configuration (rank-r = 8, target modules = query/value matrices)

lora_cfg = LoraConfig(
    r=8,
    lora_alpha=16,
    target_modules=["q_proj", "v_proj"],
    bias="none",
    task_type="CAUSAL_LM"
)

# Wrap the base model with LoRA

model = get_peft_model(base_model, lora_cfg)

# Training loop

for batch in train_dataloader:
    outputs = model(**batch)
    loss = outputs.loss
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

# Save only the LoRA weights (tiny file)

model.save_pretrained("./lora_llama")

```

Only the `A` and `B` matrices for selected linear layers receive gradients; the Llama weights remain frozen on disk.

### Adapter Layers

```python
from transformers import AutoModelForSeq2SeqLM
from transformers.adapters import AdapterConfig

model_name = "google/flan-t5-base"
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)

# Add bottleneck adapter (reduction factor = 16)

adapter_cfg = AdapterConfig(
    mh_adapter=True,
    output_adapter=True,
    reduction_factor=16,
    non_linearity="relu"
)
model.add_adapter("my_adapter", config=adapter_cfg)
model.train_adapter("my_adapter")  # Freeze base model

# Fine-tune

for batch in train_loader:
    loss = model(**batch).loss
    loss.backward()
    optimizer.step()

# Export only adapter weights (~MBs vs GBs for full model)

model.save_adapter("./my_adapter", "my_adapter")

```

### Prefix-Tuning

```python
from transformers import AutoModelForCausalLM
from peft import PrefixTuningConfig, get_peft_model

model_name = "mistralai/Mistral-7B-v0.1"
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto")

prefix_cfg = PrefixTuningConfig(
    num_virtual_tokens=20,
    task_type="CAUSAL_LM"
)

model = get_peft_model(model, prefix_cfg)

```

## Key Resources in the chiphuyen/aie-book Repository

| File | Relevance to PEFT |
|------|-------------------|
| [`resources.md`](https://github.com/chiphuyen/aie-book/blob/main/resources.md) | Curated list of seminal PEFT papers and the 7,000-word adapter survey (lines 262–265). |
| [`chapter-summaries.md`](https://github.com/chiphuyen/aie-book/blob/main/chapter-summaries.md) | High-level overview of fine-tuning challenges and LoRA benefits (lines 60–62). |
| [`README.md`](https://github.com/chiphuyen/aie-book/blob/main/README.md) | Workflow integration context for AI engineers. |
| `scripts/ai-heatmap.ipynb` | Template for visualizing PEFT training metrics. |

The repository points to the original PEFT paper introducing the concept in [`resources.md`](https://github.com/chiphuyen/aie-book/blob/main/resources.md) (lines 268–270), providing both theoretical foundations and practical implementation guides.

## Summary

- **Parameter-efficient fine-tuning techniques** freeze the pretrained backbone and train small adapter modules, reducing GPU memory requirements by 90%+.
- **LoRA** and **QLoRA** dominate current practice, offering 0.1%–0.5% parameter overhead with minimal inference latency.
- **Adapter layers** provide modular task-switching but add slight inference overhead.
- **Prefix-tuning** and **prompt-tuning** manipulate the input space rather than model weights, achieving <0.1% overhead.
- The `peft` library standardizes these implementations, allowing you to wrap existing models with a single `get_peft_model()` call.

## Frequently Asked Questions

### What is the difference between LoRA and QLoRA?

**LoRA** trains low-rank decomposition matrices while keeping the base model in full precision. **QLoRA** quantizes the base model to 4-bit precision (using Normal Float 4 or FP4) during training, then dequantizes on-the-fly for the forward pass while keeping LoRA weights in 16-bit. This allows fine-tuning 70B parameter models on single 24GB GPUs without performance degradation.

### How much GPU memory can PEFT techniques save?

Full fine-tuning of a 7B parameter model requires approximately 80–120GB of GPU memory (model weights + optimizer states + gradients). Using **LoRA** reduces this to 16–24GB, while **QLoRA** can fit the same model into 8–12GB by quantizing the frozen base weights to 4-bit precision.

### Can I combine multiple PEFT methods?

Yes, though most practitioners stick to one method. You can combine **BitFit** (bias tuning) with **LoRA** for additional capacity, or use **adapter layers** alongside **prompt-tuning**. However, the `peft` library standard interface typically expects a single PEFT configuration per model wrapper. Combining methods requires careful initialization to avoid gradient conflicts.

### When should I use adapter layers versus prefix-tuning?

Choose **adapter layers** when you need task-specific modules that can be swapped at inference time (multi-tenant scenarios) and can tolerate slight latency overhead from the additional forward passes. Choose **prefix-tuning** when you need the fastest inference possible (no extra forward passes) and are working on generation tasks where soft prompt steering proves sufficient. Prefix-tuning generally requires more training steps to converge than adapters.