# Implementing Supervised Fine-Tuning (SFT) for LLMs: A Complete Guide with TRL, Unsloth, and Axolotl

> Learn Supervised Fine-Tuning SFT for LLMs with TRL Unsloth and Axolotl. Convert pre-trained models into instruction followers efficiently reducing VRAM.

- Repository: [Maxime Labonne/llm-course](https://github.com/mlabonne/llm-course)
- Tags: how-to-guide
- Published: 2026-03-01

---

**Supervised Fine-Tuning (SFT) converts pre-trained language models into instruction-following assistants by training on prompt-response pairs, typically using parameter-efficient methods like LoRA or QLoRA to reduce VRAM requirements.**

Supervised Fine-Tuning (SFT) represents the critical first stage of post-training that transforms a generic base model into a helpful AI assistant capable of following instructions and producing well-structured answers. According to the `mlabonne/llm-course` repository, this process involves teaching models to mimic desired output patterns through carefully curated datasets while managing computational costs via efficient training techniques. Whether you are working with multi-billion parameter models on consumer GPUs or large-scale clusters, implementing SFT requires understanding the distinction between full fine-tuning and parameter-efficient approaches described in the course materials.

## What Is Supervised Fine-Tuning (SFT)?

SFT is the initial post-training stage that follows pre-training, where a base model learns to follow instructions and generate helpful responses. As outlined in [`README.md`](https://github.com/mlabonne/llm-course/blob/main/README.md) (lines 221-226), this stage bridges the gap between a model that predicts the next token statistically and one that acts as a conversational assistant. The process requires high-quality instruction datasets containing prompt-response pairs, and leverages specific training hyper-parameters to guide convergence without catastrophic forgetting.

## Core Concepts in SFT Implementation

### Full Fine-Tuning vs. Parameter-Efficient Methods

[`README.md`](https://github.com/mlabonne/llm-course/blob/main/README.md) (lines 223-224) distinguishes between two fundamental approaches:

- **Full fine-tuning** updates every parameter in the model, demanding substantial VRAM and compute resources.
- **Parameter-efficient fine-tuning** (PEFT) keeps base weights frozen and trains only small adapter matrices. **LoRA** (Low-Rank Adaptation) and **QLoRA** (quantized LoRA) are the dominant PEFT methods, with QLoRA additionally quantizing the model to 4-bit precision to dramatically reduce memory footprint.

### Critical Training Hyper-Parameters

According to [`README.md`](https://github.com/mlabonne/llm-course/blob/main/README.md) (lines 224-225), successful SFT requires tuning:
- **Learning-rate schedules** and warm-up steps
- **Batch size** and gradient accumulation steps
- **Optimizer selection** (e.g., 8-bit AdamW)
- **Weight decay** and regularization
- **LoRA-specific configurations**: rank (`r`), alpha (`alpha`), and target modules (typically `q_proj` and `v_proj`)

### Distributed Training Strategies

For multi-GPU setups, the course documentation ([`README.md`](https://github.com/mlabonne/llm-course/blob/main/README.md), lines 225-226) recommends **DeepSpeed** or **FSDP** (Fully Sharded Data Parallel), both supporting gradient checkpointing to trade computation for memory savings during distributed SFT runs.

### Monitoring and Observability

Effective SFT requires tracking metrics described in [`README.md`](https://github.com/mlabonne/llm-course/blob/main/README.md) (lines 226-227):
- Loss curves across epochs
- Learning-rate schedules and warm-up phases
- Gradient norms to detect spikes or exploding gradients

## Three Production-Ready Frameworks for SFT

The `mlabonne/llm-course` recommends three mainstream frameworks, each supporting the techniques above and referenced in the "Training techniques" section ([`README.md`](https://github.com/mlabonne/llm-course/blob/main/README.md), lines 223-224).

### TRL (Transformers Reinforcement Learning)

Hugging Face’s **TRL** library provides the `SFTTrainer` class, which automates LoRA/QLoRA insertion, gradient checkpointing, and logging.

```python
from trl import SFTTrainer
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset

model_name = "meta-llama/Meta-Llama-3.1-8B"
tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    load_in_4bit=True,           # 4‑bit quantisation (optional)

    device_map="auto",
)

# Load a simple instruction dataset

train_ds = load_dataset("json", data_files="train.jsonl")["train"]

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=train_ds,
    max_seq_length=1024,
    args=dict(
        per_device_train_batch_size=4,
        gradient_accumulation_steps=8,
        learning_rate=2e-4,
        num_train_epochs=3,
        fp16=True,
        logging_steps=10,
        output_dir="./sft-trl-llama3.1",
        # LoRA parameters

        lora_r=64,
        lora_alpha=16,
        lora_dropout=0.1,
        target_modules=["q_proj","v_proj"],
    ),
)

trainer.train()

```

The `SFTTrainer` automatically handles adapter insertion and supports QLoRA when `load_in_4bit=True`, making it ideal for implementing SFT on limited hardware.

### Unsloth (Single-GPU Optimized)

**Unsloth** provides a lightweight wrapper that automates 4-bit quantization and LoRA configuration for rapid iteration on single GPUs.

```python
from unsloth import FastLoRA
from transformers import AutoTokenizer, AutoModelForCausalLM
from datasets import load_dataset

model_id = "meta-llama/Meta-Llama-3.1-8B"
tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=True)

# Unsloth wraps the model + LoRA in a single class

model = FastLoRA.from_pretrained(
    model_id,
    quantise="4bit",       # 4‑bit quantisation + LoRA

    lora_r=32,
    lora_alpha=16,
    target_modules=["q_proj","v_proj"],
)

train_ds = load_dataset("json", data_files="train.jsonl")["train"]

model.train(
    train_dataset=train_ds,
    tokenizer=tokenizer,
    epochs=2,
    batch_size=8,
    learning_rate=1e-4,
    gradient_accumulation=4,
    output_dir="unsloth_sft",
)

```

Unsloth utilizes `bitsandbytes` for quantization and optimized LoRA kernels, enabling SFT training of 7B-8B parameter models on consumer GPUs like the RTX 3090.

### Axolotl (Config-Driven Multi-GPU)

**Axolotl** excels in enterprise and research environments requiring reproducible, config-driven workflows across multiple GPUs.

Create [`axolotl_config.yaml`](https://github.com/mlabonne/llm-course/blob/main/axolotl_config.yaml):

```yaml
model_name_or_path: meta-llama/Meta-Llama-3.1-8B
tokenizer_name: meta-llama/Meta-Llama-3.1-8B
dataset:
  path: ./train.jsonl
  format: json
  field_mapping:
    prompt: instruction
    response: output
training:
  max_seq_length: 1024
  per_device_train_batch_size: 2
  gradient_accumulation_steps: 16
  learning_rate: 2e-4
  num_train_epochs: 3
  fp16: true
  logging_steps: 5
  output_dir: ./axolotl_sft
  report_to: ["tensorboard"]
lora:
  r: 64
  alpha: 32
  dropout: 0.1
  target_modules: ["q_proj","v_proj"]
  bias: "none"

```

Launch training:

```bash
accelerate launch -m axolotl.cli.train axolotl_config.yaml

```

Axolotl parses the YAML configuration to build DeepSpeed-compatible trainers, automatically applying LoRA or QLoRA (via `bitsandbytes_config` additions) and supporting the distributed training strategies referenced in [`README.md`](https://github.com/mlabonne/llm-course/blob/main/README.md) (lines 225-226).

## Summary

- **SFT** is the essential post-training step that transforms base models into instruction-following assistants, as defined in `mlabonne/llm-course` ([`README.md`](https://github.com/mlabonne/llm-course/blob/main/README.md), lines 221-226).
- **Parameter-efficient fine-tuning** via LoRA and QLoRA reduces VRAM requirements by freezing base weights and training small adapter matrices.
- **Key hyper-parameters** include learning rate, LoRA rank/alpha, and gradient accumulation settings critical for stable convergence.
- **Three recommended frameworks** provide production paths: **TRL** for Hugging Face integration, **Unsloth** for single-GPU speed, and **Axolotl** for multi-node distributed training.
- **Monitoring** loss curves and gradient norms prevents training instability during the SFT process.

## Frequently Asked Questions

### What is the difference between SFT and pre-training?

Pre-training teaches a model general language patterns and world knowledge using vast unsupervised text corpora. SFT specifically teaches the model to follow instructions and produce helpful responses using curated prompt-response pairs, converting a text-completion model into an assistant, as described in [`README.md`](https://github.com/mlabonne/llm-course/blob/main/README.md) (lines 221-226).

### How much VRAM do I need for SFT on a 7B parameter model?

Using **QLoRA** (4-bit quantization) as implemented in Unsloth or TRL, you can fine-tune 7B-8B parameter models on approximately 16-24 GB VRAM (e.g., RTX 3090 or RTX 4090). Full fine-tuning requires significantly more memory, typically 40-60 GB VRAM or distributed setups.

### Can I use SFT with multi-GPU setups?

Yes. Frameworks like Axolotl and TRL support distributed training via **DeepSpeed** or **FSDP**, allowing you to shard model parameters and gradients across multiple GPUs. The `mlabonne/llm-course` explicitly references these strategies for scaling SFT ([`README.md`](https://github.com/mlabonne/llm-course/blob/main/README.md), lines 225-226).

### What dataset format is required for SFT training?

Most SFT frameworks expect JSONL files with fields mapped to instruction and response columns. For example, Axolotl uses `field_mapping` to align `prompt` to `instruction` and `response` to `output`, while TRL’s `SFTTrainer` can ingest Hugging Face Datasets with customizable formatting functions to process conversational or instruction-following data structures.