Implementing Supervised Fine-Tuning (SFT) for LLMs: A Complete Guide with TRL, Unsloth, and Axolotl

Supervised Fine-Tuning (SFT) converts pre-trained language models into instruction-following assistants by training on prompt-response pairs, typically using parameter-efficient methods like LoRA or QLoRA to reduce VRAM requirements.

Supervised Fine-Tuning (SFT) represents the critical first stage of post-training that transforms a generic base model into a helpful AI assistant capable of following instructions and producing well-structured answers. According to the mlabonne/llm-course repository, this process involves teaching models to mimic desired output patterns through carefully curated datasets while managing computational costs via efficient training techniques. Whether you are working with multi-billion parameter models on consumer GPUs or large-scale clusters, implementing SFT requires understanding the distinction between full fine-tuning and parameter-efficient approaches described in the course materials.

What Is Supervised Fine-Tuning (SFT)?

SFT is the initial post-training stage that follows pre-training, where a base model learns to follow instructions and generate helpful responses. As outlined in README.md (lines 221-226), this stage bridges the gap between a model that predicts the next token statistically and one that acts as a conversational assistant. The process requires high-quality instruction datasets containing prompt-response pairs, and leverages specific training hyper-parameters to guide convergence without catastrophic forgetting.

Core Concepts in SFT Implementation

Full Fine-Tuning vs. Parameter-Efficient Methods

README.md (lines 223-224) distinguishes between two fundamental approaches:

• Full fine-tuning updates every parameter in the model, demanding substantial VRAM and compute resources.
• Parameter-efficient fine-tuning (PEFT) keeps base weights frozen and trains only small adapter matrices. LoRA (Low-Rank Adaptation) and QLoRA (quantized LoRA) are the dominant PEFT methods, with QLoRA additionally quantizing the model to 4-bit precision to dramatically reduce memory footprint.

Critical Training Hyper-Parameters

According to README.md (lines 224-225), successful SFT requires tuning:

• Learning-rate schedules and warm-up steps
• Batch size and gradient accumulation steps
• Optimizer selection (e.g., 8-bit AdamW)
• Weight decay and regularization
• LoRA-specific configurations: rank (r), alpha (alpha), and target modules (typically q_proj and v_proj)

Distributed Training Strategies

For multi-GPU setups, the course documentation (README.md, lines 225-226) recommends DeepSpeed or FSDP (Fully Sharded Data Parallel), both supporting gradient checkpointing to trade computation for memory savings during distributed SFT runs.

Monitoring and Observability

Effective SFT requires tracking metrics described in README.md (lines 226-227):

• Loss curves across epochs
• Learning-rate schedules and warm-up phases
• Gradient norms to detect spikes or exploding gradients

Three Production-Ready Frameworks for SFT

The mlabonne/llm-course recommends three mainstream frameworks, each supporting the techniques above and referenced in the "Training techniques" section (README.md, lines 223-224).

TRL (Transformers Reinforcement Learning)

Hugging Face’s TRL library provides the SFTTrainer class, which automates LoRA/QLoRA insertion, gradient checkpointing, and logging.

from trl import SFTTrainer
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset

model_name = "meta-llama/Meta-Llama-3.1-8B"
tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=True)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    load_in_4bit=True,           # 4‑bit quantisation (optional)

    device_map="auto",
)

# Load a simple instruction dataset

train_ds = load_dataset("json", data_files="train.jsonl")["train"]

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=train_ds,
    max_seq_length=1024,
    args=dict(
        per_device_train_batch_size=4,
        gradient_accumulation_steps=8,
        learning_rate=2e-4,
        num_train_epochs=3,
        fp16=True,
        logging_steps=10,
        output_dir="./sft-trl-llama3.1",
        # LoRA parameters

        lora_r=64,
        lora_alpha=16,
        lora_dropout=0.1,
        target_modules=["q_proj","v_proj"],
    ),
)

trainer.train()

The SFTTrainer automatically handles adapter insertion and supports QLoRA when load_in_4bit=True, making it ideal for implementing SFT on limited hardware.

Unsloth (Single-GPU Optimized)

Unsloth provides a lightweight wrapper that automates 4-bit quantization and LoRA configuration for rapid iteration on single GPUs.

from unsloth import FastLoRA
from transformers import AutoTokenizer, AutoModelForCausalLM
from datasets import load_dataset

model_id = "meta-llama/Meta-Llama-3.1-8B"
tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=True)

# Unsloth wraps the model + LoRA in a single class

model = FastLoRA.from_pretrained(
    model_id,
    quantise="4bit",       # 4‑bit quantisation + LoRA

    lora_r=32,
    lora_alpha=16,
    target_modules=["q_proj","v_proj"],
)

train_ds = load_dataset("json", data_files="train.jsonl")["train"]

model.train(
    train_dataset=train_ds,
    tokenizer=tokenizer,
    epochs=2,
    batch_size=8,
    learning_rate=1e-4,
    gradient_accumulation=4,
    output_dir="unsloth_sft",
)

Unsloth utilizes bitsandbytes for quantization and optimized LoRA kernels, enabling SFT training of 7B-8B parameter models on consumer GPUs like the RTX 3090.

Axolotl (Config-Driven Multi-GPU)

Axolotl excels in enterprise and research environments requiring reproducible, config-driven workflows across multiple GPUs.

Create axolotl_config.yaml:

model_name_or_path: meta-llama/Meta-Llama-3.1-8B
tokenizer_name: meta-llama/Meta-Llama-3.1-8B
dataset:
  path: ./train.jsonl
  format: json
  field_mapping:
    prompt: instruction
    response: output
training:
  max_seq_length: 1024
  per_device_train_batch_size: 2
  gradient_accumulation_steps: 16
  learning_rate: 2e-4
  num_train_epochs: 3
  fp16: true
  logging_steps: 5
  output_dir: ./axolotl_sft
  report_to: ["tensorboard"]
lora:
  r: 64
  alpha: 32
  dropout: 0.1
  target_modules: ["q_proj","v_proj"]
  bias: "none"

Launch training:

accelerate launch -m axolotl.cli.train axolotl_config.yaml

Axolotl parses the YAML configuration to build DeepSpeed-compatible trainers, automatically applying LoRA or QLoRA (via bitsandbytes_config additions) and supporting the distributed training strategies referenced in README.md (lines 225-226).

Summary

• SFT is the essential post-training step that transforms base models into instruction-following assistants, as defined in mlabonne/llm-course (README.md, lines 221-226).
• Parameter-efficient fine-tuning via LoRA and QLoRA reduces VRAM requirements by freezing base weights and training small adapter matrices.
• Key hyper-parameters include learning rate, LoRA rank/alpha, and gradient accumulation settings critical for stable convergence.
• Three recommended frameworks provide production paths: TRL for Hugging Face integration, Unsloth for single-GPU speed, and Axolotl for multi-node distributed training.
• Monitoring loss curves and gradient norms prevents training instability during the SFT process.

Frequently Asked Questions

What is the difference between SFT and pre-training?

Pre-training teaches a model general language patterns and world knowledge using vast unsupervised text corpora. SFT specifically teaches the model to follow instructions and produce helpful responses using curated prompt-response pairs, converting a text-completion model into an assistant, as described in README.md (lines 221-226).

How much VRAM do I need for SFT on a 7B parameter model?

Using QLoRA (4-bit quantization) as implemented in Unsloth or TRL, you can fine-tune 7B-8B parameter models on approximately 16-24 GB VRAM (e.g., RTX 3090 or RTX 4090). Full fine-tuning requires significantly more memory, typically 40-60 GB VRAM or distributed setups.

Can I use SFT with multi-GPU setups?

Yes. Frameworks like Axolotl and TRL support distributed training via DeepSpeed or FSDP, allowing you to shard model parameters and gradients across multiple GPUs. The mlabonne/llm-course explicitly references these strategies for scaling SFT (README.md, lines 225-226).

What dataset format is required for SFT training?

Most SFT frameworks expect JSONL files with fields mapped to instruction and response columns. For example, Axolotl uses field_mapping to align prompt to instruction and response to output, while TRL’s SFTTrainer can ingest Hugging Face Datasets with customizable formatting functions to process conversational or instruction-following data structures.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}