Supported Training Approaches in LlamaFactory: Complete Guide to LLM Fine-Tuning

LlamaFactory supports eight production-ready training approaches—Pre-Training, Supervised Fine-Tuning (SFT), Reward Modeling (RM), PPO, DPO, KTO, ORPO, and SimPO—that are compatible with full-tuning, LoRA, QLoRA, and other parameter-efficient fine-tuning paradigms.

LlamaFactory is a unified open-source framework for fine-tuning large language models. The repository implements a comprehensive architecture of supported training approaches that spans continual pre-training, supervised instruction tuning, and modern preference alignment algorithms. Each method is deeply integrated with the codebase through dedicated workflow modules and trainer classes.

Complete List of Supported Training Approaches

LlamaFactory enumerates its training capabilities in the README and implements them across dedicated submodules. The framework supports the following eight approaches, all of which work with Full-tuning, Freeze-tuning, LoRA, QLoRA, OFT, and QOFT:

Approach	Full	Freeze	LoRA	QLoRA	OFT	QOFT
Pre-Training	✅	✅	✅	✅	✅	✅
Supervised Fine-Tuning (SFT)	✅	✅	✅	✅	✅	✅
Reward Modeling (RM)	✅	✅	✅	✅	✅	✅
PPO Training	✅	✅	✅	✅	✅	✅
DPO Training	✅	✅	✅	✅	✅	✅
KTO Training	✅	✅	✅	✅	✅	✅
ORPO Training	✅	✅	✅	✅	✅	✅
SimPO Training	✅	✅	✅	✅	✅	✅

Architecture and Implementation

Workflow Modules and Trainer Classes

Each training approach is isolated in its own subdirectory under src/llamafactory/train/ with specialized trainer implementations:

• Pre-Training → src/llamafactory/train/pt/* uses CustomTrainer for continual pre-training on raw corpora.

• SFT → src/llamafactory/train/sft/* implements CustomSeq2SeqTrainer for instruction-based supervised fine-tuning.

• Reward Modeling → src/llamafactory/train/rm/* contains PairwiseTrainer for learning preference scores from comparison data.

• PPO → src/llamafactory/train/ppo/* provides CustomPPOTrainer extending trl.PPOTrainer for proximal policy optimization.

• DPO → src/llamafactory/train/dpo/* offers CustomDPOTrainer for direct preference optimization without explicit reward models.

• KTO → src/llamafactory/train/kto/* implements CustomKTOTrainer based on trl.KTOTrainer for Kahneman-Tversky optimization.

• ORPO → src/llamafactory/train/orpo/* mirrors DPO logic for odds ratio preference optimization.

• SimPO → src/llamafactory/train/simpo/* follows patterns similar to PPO for simple preference optimization.

Training Type Registry

The mapping from human-readable names to internal identifiers resides in src/llamafactory/extras/constants.py. For example, "Pre-Training" maps to the internal "pt" flag, while SFT maps to "sft" and DPO maps to "dpo". This registry allows the CLI and Python API to route training requests to the correct workflow modules.

How to Run Each Training Approach

Supervised Fine-Tuning (SFT)

Execute SFT via the CLI using the sft finetuning type. The workflow is orchestrated by src/llamafactory/train/sft/workflow.py and src/llamafactory/train/sft/trainer.py.

llamafactory train \
  --finetuning_type sft \
  --model_name_or_path meta-llama/Meta-Llama-3-8B \
  --dataset alpaca_en \
  --output_dir ./outputs/sft_llama3 \
  --do_train true \
  --per_device_train_batch_size 4 \
  --lr_scheduler_type cosine \
  --learning_rate 2e-5 \
  --num_train_epochs 3

Reward Modeling (RM)

Launch reward modeling through the Python API. Key files include src/llamafactory/train/rm/workflow.py and src/llamafactory/train/rm/trainer.py.

from llamafactory.train.rm.workflow import run_rm

run_rm(
    model_name_or_path="meta-llama/Meta-Llama-3-8B",
    dataset="openassistant_rlhf",
    output_dir="./outputs/rm_llama3",
    learning_rate=1e-5,
    num_train_epochs=2,
    per_device_train_batch_size=8,
)

PPO Training

PPO training leverages src/llamafactory/train/ppo/workflow.py and src/llamafactory/train/ppo/trainer.py, extending trl.PPOTrainer with custom logging hooks.

llamafactory train \
  --finetuning_type ppo \
  --model_name_or_path meta-llama/Meta-Llama-3-8B \
  --dataset openai_summarize \
  --output_dir ./outputs/ppo_llama3 \
  --do_train true \
  --ppo_epochs 4 \
  --learning_rate 5e-6 \
  --per_device_train_batch_size 2

DPO Training

Direct Preference Optimization uses src/llamafactory/train/dpo/workflow.py and src/llamafactory/train/dpo/trainer.py, wrapping trl.DPOTrainer.

from llamafactory.train.dpo.workflow import run_dpo

run_dpo(
    model_name_or_path="meta-llama/Meta-Llama-3-8B",
    dataset="hiyouga/DPO-En-Zh-20k",
    output_dir="./outputs/dpo_llama3",
    learning_rate=3e-5,
    num_train_epochs=1,
    per_device_train_batch_size=4,
)

KTO Training

Run KTO training via CLI. The implementation resides in src/llamafactory/train/kto/workflow.py and src/llamafactory/train/kto/trainer.py.

llamafactory train \
  --finetuning_type kto \
  --model_name_or_path meta-llama/Meta-Llama-3-8B \
  --dataset kto_en \
  --output_dir ./outputs/kto_llama3 \
  --do_train true \
  --learning_rate 2e-5 \
  --num_train_epochs 2

ORPO and SimPO

ORPO (Odds Ratio Preference Optimization) and SimPO (Simple Preference Optimization) follow initialization patterns similar to DPO and PPO respectively. ORPO logic is implemented in src/llamafactory/train/orpo/*, while SimPO resides in src/llamafactory/train/simpo/*, both supporting the full suite of parameter-efficient fine-tuning methods.

Summary

• LlamaFactory implements eight distinct supported training approaches through isolated workflow modules in src/llamafactory/train/.
• All approaches support Full-tuning, Freeze-tuning, LoRA, QLoRA, OFT, and QOFT paradigms.
• Pre-Training and SFT use CustomTrainer and CustomSeq2SeqTrainer for standard supervised objectives.
• RM, PPO, DPO, and KTO provide specialized trainers—PairwiseTrainer, CustomPPOTrainer, CustomDPOTrainer, and CustomKTOTrainer—for reinforcement learning and preference alignment.
• Training type identifiers are centralized in src/llamafactory/extras/constants.py to ensure consistent routing between CLI arguments and internal implementations.

Frequently Asked Questions

What is the difference between SFT and DPO in LlamaFactory?

Supervised Fine-Tuning (SFT) optimizes the model on instruction-output pairs using standard next-token prediction loss, implemented via CustomSeq2SeqTrainer in src/llamafactory/train/sft/. DPO (Direct Preference Optimization) skips explicit reward modeling and directly optimizes on pairwise preference data using the CustomDPOTrainer in src/llamafactory/train/dpo/. While SFT teaches the model format and knowledge, DPO aligns it with human preferences.

Which training approach should I use for reinforcement learning from human feedback?

For RLHF pipelines, start with Reward Modeling (RM) to train a scoring model using PairwiseTrainer, then apply PPO Training to optimize the policy against that reward model using CustomPPOTrainer. Alternatively, use DPO Training or KTO Training to skip the separate reward modeling phase and align directly from preference data.

Does LlamaFactory support full parameter fine-tuning for all training approaches?

Yes. According to the source code in src/llamafactory/extras/constants.py and the README compatibility matrix, all eight approaches—including Pre-Training, SFT, RM, PPO, DPO, KTO, ORPO, and SimPO—support Full-tuning, Freeze-tuning, LoRA, QLoRA, OFT, and QOFT. You specify the paradigm via the --finetuning_type argument in the CLI or the corresponding parameter in the Python API.

Where are the training configurations defined in the LlamaFactory source code?

Training approach identifiers and their mappings to internal flags are defined in src/llamafactory/extras/constants.py. The execution logic for each approach resides in its respective subdirectory under src/llamafactory/train/, with workflow.py files handling argument parsing and orchestration, and trainer.py files implementing the specific optimization loops.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}