# Supported Training Approaches in LlamaFactory: Complete Guide to LLM Fine-Tuning

> Explore LlamaFactory's eight supported LLM fine-tuning approaches including Pre-Training, SFT, PPO, DPO, KTO, ORPO and SimPO. Achieve efficient fine-tuning with LoRA and QLoRA.

- Repository: [Yaowei Zheng/LlamaFactory](https://github.com/hiyouga/LlamaFactory)
- Tags: how-to-guide
- Published: 2026-03-04

---

**LlamaFactory supports eight production-ready training approaches—Pre-Training, Supervised Fine-Tuning (SFT), Reward Modeling (RM), PPO, DPO, KTO, ORPO, and SimPO—that are compatible with full-tuning, LoRA, QLoRA, and other parameter-efficient fine-tuning paradigms.**

LlamaFactory is a unified open-source framework for fine-tuning large language models. The repository implements a comprehensive architecture of **supported training approaches** that spans continual pre-training, supervised instruction tuning, and modern preference alignment algorithms. Each method is deeply integrated with the codebase through dedicated workflow modules and trainer classes.

## Complete List of Supported Training Approaches

LlamaFactory enumerates its training capabilities in the README and implements them across dedicated submodules. The framework supports the following eight approaches, all of which work with **Full-tuning**, **Freeze-tuning**, **LoRA**, **QLoRA**, **OFT**, and **QOFT**:

| Approach | Full | Freeze | LoRA | QLoRA | OFT | QOFT |
|----------|------|--------|------|-------|-----|------|
| **Pre-Training** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| **Supervised Fine-Tuning (SFT)** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| **Reward Modeling (RM)** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| **PPO Training** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| **DPO Training** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| **KTO Training** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| **ORPO Training** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
| **SimPO Training** | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |

## Architecture and Implementation

### Workflow Modules and Trainer Classes

Each training approach is isolated in its own subdirectory under `src/llamafactory/train/` with specialized trainer implementations:

- **Pre-Training** → `src/llamafactory/train/pt/*` uses `CustomTrainer` for continual pre-training on raw corpora.

- **SFT** → `src/llamafactory/train/sft/*` implements `CustomSeq2SeqTrainer` for instruction-based supervised fine-tuning.

- **Reward Modeling** → `src/llamafactory/train/rm/*` contains `PairwiseTrainer` for learning preference scores from comparison data.

- **PPO** → `src/llamafactory/train/ppo/*` provides `CustomPPOTrainer` extending `trl.PPOTrainer` for proximal policy optimization.

- **DPO** → `src/llamafactory/train/dpo/*` offers `CustomDPOTrainer` for direct preference optimization without explicit reward models.

- **KTO** → `src/llamafactory/train/kto/*` implements `CustomKTOTrainer` based on `trl.KTOTrainer` for Kahneman-Tversky optimization.

- **ORPO** → `src/llamafactory/train/orpo/*` mirrors DPO logic for odds ratio preference optimization.

- **SimPO** → `src/llamafactory/train/simpo/*` follows patterns similar to PPO for simple preference optimization.

### Training Type Registry

The mapping from human-readable names to internal identifiers resides in [`src/llamafactory/extras/constants.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/extras/constants.py). For example, `"Pre-Training"` maps to the internal `"pt"` flag, while SFT maps to `"sft"` and DPO maps to `"dpo"`. This registry allows the CLI and Python API to route training requests to the correct workflow modules.

## How to Run Each Training Approach

### Supervised Fine-Tuning (SFT)

Execute SFT via the CLI using the `sft` finetuning type. The workflow is orchestrated by [`src/llamafactory/train/sft/workflow.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/train/sft/workflow.py) and [`src/llamafactory/train/sft/trainer.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/train/sft/trainer.py).

```bash
llamafactory train \
  --finetuning_type sft \
  --model_name_or_path meta-llama/Meta-Llama-3-8B \
  --dataset alpaca_en \
  --output_dir ./outputs/sft_llama3 \
  --do_train true \
  --per_device_train_batch_size 4 \
  --lr_scheduler_type cosine \
  --learning_rate 2e-5 \
  --num_train_epochs 3

```

### Reward Modeling (RM)

Launch reward modeling through the Python API. Key files include [`src/llamafactory/train/rm/workflow.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/train/rm/workflow.py) and [`src/llamafactory/train/rm/trainer.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/train/rm/trainer.py).

```python
from llamafactory.train.rm.workflow import run_rm

run_rm(
    model_name_or_path="meta-llama/Meta-Llama-3-8B",
    dataset="openassistant_rlhf",
    output_dir="./outputs/rm_llama3",
    learning_rate=1e-5,
    num_train_epochs=2,
    per_device_train_batch_size=8,
)

```

### PPO Training

PPO training leverages [`src/llamafactory/train/ppo/workflow.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/train/ppo/workflow.py) and [`src/llamafactory/train/ppo/trainer.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/train/ppo/trainer.py), extending `trl.PPOTrainer` with custom logging hooks.

```bash
llamafactory train \
  --finetuning_type ppo \
  --model_name_or_path meta-llama/Meta-Llama-3-8B \
  --dataset openai_summarize \
  --output_dir ./outputs/ppo_llama3 \
  --do_train true \
  --ppo_epochs 4 \
  --learning_rate 5e-6 \
  --per_device_train_batch_size 2

```

### DPO Training

Direct Preference Optimization uses [`src/llamafactory/train/dpo/workflow.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/train/dpo/workflow.py) and [`src/llamafactory/train/dpo/trainer.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/train/dpo/trainer.py), wrapping `trl.DPOTrainer`.

```python
from llamafactory.train.dpo.workflow import run_dpo

run_dpo(
    model_name_or_path="meta-llama/Meta-Llama-3-8B",
    dataset="hiyouga/DPO-En-Zh-20k",
    output_dir="./outputs/dpo_llama3",
    learning_rate=3e-5,
    num_train_epochs=1,
    per_device_train_batch_size=4,
)

```

### KTO Training

Run KTO training via CLI. The implementation resides in [`src/llamafactory/train/kto/workflow.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/train/kto/workflow.py) and [`src/llamafactory/train/kto/trainer.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/train/kto/trainer.py).

```bash
llamafactory train \
  --finetuning_type kto \
  --model_name_or_path meta-llama/Meta-Llama-3-8B \
  --dataset kto_en \
  --output_dir ./outputs/kto_llama3 \
  --do_train true \
  --learning_rate 2e-5 \
  --num_train_epochs 2

```

### ORPO and SimPO

**ORPO** (Odds Ratio Preference Optimization) and **SimPO** (Simple Preference Optimization) follow initialization patterns similar to DPO and PPO respectively. ORPO logic is implemented in `src/llamafactory/train/orpo/*`, while SimPO resides in `src/llamafactory/train/simpo/*`, both supporting the full suite of parameter-efficient fine-tuning methods.

## Summary

- LlamaFactory implements eight distinct **supported training approaches** through isolated workflow modules in `src/llamafactory/train/`.
- All approaches support **Full-tuning**, **Freeze-tuning**, **LoRA**, **QLoRA**, **OFT**, and **QOFT** paradigms.
- **Pre-Training** and **SFT** use `CustomTrainer` and `CustomSeq2SeqTrainer` for standard supervised objectives.
- **RM**, **PPO**, **DPO**, and **KTO** provide specialized trainers—`PairwiseTrainer`, `CustomPPOTrainer`, `CustomDPOTrainer`, and `CustomKTOTrainer`—for reinforcement learning and preference alignment.
- Training type identifiers are centralized in [`src/llamafactory/extras/constants.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/extras/constants.py) to ensure consistent routing between CLI arguments and internal implementations.

## Frequently Asked Questions

### What is the difference between SFT and DPO in LlamaFactory?

**Supervised Fine-Tuning (SFT)** optimizes the model on instruction-output pairs using standard next-token prediction loss, implemented via `CustomSeq2SeqTrainer` in `src/llamafactory/train/sft/`. **DPO (Direct Preference Optimization)** skips explicit reward modeling and directly optimizes on pairwise preference data using the `CustomDPOTrainer` in `src/llamafactory/train/dpo/`. While SFT teaches the model format and knowledge, DPO aligns it with human preferences.

### Which training approach should I use for reinforcement learning from human feedback?

For RLHF pipelines, start with **Reward Modeling (RM)** to train a scoring model using `PairwiseTrainer`, then apply **PPO Training** to optimize the policy against that reward model using `CustomPPOTrainer`. Alternatively, use **DPO Training** or **KTO Training** to skip the separate reward modeling phase and align directly from preference data.

### Does LlamaFactory support full parameter fine-tuning for all training approaches?

Yes. According to the source code in [`src/llamafactory/extras/constants.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/extras/constants.py) and the README compatibility matrix, all eight approaches—including Pre-Training, SFT, RM, PPO, DPO, KTO, ORPO, and SimPO—support **Full-tuning**, **Freeze-tuning**, **LoRA**, **QLoRA**, **OFT**, and **QOFT**. You specify the paradigm via the `--finetuning_type` argument in the CLI or the corresponding parameter in the Python API.

### Where are the training configurations defined in the LlamaFactory source code?

Training approach identifiers and their mappings to internal flags are defined in [`src/llamafactory/extras/constants.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/extras/constants.py). The execution logic for each approach resides in its respective subdirectory under `src/llamafactory/train/`, with [`workflow.py`](https://github.com/hiyouga/LlamaFactory/blob/main/workflow.py) files handling argument parsing and orchestration, and [`trainer.py`](https://github.com/hiyouga/LlamaFactory/blob/main/trainer.py) files implementing the specific optimization loops.