# What Training Workflows Are Available in LlamaFactory: The Complete Guide

> Explore LlamaFactory's seven training workflows: SFT, RM, PT, PPO, KTO, DPO, and MCA. This guide details each modular entry point for efficient model fine-tuning.

- Repository: [Yaowei Zheng/LlamaFactory](https://github.com/hiyouga/LlamaFactory)
- Tags: how-to-guide
- Published: 2026-03-04

---

**LlamaFactory provides seven distinct training workflows—SFT, RM, PT, PPO, KTO, DPO, and MCA—each implemented as modular `run_*` entry points in the `src/llamafactory/train/` directory and dispatched through the `run_exp` function in [`tuner.py`](https://github.com/hiyouga/LlamaFactory/blob/main/tuner.py).**

LlamaFactory is a unified framework for fine-tuning large language models that supports multiple training paradigms through configurable workflows. Understanding what training workflows are available in LlamaFactory enables practitioners to select the optimal approach for everything from supervised instruction tuning to reinforcement learning from human feedback (RLHF). Each workflow is encapsulated in its own Python module and can be invoked via the `llamafactory-cli` command-line interface or imported programmatically for custom pipelines.

## Overview of Training Workflows

LlamaFactory organizes training logic into discrete, reusable workflows located in `src/llamafactory/train/`. Each workflow corresponds to a specific fine-tuning paradigm and exposes a standardized `run_*` function that handles model initialization, dataset preparation, and trainer instantiation.

The framework supports seven primary workflows: **Supervised Fine-Tuning (SFT)**, **Reward Modeling (RM)**, **Pre-Training (PT)**, **Proximal Policy Optimization (PPO)**, **KL-Controlled Training (KTO)**, **Direct Preference Optimization (DPO)**, and **Multi-Modal Causal (MCA)**. These workflows are orchestrated by the `run_exp` dispatcher in [`src/llamafactory/train/tuner.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/train/tuner.py), which selects the appropriate module based on the `--stage` argument provided in the configuration.

## Core Training Workflows Explained

### Supervised Fine-Tuning (SFT)

The **SFT workflow** handles instruction tuning on labeled datasets, making it ideal for chatbots and instruction-following models. Implemented in [`src/llamafactory/train/sft/workflow.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/train/sft/workflow.py) via the `run_sft` function at line 41, this workflow utilizes the `CustomSeq2SeqTrainer` class to perform standard next-token prediction on prompt-response pairs.

### Reward Modeling (RM)

The **RM workflow** trains a reward model to score generated responses, serving as a critical component for RLHF pipelines. Found in [`src/llamafactory/train/rm/workflow.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/train/rm/workflow.py) at line 35, the `run_rm` function employs the `PairwiseTrainer` to process preference data where one response is ranked higher than another.

### Pre-Training (PT)

The **PT workflow** performs standard causal language model pre-training through next-token prediction on large unlabeled corpora. Located in [`src/llamafactory/train/pt/workflow.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/train/pt/workflow.py) at line 36, the `run_pt` function uses the base `CustomTrainer` with a causal language modeling collator to train models from scratch or continue pre-training existing checkpoints.

### Proximal Policy Optimization (PPO)

The **PPO workflow** implements RLHF fine-tuning using the PPO algorithm with a reference model and reward model. Defined in [`src/llamafactory/train/ppo/workflow.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/train/ppo/workflow.py) at line 34, the `run_ppo` function instantiates the `CustomPPOTrainer` to align language models with human preferences while preventing catastrophic forgetting through KL divergence penalties.

### KL-Controlled Training (KTO)

The **KTO workflow** provides KL-controlled fine-tuning that explicitly penalizes divergence from a reference model to reduce hallucinations. The `run_kto` function in [`src/llamafactory/train/kto/workflow.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/train/kto/workflow.py) at line 35 leverages the `CustomKTOTrainer` to optimize models using a KL-regularized objective that maintains output quality while ensuring safety.

### Direct Preference Optimization (DPO)

The **DPO workflow** optimizes models directly against pairwise preferences without requiring a separate reward model, offering a simpler alternative to traditional RLHF. Implemented in [`src/llamafactory/train/dpo/workflow.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/train/dpo/workflow.py) at line 35, the `run_dpo` function uses the `CustomDPOTrainer` to align models using preference data through a classification-based objective.

### Multi-Modal Causal (MCA)

The **MCA workflows** extend standard training paradigms to vision-language models, supporting PT, SFT, and DPO variants for multi-modal data. Located in [`src/llamafactory/train/mca/workflow.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/train/mca/workflow.py) at line 114, these specialized workflows adapt the standard training loops to handle image-text interleaved sequences through dedicated multi-modal adapters and collators.

## Workflow Architecture and Orchestration

All training workflows share a common architecture that promotes code reusability and consistent behavior. The `run_exp` function in [`src/llamafactory/train/tuner.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/train/tuner.py) (lines 91-115) serves as the central dispatcher, routing execution to the appropriate `run_*` function based on the `--stage` parameter.

Each workflow follows a standardized initialization sequence:

1. **Tokenizer loading** via `load_tokenizer` from [`src/llamafactory/model.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/model.py)
2. **Dataset construction** via `get_dataset` from [`src/llamafactory/data/__init__.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/data/__init__.py)
3. **Trainer instantiation** using workflow-specific classes such as `CustomSeq2SeqTrainer`, `PairwiseTrainer`, or `CustomPPOTrainer`

Configuration management is handled by [`src/llamafactory/v1/config/arg_parser.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/v1/config/arg_parser.py), which parses the `--stage` flag (accepting values like `sft`, `rm`, `pt`, `ppo`, `kto`, `dpo`) alongside common hyperparameters such as `--model_name_or_path` and `--train_file`.

## Running Training Workflows

### Command-Line Interface

You can launch any training workflow using the `llamafactory-cli` command with the appropriate `--stage` argument. Below are minimal examples for each supported paradigm.

Supervised Fine-Tuning:

```bash
llamafactory-cli train \
  --stage sft \
  --model_name_or_path meta-llama/Meta-Llama-3-8B-Instruct \
  --train_file data/v1_sft_demo.yaml \
  --output_dir ./outputs/sft

```

Reward Modeling:

```bash
llamafactory-cli train \
  --stage rm \
  --model_name_or_path meta-llama/Meta-Llama-3-8B-Instruct \
  --train_file data/v1_dpo_demo.yaml \
  --output_dir ./outputs/rm

```

Pre-Training:

```bash
llamafactory-cli train \
  --stage pt \
  --model_name_or_path facebook/opt-125m \
  --train_file data/c4_demo.jsonl \
  --output_dir ./outputs/pt

```

PPO for RLHF:

```bash
llamafactory-cli train \
  --stage ppo \
  --model_name_or_path meta-llama/Meta-Llama-3-8B-Instruct \
  --train_file data/v1_sft_demo.yaml \
  --reward_model_name_or_path ./outputs/rm \
  --output_dir ./outputs/ppo

```

KL-Controlled Training:

```bash
llamafactory-cli train \
  --stage kto \
  --model_name_or_path meta-llama/Meta-Llama-3-8B-Instruct \
  --train_file data/v1_dpo_demo.yaml \
  --output_dir ./outputs/kto

```

Direct Preference Optimization:

```bash
llamafactory-cli train \
  --stage dpo \
  --model_name_or_path meta-llama/Meta-Llama-3-8B-Instruct \
  --train_file data/v1_dpo_demo.yaml \
  --output_dir ./outputs/dpo

```

Multi-Modal SFT:

```bash
llamafactory-cli train \
  --stage sft \
  --model_name_or_path Qwen/Qwen-VL-Chat \
  --train_file data/mllm_demo.json \
  --output_dir ./outputs/mca_sft \
  --use_mca

```

### Programmatic Execution

For integration into custom Python applications, import the workflow functions directly from their respective modules:

```python
from llamafactory.train.sft.workflow import run_sft

# Assuming model_args, data_args, training_args, finetuning_args, generating_args are defined

run_sft(model_args, data_args, training_args, finetuning_args, generating_args)

```

This pattern applies to all workflows (`run_rm`, `run_pt`, `run_ppo`, `run_kto`, `run_dpo`) and accepts standardized argument dataclasses parsed from configuration files or command-line arguments.

## Summary

- LlamaFactory implements **seven distinct training workflows**: SFT, RM, PT, PPO, KTO, DPO, and MCA, each optimized for specific fine-tuning objectives.
- All workflows are dispatched through **`run_exp`** in [`src/llamafactory/train/tuner.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/train/tuner.py) based on the `--stage` argument.
- Each workflow resides in its own module under `src/llamafactory/train/` (e.g., [`sft/workflow.py`](https://github.com/hiyouga/LlamaFactory/blob/main/sft/workflow.py), [`dpo/workflow.py`](https://github.com/hiyouga/LlamaFactory/blob/main/dpo/workflow.py)) and exposes a `run_*` entry point.
- Workflows share common utilities including **`load_tokenizer`** from [`model.py`](https://github.com/hiyouga/LlamaFactory/blob/main/model.py) and **`get_dataset`** from [`data/__init__.py`](https://github.com/hiyouga/LlamaFactory/blob/main/data/__init__.py) to ensure consistency.
- Execution is supported via both **`llamafactory-cli`** command-line interface and direct Python imports for custom pipelines.

## Frequently Asked Questions

### What is the difference between SFT and DPO workflows in LlamaFactory?

The **SFT workflow** trains models on single-turn or multi-turn instruction-response pairs using standard supervised learning, while the **DPO workflow** optimizes models directly on preference pairs (chosen vs. rejected responses) without requiring a separate reward model. DPO implements a classification-based objective in `CustomDPOTrainer` whereas SFT uses standard next-token prediction in `CustomSeq2SeqTrainer`.

### How do I switch between different training workflows?

Switching workflows requires changing the `--stage` argument when invoking `llamafactory-cli train` or setting the corresponding parameter when calling `run_exp` programmatically. Valid stage values are `sft`, `rm`, `pt`, `ppo`, `kto`, `dpo`, and `mca` (for multi-modal variants), with each value triggering a different `run_*` function in [`src/llamafactory/train/tuner.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/train/tuner.py).

### Can LlamaFactory handle multi-modal training workflows?

Yes, the **MCA (Multi-Modal Causal)** workflows extend standard training paradigms to vision-language models, providing specialized implementations of PT, SFT, and DPO in [`src/llamafactory/train/mca/workflow.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/train/mca/workflow.py). These workflows handle image-text interleaved sequences and are activated using the `--use_mca` flag alongside standard stage arguments.

### When should I use the Reward Modeling workflow versus DPO?

Use the **RM workflow** when building a complete RLHF pipeline that includes PPO fine-tuning, as it trains a dedicated reward model to score outputs. Use **DPO** when you want to align models using preference data without the computational overhead of training a separate reward model and reference policy, making it suitable for simpler preference optimization tasks.