What Training Workflows Are Available in LlamaFactory: The Complete Guide

LlamaFactory provides seven distinct training workflows—SFT, RM, PT, PPO, KTO, DPO, and MCA—each implemented as modular run_* entry points in the src/llamafactory/train/ directory and dispatched through the run_exp function in tuner.py.

LlamaFactory is a unified framework for fine-tuning large language models that supports multiple training paradigms through configurable workflows. Understanding what training workflows are available in LlamaFactory enables practitioners to select the optimal approach for everything from supervised instruction tuning to reinforcement learning from human feedback (RLHF). Each workflow is encapsulated in its own Python module and can be invoked via the llamafactory-cli command-line interface or imported programmatically for custom pipelines.

Overview of Training Workflows

LlamaFactory organizes training logic into discrete, reusable workflows located in src/llamafactory/train/. Each workflow corresponds to a specific fine-tuning paradigm and exposes a standardized run_* function that handles model initialization, dataset preparation, and trainer instantiation.

The framework supports seven primary workflows: Supervised Fine-Tuning (SFT), Reward Modeling (RM), Pre-Training (PT), Proximal Policy Optimization (PPO), KL-Controlled Training (KTO), Direct Preference Optimization (DPO), and Multi-Modal Causal (MCA). These workflows are orchestrated by the run_exp dispatcher in src/llamafactory/train/tuner.py, which selects the appropriate module based on the --stage argument provided in the configuration.

Core Training Workflows Explained

Supervised Fine-Tuning (SFT)

The SFT workflow handles instruction tuning on labeled datasets, making it ideal for chatbots and instruction-following models. Implemented in src/llamafactory/train/sft/workflow.py via the run_sft function at line 41, this workflow utilizes the CustomSeq2SeqTrainer class to perform standard next-token prediction on prompt-response pairs.

Reward Modeling (RM)

The RM workflow trains a reward model to score generated responses, serving as a critical component for RLHF pipelines. Found in src/llamafactory/train/rm/workflow.py at line 35, the run_rm function employs the PairwiseTrainer to process preference data where one response is ranked higher than another.

Pre-Training (PT)

The PT workflow performs standard causal language model pre-training through next-token prediction on large unlabeled corpora. Located in src/llamafactory/train/pt/workflow.py at line 36, the run_pt function uses the base CustomTrainer with a causal language modeling collator to train models from scratch or continue pre-training existing checkpoints.

Proximal Policy Optimization (PPO)

The PPO workflow implements RLHF fine-tuning using the PPO algorithm with a reference model and reward model. Defined in src/llamafactory/train/ppo/workflow.py at line 34, the run_ppo function instantiates the CustomPPOTrainer to align language models with human preferences while preventing catastrophic forgetting through KL divergence penalties.

KL-Controlled Training (KTO)

The KTO workflow provides KL-controlled fine-tuning that explicitly penalizes divergence from a reference model to reduce hallucinations. The run_kto function in src/llamafactory/train/kto/workflow.py at line 35 leverages the CustomKTOTrainer to optimize models using a KL-regularized objective that maintains output quality while ensuring safety.

Direct Preference Optimization (DPO)

The DPO workflow optimizes models directly against pairwise preferences without requiring a separate reward model, offering a simpler alternative to traditional RLHF. Implemented in src/llamafactory/train/dpo/workflow.py at line 35, the run_dpo function uses the CustomDPOTrainer to align models using preference data through a classification-based objective.

Multi-Modal Causal (MCA)

The MCA workflows extend standard training paradigms to vision-language models, supporting PT, SFT, and DPO variants for multi-modal data. Located in src/llamafactory/train/mca/workflow.py at line 114, these specialized workflows adapt the standard training loops to handle image-text interleaved sequences through dedicated multi-modal adapters and collators.

Workflow Architecture and Orchestration

All training workflows share a common architecture that promotes code reusability and consistent behavior. The run_exp function in src/llamafactory/train/tuner.py (lines 91-115) serves as the central dispatcher, routing execution to the appropriate run_* function based on the --stage parameter.

Each workflow follows a standardized initialization sequence:

1. Tokenizer loading via load_tokenizer from src/llamafactory/model.py
2. Dataset construction via get_dataset from src/llamafactory/data/__init__.py
3. Trainer instantiation using workflow-specific classes such as CustomSeq2SeqTrainer, PairwiseTrainer, or CustomPPOTrainer

Configuration management is handled by src/llamafactory/v1/config/arg_parser.py, which parses the --stage flag (accepting values like sft, rm, pt, ppo, kto, dpo) alongside common hyperparameters such as --model_name_or_path and --train_file.

Running Training Workflows

Command-Line Interface

You can launch any training workflow using the llamafactory-cli command with the appropriate --stage argument. Below are minimal examples for each supported paradigm.

Supervised Fine-Tuning:

llamafactory-cli train \
  --stage sft \
  --model_name_or_path meta-llama/Meta-Llama-3-8B-Instruct \
  --train_file data/v1_sft_demo.yaml \
  --output_dir ./outputs/sft

Reward Modeling:

llamafactory-cli train \
  --stage rm \
  --model_name_or_path meta-llama/Meta-Llama-3-8B-Instruct \
  --train_file data/v1_dpo_demo.yaml \
  --output_dir ./outputs/rm

Pre-Training:

llamafactory-cli train \
  --stage pt \
  --model_name_or_path facebook/opt-125m \
  --train_file data/c4_demo.jsonl \
  --output_dir ./outputs/pt

PPO for RLHF:

llamafactory-cli train \
  --stage ppo \
  --model_name_or_path meta-llama/Meta-Llama-3-8B-Instruct \
  --train_file data/v1_sft_demo.yaml \
  --reward_model_name_or_path ./outputs/rm \
  --output_dir ./outputs/ppo

KL-Controlled Training:

llamafactory-cli train \
  --stage kto \
  --model_name_or_path meta-llama/Meta-Llama-3-8B-Instruct \
  --train_file data/v1_dpo_demo.yaml \
  --output_dir ./outputs/kto

Direct Preference Optimization:

llamafactory-cli train \
  --stage dpo \
  --model_name_or_path meta-llama/Meta-Llama-3-8B-Instruct \
  --train_file data/v1_dpo_demo.yaml \
  --output_dir ./outputs/dpo

Multi-Modal SFT:

llamafactory-cli train \
  --stage sft \
  --model_name_or_path Qwen/Qwen-VL-Chat \
  --train_file data/mllm_demo.json \
  --output_dir ./outputs/mca_sft \
  --use_mca

Programmatic Execution

For integration into custom Python applications, import the workflow functions directly from their respective modules:

from llamafactory.train.sft.workflow import run_sft

# Assuming model_args, data_args, training_args, finetuning_args, generating_args are defined

run_sft(model_args, data_args, training_args, finetuning_args, generating_args)

This pattern applies to all workflows (run_rm, run_pt, run_ppo, run_kto, run_dpo) and accepts standardized argument dataclasses parsed from configuration files or command-line arguments.

Summary

• LlamaFactory implements seven distinct training workflows: SFT, RM, PT, PPO, KTO, DPO, and MCA, each optimized for specific fine-tuning objectives.
• All workflows are dispatched through run_exp in src/llamafactory/train/tuner.py based on the --stage argument.
• Each workflow resides in its own module under src/llamafactory/train/ (e.g., sft/workflow.py, dpo/workflow.py) and exposes a run_* entry point.
• Workflows share common utilities including load_tokenizer from model.py and get_dataset from data/__init__.py to ensure consistency.
• Execution is supported via both llamafactory-cli command-line interface and direct Python imports for custom pipelines.

Frequently Asked Questions

What is the difference between SFT and DPO workflows in LlamaFactory?

The SFT workflow trains models on single-turn or multi-turn instruction-response pairs using standard supervised learning, while the DPO workflow optimizes models directly on preference pairs (chosen vs. rejected responses) without requiring a separate reward model. DPO implements a classification-based objective in CustomDPOTrainer whereas SFT uses standard next-token prediction in CustomSeq2SeqTrainer.

How do I switch between different training workflows?

Switching workflows requires changing the --stage argument when invoking llamafactory-cli train or setting the corresponding parameter when calling run_exp programmatically. Valid stage values are sft, rm, pt, ppo, kto, dpo, and mca (for multi-modal variants), with each value triggering a different run_* function in src/llamafactory/train/tuner.py.

Can LlamaFactory handle multi-modal training workflows?

Yes, the MCA (Multi-Modal Causal) workflows extend standard training paradigms to vision-language models, providing specialized implementations of PT, SFT, and DPO in src/llamafactory/train/mca/workflow.py. These workflows handle image-text interleaved sequences and are activated using the --use_mca flag alongside standard stage arguments.

When should I use the Reward Modeling workflow versus DPO?

Use the RM workflow when building a complete RLHF pipeline that includes PPO fine-tuning, as it trains a dedicated reward model to score outputs. Use DPO when you want to align models using preference data without the computational overhead of training a separate reward model and reference policy, making it suitable for simpler preference optimization tasks.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}