How LlamaFactory Orchestrates the Training Process: From CLI to Distributed Workflows

LlamaFactory orchestrates the training process through a centralized run_exp() function that dispatches to either single-process or Ray-based distributed execution paths, ultimately delegating to stage-specific workflows for PT, SFT, DPO, PPO, KTO, and RM training.

The hiyouga/LlamaFactory repository implements a modular orchestration layer that abstracts the complexity of large language model fine-tuning behind a unified interface. Whether you are running pre-training on a single GPU or launching a distributed DPO job across a Ray cluster, the same underlying pipeline handles argument parsing, resource management, callback injection, and model export.

CLI Entry Point and Central Dispatch

The Minimal Entry in src/train.py

The journey begins at src/train.py, which defines a minimal main() function that immediately delegates to the central orchestrator. This design keeps the entry point lightweight and pushes all logic into the library code.

# src/train.py (lines 18-29)

def main():
    run_exp()

Argument Parsing and Ray Detection in run_exp()

The run_exp() function in src/llamafactory/train/tuner.py (lines 115-126) serves as the primary traffic controller. It first normalizes command-line arguments using read_args(), checks for help flags, and extracts Ray configuration via get_ray_args().

Based on the ray_args.use_ray flag, it dispatches to either the distributed or single-process path:

def run_exp(...):
    args = read_args(args)
    if "-h" in args or "--help" in args:
        get_train_args(args)
    ray_args = get_ray_args(args)
    callbacks = callbacks or []
    if ray_args.use_ray:
        _ray_training_function(ray_args, config={"args": args, "callbacks": callbacks})
    else:
        _training_function(config={"args": args, "callbacks": callbacks})

Single-Process vs. Distributed Execution Paths

The _training_function() Pipeline

For standard single-GPU or multi-GPU (non-Ray) execution, _training_function() (lines 57-104 in tuner.py) builds the complete TrainerArguments tuple, registers core callbacks, and switches to the appropriate stage workflow.

The execution flow follows this pattern:

_training_function()
├─ parse args → model_args, data_args, training_args, finetuning_args, generating_args
├─ add core callbacks (LogCallback, PissaConvertCallback, SwanLabCallback, EarlyStoppingCallback, ReporterCallback)
└─ stage switch → run_pt / run_sft / run_rm / run_ppo / run_dpo / run_kto

Ray-Based Distributed Training with _ray_training_function()

When scaling across multiple nodes, _ray_training_function() (lines 51-86) orchestrates the distributed environment. It creates a Ray placement group to reserve the required GPUs or NPUs, launches a Worker process on each rank, and forwards the configuration to each worker's training function.

The Worker class (lines 27-49) handles device visibility setup before invoking the shared _training_function():

Worker.__init__() → _setup_env_visible_devices() → set LOCAL_RANK
Worker._training_function() → _training_function(config)

This design ensures that the same training logic executes whether on a single laptop or a 64-GPU cluster, with Ray handling the distribution concerns.

Stage-Specific Workflow Implementation

Common Workflow Pattern Across All Stages

Each fine-tuning stage (PT, SFT, RM, PPO, DPO, KTO) follows a consistent workflow pattern implemented in dedicated modules under src/llamafactory/train/. The typical sequence includes:

• Load tokenizer and template via load_tokenizer() and get_template_and_fix_tokenizer()
• Prepare dataset using get_dataset()
• Instantiate model through load_model()
• Create data collator (e.g., SFTDataCollatorWith4DAttentionMask for SFT, DataCollatorForLanguageModeling for PT)
• Build a Trainer (CustomSeq2SeqTrainer, CustomTrainer, or KTrainer for KTO)
• Execute train/eval/predict and optionally plot loss curves
• Generate model card and push to Hub if configured

SFT and PT Workflow Examples

The SFT workflow in src/llamafactory/train/sft/workflow.py (lines 41-78) demonstrates this pattern for supervised fine-tuning, handling the 4D attention mask creation and sequence-to-sequence trainer setup.

The PT workflow in src/llamafactory/train/pt/workflow.py (lines 36-60) implements causal language modeling pre-training, using the standard DataCollatorForLanguageModeling and the base CustomTrainer.

Callback Injection and Model Export

Before the Trainer executes, _training_function injects core callbacks (lines 62-73 in tuner.py) to handle cross-cutting concerns:

• LogCallback – Unified metric logging
• PissaConvertCallback – Optional PiSSA format conversion
• SwanLabCallback – Integration with SwanLab experiment tracking
• EarlyStoppingCallback – Training termination on plateau
• ReporterCallback – Metadata gathering for model cards

After training completes, export_model() (in tuner.py) handles serialization, optional value-head copying, tokenizer saving, and even generates an Ollama Modelfile for easy deployment.

Summary

• Centralized orchestration happens in run_exp() within src/llamafactory/train/tuner.py, which acts as the main dispatcher for all training jobs.
• Dual execution paths support both single-process training via _training_function() and distributed Ray-based training via _ray_training_function() using the Worker class.
• Stage-specific workflows for PT, SFT, DPO, PPO, KTO, and RM follow a consistent pattern: load tokenizer → prepare dataset → load model → create collator → build trainer → train → export.
• Callback injection before training ensures consistent logging, monitoring, and model card generation across all stages.
• Unified export via export_model() handles serialization and deployment artifacts regardless of the training stage or execution mode.

Frequently Asked Questions

What is the entry point for starting a training job in LlamaFactory?

The entry point is src/train.py, which contains a minimal main() function that immediately delegates to run_exp() in src/llamafactory/train/tuner.py. This design keeps the CLI thin and pushes all orchestration logic into the library.

How does LlamaFactory handle distributed training across multiple nodes?

LlamaFactory uses Ray for distributed orchestration. When --use_ray is enabled, _ray_training_function() creates a Ray placement group to reserve GPUs/NPUs, then launches Worker processes on each rank. Each worker sets its LOCAL_RANK environment variable and calls the same _training_function() used in single-process mode, ensuring identical execution logic across both paths.

Can I use LlamaFactory's training orchestration programmatically without the CLI?

Yes. You can import run_exp() from llamafactory.train.tuner and pass a dictionary of arguments that mirrors the CLI flags. This allows you to trigger PT, SFT, DPO, or other stages directly from Python scripts or Jupyter notebooks while maintaining the same callback injection and export functionality.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}