# How LlamaFactory Orchestrates the Training Process: From CLI to Distributed Workflows

> Discover how LlamaFactory orchestrates training from CLI to distributed workflows. Learn its dispatch mechanism and stage-specific training paths for efficient LLM fine-tuning.

- Repository: [Yaowei Zheng/LlamaFactory](https://github.com/hiyouga/LlamaFactory)
- Tags: internals
- Published: 2026-03-04

---

**LlamaFactory orchestrates the training process through a centralized `run_exp()` function that dispatches to either single-process or Ray-based distributed execution paths, ultimately delegating to stage-specific workflows for PT, SFT, DPO, PPO, KTO, and RM training.**

The `hiyouga/LlamaFactory` repository implements a modular orchestration layer that abstracts the complexity of large language model fine-tuning behind a unified interface. Whether you are running pre-training on a single GPU or launching a distributed DPO job across a Ray cluster, the same underlying pipeline handles argument parsing, resource management, callback injection, and model export.

## CLI Entry Point and Central Dispatch

### The Minimal Entry in [`src/train.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/train.py)

The journey begins at [`src/train.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/train.py), which defines a minimal `main()` function that immediately delegates to the central orchestrator. This design keeps the entry point lightweight and pushes all logic into the library code.

```python

# src/train.py (lines 18-29)

def main():
    run_exp()

```

### Argument Parsing and Ray Detection in `run_exp()`

The `run_exp()` function in [`src/llamafactory/train/tuner.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/train/tuner.py) (lines 115-126) serves as the primary traffic controller. It first normalizes command-line arguments using `read_args()`, checks for help flags, and extracts Ray configuration via `get_ray_args()`.

Based on the `ray_args.use_ray` flag, it dispatches to either the distributed or single-process path:

```python
def run_exp(...):
    args = read_args(args)
    if "-h" in args or "--help" in args:
        get_train_args(args)
    ray_args = get_ray_args(args)
    callbacks = callbacks or []
    if ray_args.use_ray:
        _ray_training_function(ray_args, config={"args": args, "callbacks": callbacks})
    else:
        _training_function(config={"args": args, "callbacks": callbacks})

```

## Single-Process vs. Distributed Execution Paths

### The `_training_function()` Pipeline

For standard single-GPU or multi-GPU (non-Ray) execution, `_training_function()` (lines 57-104 in [`tuner.py`](https://github.com/hiyouga/LlamaFactory/blob/main/tuner.py)) builds the complete `TrainerArguments` tuple, registers core callbacks, and switches to the appropriate stage workflow.

The execution flow follows this pattern:

```text
_training_function()
├─ parse args → model_args, data_args, training_args, finetuning_args, generating_args
├─ add core callbacks (LogCallback, PissaConvertCallback, SwanLabCallback, EarlyStoppingCallback, ReporterCallback)
└─ stage switch → run_pt / run_sft / run_rm / run_ppo / run_dpo / run_kto

```

### Ray-Based Distributed Training with `_ray_training_function()`

When scaling across multiple nodes, `_ray_training_function()` (lines 51-86) orchestrates the distributed environment. It creates a Ray **placement group** to reserve the required GPUs or NPUs, launches a `Worker` process on each rank, and forwards the configuration to each worker's training function.

The **Worker** class (lines 27-49) handles device visibility setup before invoking the shared `_training_function()`:

```text
Worker.__init__() → _setup_env_visible_devices() → set LOCAL_RANK
Worker._training_function() → _training_function(config)

```

This design ensures that the same training logic executes whether on a single laptop or a 64-GPU cluster, with Ray handling the distribution concerns.

## Stage-Specific Workflow Implementation

### Common Workflow Pattern Across All Stages

Each fine-tuning stage (PT, SFT, RM, PPO, DPO, KTO) follows a consistent workflow pattern implemented in dedicated modules under `src/llamafactory/train/`. The typical sequence includes:

- **Load tokenizer and template** via `load_tokenizer()` and `get_template_and_fix_tokenizer()`
- **Prepare dataset** using `get_dataset()`
- **Instantiate model** through `load_model()`
- **Create data collator** (e.g., `SFTDataCollatorWith4DAttentionMask` for SFT, `DataCollatorForLanguageModeling` for PT)
- **Build a Trainer** (`CustomSeq2SeqTrainer`, `CustomTrainer`, or `KTrainer` for KTO)
- **Execute train/eval/predict** and optionally plot loss curves
- **Generate model card** and push to Hub if configured

### SFT and PT Workflow Examples

The **SFT workflow** in [`src/llamafactory/train/sft/workflow.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/train/sft/workflow.py) (lines 41-78) demonstrates this pattern for supervised fine-tuning, handling the 4D attention mask creation and sequence-to-sequence trainer setup.

The **PT workflow** in [`src/llamafactory/train/pt/workflow.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/train/pt/workflow.py) (lines 36-60) implements causal language modeling pre-training, using the standard `DataCollatorForLanguageModeling` and the base `CustomTrainer`.

## Callback Injection and Model Export

Before the Trainer executes, `_training_function` injects core callbacks (lines 62-73 in [`tuner.py`](https://github.com/hiyouga/LlamaFactory/blob/main/tuner.py)) to handle cross-cutting concerns:

- **`LogCallback`** – Unified metric logging
- **`PissaConvertCallback`** – Optional PiSSA format conversion
- **`SwanLabCallback`** – Integration with SwanLab experiment tracking
- **`EarlyStoppingCallback`** – Training termination on plateau
- **`ReporterCallback`** – Metadata gathering for model cards

After training completes, `export_model()` (in [`tuner.py`](https://github.com/hiyouga/LlamaFactory/blob/main/tuner.py)) handles serialization, optional value-head copying, tokenizer saving, and even generates an Ollama `Modelfile` for easy deployment.

## Summary

- **Centralized orchestration** happens in `run_exp()` within [`src/llamafactory/train/tuner.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/train/tuner.py), which acts as the main dispatcher for all training jobs.
- **Dual execution paths** support both single-process training via `_training_function()` and distributed Ray-based training via `_ray_training_function()` using the `Worker` class.
- **Stage-specific workflows** for PT, SFT, DPO, PPO, KTO, and RM follow a consistent pattern: load tokenizer → prepare dataset → load model → create collator → build trainer → train → export.
- **Callback injection** before training ensures consistent logging, monitoring, and model card generation across all stages.
- **Unified export** via `export_model()` handles serialization and deployment artifacts regardless of the training stage or execution mode.

## Frequently Asked Questions

### What is the entry point for starting a training job in LlamaFactory?

The entry point is [`src/train.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/train.py), which contains a minimal `main()` function that immediately delegates to `run_exp()` in [`src/llamafactory/train/tuner.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/llamafactory/train/tuner.py). This design keeps the CLI thin and pushes all orchestration logic into the library.

### How does LlamaFactory handle distributed training across multiple nodes?

LlamaFactory uses Ray for distributed orchestration. When `--use_ray` is enabled, `_ray_training_function()` creates a Ray placement group to reserve GPUs/NPUs, then launches `Worker` processes on each rank. Each worker sets its `LOCAL_RANK` environment variable and calls the same `_training_function()` used in single-process mode, ensuring identical execution logic across both paths.

### Can I use LlamaFactory's training orchestration programmatically without the CLI?

Yes. You can import `run_exp()` from `llamafactory.train.tuner` and pass a dictionary of arguments that mirrors the CLI flags. This allows you to trigger PT, SFT, DPO, or other stages directly from Python scripts or Jupyter notebooks while maintaining the same callback injection and export functionality.