# How LlamaFactory Facilitates Data Processing and Formatting for Training: A Deep Dive into the Pipeline

> Discover LlamaFactory's four-stage pipeline: Load, Convert, Process, and Collate. Streamline your data processing and formatting for efficient AI model training. Learn more.

- Repository: [Yaowei Zheng/LlamaFactory](https://github.com/hiyouga/LlamaFactory)
- Tags: deep-dive
- Published: 2026-03-04

---

**LlamaFactory provides a modular, four-stage pipeline—Load, Convert, Process, and Collate—that transforms raw datasets into token-level tensors ready for fine-tuning, supporting everything from supervised fine-tuning to multimodal RLHF.**

LlamaFactory is an open-source unified framework for fine-tuning large language models. Its data processing and formatting for training is built around a flexible, plug-and-play architecture that handles JSON, CSV, Hugging Face Hub datasets, and local files, converting them into the exact token-level tensors required by a 🤗 Transformers trainer.

## The Four-Stage Data Pipeline Architecture

The pipeline consists of four tightly integrated components that work sequentially to prepare training data.

### Stage 1: Loading Raw Datasets

The entry point for LlamaFactory data processing is [`llamafactory/data/loader.py`](https://github.com/hiyouga/LlamaFactory/blob/main/llamafactory/data/loader.py). The `get_dataset()` function serves as the main interface called by training scripts ([`src/train.py`](https://github.com/hiyouga/LlamaFactory/blob/main/src/train.py)). It first checks for a pre-tokenized cache; if none exists, it proceeds to load raw datasets via `_load_single_dataset()`.

This component parses the `DataArguments` description, selects the appropriate loading strategy (`hf_hub`, `file`, `cloud_file`), streams or caches the dataset, and optionally limits the number of samples.

### Stage 2: Schema Alignment and Conversion

Once loaded, raw dataset objects pass through `align_dataset()` in [`llamafactory/data/loader.py`](https://github.com/hiyouga/LlamaFactory/blob/main/llamafactory/data/loader.py), which internally calls [`llamafactory/data/converter.py`](https://github.com/hiyouga/LlamaFactory/blob/main/llamafactory/data/converter.py). The converter aligns raw Hugging Face `datasets.Dataset` objects to LlamaFactory’s unified schema using fields like `_prompt`, `_response`, `_system`, and `_images`.

This standardization ensures that downstream processors can rely on a stable field layout regardless of the original dataset format.

### Stage 3: Stage-Specific Processing

The core tokenization logic resides in `llamafactory/data/processor/*.py` (e.g., [`supervised.py`](https://github.com/hiyouga/LlamaFactory/blob/main/supervised.py), [`unsupervised.py`](https://github.com/hiyouga/LlamaFactory/blob/main/unsupervised.py), [`pairwise.py`](https://github.com/hiyouga/LlamaFactory/blob/main/pairwise.py)). Depending on the chosen `stage` parameter, `_get_dataset_processor()` selects the appropriate processor class (`SupervisedDatasetProcessor`, `UnsupervisedDatasetProcessor`, `PairwiseDatasetProcessor`, etc.).

The processor’s `preprocess_dataset()` iterates over each example and calls `_encode_data_example()`, which:

- Invokes `Template._encode()` from [`llamafactory/data/template.py`](https://github.com/hiyouga/LlamaFactory/blob/main/llamafactory/data/template.py) to format conversations according to the selected chat template using methods like `format_user` and `format_assistant`
- Optionally runs `mm_plugin.process_messages()` and `mm_plugin.process_token_ids()` for vision/audio tokens via [`llamafactory/data/mm_plugin.py`](https://github.com/hiyouga/LlamaFactory/blob/main/llamafactory/data/mm_plugin.py)
- Applies `infer_seqlen()` from [`processor_utils.py`](https://github.com/hiyouga/LlamaFactory/blob/main/processor_utils.py) to respect `cutoff_len`
- Generates `input_ids`, `labels`, and multimodal metadata

The `greedy_knapsack()` algorithm in [`processor_utils.py`](https://github.com/hiyouga/LlamaFactory/blob/main/processor_utils.py) automatically packs multiple short examples into a single `cutoff_len`-sized sequence, improving GPU utilization.

### Stage 4: Batch Collation

Finally, [`llamafactory/data/collator.py`](https://github.com/hiyouga/LlamaFactory/blob/main/llamafactory/data/collator.py) takes the list-of-samples produced by the processor and creates batch-wise `torch` tensors. The collator handles padding to the longest sequence in the batch, stacking images/video/audio lists, and preparing the final tensor dictionary ready for the trainer.

## Template-Driven Text Formatting

LlamaFactory decouples conversation formatting from tokenization through the `Template` class in [`llamafactory/data/template.py`](https://github.com/hiyouga/LlamaFactory/blob/main/llamafactory/data/template.py). Templates define how to assemble user and assistant messages using `format_user` and `format_assistant` methods, handling special tokens, system prompts, and stop words.

The design is modular: swapping a template requires only a change in the registration call (see the many `register_template()` calls at the end of [`template.py`](https://github.com/hiyouga/LlamaFactory/blob/main/template.py)). This makes it trivial to support new model families and custom tokenization rules without touching the core loading logic.

## Multimodal Data Handling

For vision and audio training, LlamaFactory uses the `mm_plugin` abstraction in [`llamafactory/data/mm_plugin.py`](https://github.com/hiyouga/LlamaFactory/blob/main/llamafactory/data/mm_plugin.py). Plugins like `qwen2_vl` or `ernie_vl` inject image/video/audio tokens seamlessly into the text stream via `process_messages()` and `process_token_ids()`.

This allows the same four-stage pipeline to handle pure text and vision-language data uniformly, with the processor and collator managing multimodal tensors through the unified schema fields like `_images`.

## Practical Implementation Examples

The following examples demonstrate how to use LlamaFactory's data processing pipeline programmatically.

```python

# -------------------------------------------------

# Example: building a training dataset for SFT

# -------------------------------------------------

from llamafactory.hparams import ModelArguments, DataArguments
from llamafactory.train import get_train_args
from llamafactory.data.loader import get_dataset
from transformers import AutoTokenizer

# 1. Define model & data args (normally parsed from CLI)

model_args = ModelArguments(model_name_or_path="meta-llama/Llama-2-7b-chat-hf")
data_args  = DataArguments(
    dataset="json",
    dataset_dir="./data",
    dataset_name="my_sft_data",
    cutoff_len=2048,
    packing=False,
    train_on_prompt=False,
)

# 2. Load tokenizer and the chat template

tokenizer = AutoTokenizer.from_pretrained(model_args.model_name_or_path, trust_remote_code=True)

# 3. Build the DatasetModule (train + optional eval)

train_module = get_dataset(
    template=None,                # will pick the default template for the tokenizer

    model_args=model_args,
    data_args=data_args,
    training_args=get_train_args(),
    stage="sft",                  # Supervised fine‑tuning

    tokenizer=tokenizer,
)

# 4. Inspect a single processed example

example = next(iter(train_module["train_dataset"]))
print("input_ids length:", len(example["input_ids"]))
print("labels snippet:", example["labels"][:10])
print("has images?", bool(example["images"]))

```

```python

# -------------------------------------------------

# Example: using a custom chat template and multimodal plugin

# -------------------------------------------------

from llamafactory.data.template import register_template, get_template_and_fix_tokenizer
from llamafactory.data.mm_plugin import get_mm_plugin
from llamafactory.data.formatter import StringFormatter

# Register a simple Qwen‑VL style template that knows about <img> tokens

register_template(
    name="qwen_vl",
    format_user=StringFormatter(slots=["<|im_start|>user\n{{content}}<|im_end|>