How LlamaFactory Facilitates Data Processing and Formatting for Training: A Deep Dive into the Pipeline

LlamaFactory provides a modular, four-stage pipeline—Load, Convert, Process, and Collate—that transforms raw datasets into token-level tensors ready for fine-tuning, supporting everything from supervised fine-tuning to multimodal RLHF.

LlamaFactory is an open-source unified framework for fine-tuning large language models. Its data processing and formatting for training is built around a flexible, plug-and-play architecture that handles JSON, CSV, Hugging Face Hub datasets, and local files, converting them into the exact token-level tensors required by a 🤗 Transformers trainer.

The Four-Stage Data Pipeline Architecture

The pipeline consists of four tightly integrated components that work sequentially to prepare training data.

Stage 1: Loading Raw Datasets

The entry point for LlamaFactory data processing is llamafactory/data/loader.py. The get_dataset() function serves as the main interface called by training scripts (src/train.py). It first checks for a pre-tokenized cache; if none exists, it proceeds to load raw datasets via _load_single_dataset().

This component parses the DataArguments description, selects the appropriate loading strategy (hf_hub, file, cloud_file), streams or caches the dataset, and optionally limits the number of samples.

Stage 2: Schema Alignment and Conversion

Once loaded, raw dataset objects pass through align_dataset() in llamafactory/data/loader.py, which internally calls llamafactory/data/converter.py. The converter aligns raw Hugging Face datasets.Dataset objects to LlamaFactory’s unified schema using fields like _prompt, _response, _system, and _images.

This standardization ensures that downstream processors can rely on a stable field layout regardless of the original dataset format.

Stage 3: Stage-Specific Processing

The core tokenization logic resides in llamafactory/data/processor/*.py (e.g., supervised.py, unsupervised.py, pairwise.py). Depending on the chosen stage parameter, _get_dataset_processor() selects the appropriate processor class (SupervisedDatasetProcessor, UnsupervisedDatasetProcessor, PairwiseDatasetProcessor, etc.).

The processor’s preprocess_dataset() iterates over each example and calls _encode_data_example(), which:

• Invokes Template._encode() from llamafactory/data/template.py to format conversations according to the selected chat template using methods like format_user and format_assistant
• Optionally runs mm_plugin.process_messages() and mm_plugin.process_token_ids() for vision/audio tokens via llamafactory/data/mm_plugin.py
• Applies infer_seqlen() from processor_utils.py to respect cutoff_len
• Generates input_ids, labels, and multimodal metadata

The greedy_knapsack() algorithm in processor_utils.py automatically packs multiple short examples into a single cutoff_len-sized sequence, improving GPU utilization.

Stage 4: Batch Collation

Finally, llamafactory/data/collator.py takes the list-of-samples produced by the processor and creates batch-wise torch tensors. The collator handles padding to the longest sequence in the batch, stacking images/video/audio lists, and preparing the final tensor dictionary ready for the trainer.

Template-Driven Text Formatting

LlamaFactory decouples conversation formatting from tokenization through the Template class in llamafactory/data/template.py. Templates define how to assemble user and assistant messages using format_user and format_assistant methods, handling special tokens, system prompts, and stop words.

The design is modular: swapping a template requires only a change in the registration call (see the many register_template() calls at the end of template.py). This makes it trivial to support new model families and custom tokenization rules without touching the core loading logic.

Multimodal Data Handling

For vision and audio training, LlamaFactory uses the mm_plugin abstraction in llamafactory/data/mm_plugin.py. Plugins like qwen2_vl or ernie_vl inject image/video/audio tokens seamlessly into the text stream via process_messages() and process_token_ids().

This allows the same four-stage pipeline to handle pure text and vision-language data uniformly, with the processor and collator managing multimodal tensors through the unified schema fields like _images.

Practical Implementation Examples

The following examples demonstrate how to use LlamaFactory's data processing pipeline programmatically.

# -------------------------------------------------

# Example: building a training dataset for SFT

# -------------------------------------------------

from llamafactory.hparams import ModelArguments, DataArguments
from llamafactory.train import get_train_args
from llamafactory.data.loader import get_dataset
from transformers import AutoTokenizer

# 1. Define model & data args (normally parsed from CLI)

model_args = ModelArguments(model_name_or_path="meta-llama/Llama-2-7b-chat-hf")
data_args  = DataArguments(
    dataset="json",
    dataset_dir="./data",
    dataset_name="my_sft_data",
    cutoff_len=2048,
    packing=False,
    train_on_prompt=False,
)

# 2. Load tokenizer and the chat template

tokenizer = AutoTokenizer.from_pretrained(model_args.model_name_or_path, trust_remote_code=True)

# 3. Build the DatasetModule (train + optional eval)

train_module = get_dataset(
    template=None,                # will pick the default template for the tokenizer

    model_args=model_args,
    data_args=data_args,
    training_args=get_train_args(),
    stage="sft",                  # Supervised fine‑tuning

    tokenizer=tokenizer,
)

# 4. Inspect a single processed example

example = next(iter(train_module["train_dataset"]))
print("input_ids length:", len(example["input_ids"]))
print("labels snippet:", example["labels"][:10])
print("has images?", bool(example["images"]))

# -------------------------------------------------

# Example: using a custom chat template and multimodal plugin

# -------------------------------------------------

from llamafactory.data.template import register_template, get_template_and_fix_tokenizer
from llamafactory.data.mm_plugin import get_mm_plugin
from llamafactory.data.formatter import StringFormatter

# Register a simple Qwen‑VL style template that knows about <img> tokens

register_template(
    name="qwen_vl",
    format_user=StringFormatter(slots=["<|im_start|>user\n{{content}}<|im_end|>

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}