How LlamaFactory Integrates with Experiment Tracking Tools: A Complete Technical Guide

LlamaFactory routes training metrics and hyperparameters to external experiment tracking platforms through a unified report_to option that connects Web UI selections, configuration validation, and runtime callbacks.

LlamaFactory provides a seamless bridge between fine-tuning workflows and experiment tracking ecosystems. The open-source repository implements a three-layer architecture—spanning user interface components, argument parsers, and training callbacks—to ensure that how LlamaFactory integrates with experiment tracking tools remains consistent across Weights & Biases, TensorBoard, Trackio, and other supported backends.

The Three-Layer Integration Architecture

LlamaFactory decouples the user-facing selection from runtime execution through discrete validation and callback layers. This design makes the system extensible: adding a new tracker requires updating the dropdown, extending the parser, and implementing a hook in the callback class.

UI Selection Layer

The Web UI exposes experiment tracking configuration through the Training panel in src/llamafactory/webui/components/train.py. Users select their preferred backend from a dropdown populated with the following choices:

• none
• wandb
• mlflow
• neptune
• tensorboard
• trackio
• all

This component stores the selection in the report_to field, which propagates through the training configuration pipeline.

Configuration Validation

Before training begins, the report_to argument undergoes normalization and validation in src/llamafactory/hparams/parser.py. The parser converts the input to a list format and enforces stage-specific constraints. For example, PPO training only accepts wandb, tensorboard, trackio, or none; other stages like SFT or DPO additionally permit mlflow and neptune. This validation step prevents runtime errors by catching incompatible tracker configurations early in the execution flow.

Runtime Callback Execution

The ReporterCallback class in src/llamafactory/train/callbacks.py handles the actual integration. As a subclass of TrainerCallback, it implements the on_train_begin method to initialize external loggers immediately before training starts. When report_to contains "wandb", the callback imports the wandb SDK, sets the project name from the WANDB_PROJECT environment variable (defaulting to "llamafactory"), and pushes the full configuration—including model_args, data_args, finetuning_args, and generating_args—via wandb.config.update().

Similarly, when "trackio" is specified, the callback imports the trackio SDK and updates the configuration object with the same argument dictionaries. If finetuning_args.use_swanlab evaluates to true, the logger initializes SwanLab using an identical pattern.

Platform-Specific Implementations

Each supported tracking platform leverages distinct initialization patterns while maintaining consistent configuration export.

Weights & Biases (W&B)

For W&B integration, LlamaFactory relies on environment variables and automatic SDK initialization. The ReporterCallback checks for the "wandb" string in report_to, then invokes wandb.init() implicitly through the SDK. All hyperparameters and model configurations serialize automatically through the callback's dictionary export, making them browsable in the W&B dashboard without additional instrumentation code.

TensorBoard Integration

TensorBoard support requires no custom callback code. LlamaFactory inherits from Hugging Face's Seq2SeqTrainingArguments (defined in src/llamafactory/hparams/training_args.py), which exposes native report_to=["tensorboard"] functionality. When this flag is active, the underlying Transformers Trainer writes scalar logs to the specified logging_dir, typically ./logs/, enabling visualization via tensorboard --logdir.

Trackio and SwanLab

For Trackio, the Web UI renders additional fields under Trackio Settings: Project Name and Space ID. These values propagate to trackio.config during callback initialization, routing metrics to private Hugging Face Spaces or public Trackio dashboards. SwanLab follows an analogous path when users enable use_swanlab in the fine-tuning arguments, piggybacking on the same configuration export mechanism.

MLflow and Neptune

MLflow and Neptune currently appear in the UI dropdown but remain restricted to specific training stages. As of the current implementation, these backends are available in the Web UI but require manual validation extension in parser.py for full CLI support across all training paradigms.

Configuring Experiment Tracking in Practice

Enable Weights & Biases via CLI

Set the project environment variable and pass the report_to flag:

export WANDB_PROJECT=my-llamafactory-run
llamafactory-cli train \
  --model_name_or_path meta-llama/Meta-Llama-3-8B \
  --train_file data/train.json \
  --report_to wandb \
  --output_dir ./output

The ReporterCallback automatically triggers wandb.init() and uploads the complete configuration object.

Configure TensorBoard Logging

TensorBoard requires only the report target and logging directory:

llamafactory-cli train \
  --report_to tensorboard \
  --logging_dir ./tb_logs \
  --output_dir ./output

Visualize results after training:

tensorboard --logdir ./tb_logs

Set Up Trackio from the Web UI

In the interface, expand Trackio Settings and configure:

# Project Name: "my-hf-space"

# Trackio Space ID: "username/trackio-space"

# Enable external logger: select "trackio"

When training starts, the callback executes:

trackio.config.update({
    "model_args": model_args.to_dict(),
    "data_args": data_args.to_dict(),
    "finetuning_args": finetuning_args.to_dict(),
    "generating_args": generating_args.to_dict(),
})

All metrics stream to the specified Trackio dashboard automatically.

Summary

• Three-layer architecture: LlamaFactory separates UI selection (train.py), configuration validation (parser.py), and runtime execution (callbacks.py) to ensure robust experiment tracking integration.
• ReporterCallback: The core integration point lives in src/llamafactory/train/callbacks.py, handling initialization for W&B, Trackio, and SwanLab via on_train_begin.
• Native TensorBoard support: Inherited from Transformers' Seq2SeqTrainingArguments, requiring no custom callback code when report_to includes "tensorboard".
• Extensible design: Adding new trackers involves updating the dropdown in the Web UI, extending validation rules in the parser, and adding initialization blocks to ReporterCallback.

Frequently Asked Questions

Which experiment tracking tools does LlamaFactory support natively?

LlamaFactory supports Weights & Biases, TensorBoard, Trackio, SwanLab, MLflow, and Neptune through the unified report_to interface. However, PPO training restricts valid options to wandb, tensorboard, trackio, and none, while supervised fine-tuning (SFT) and direct preference optimization (DPO) stages accept the full suite including mlflow and neptune.

How do I configure Weights & Biases logging from the command line?

Export the WANDB_PROJECT environment variable to define your project name, then append --report_to wandb to your llamafactory-cli train command. The ReporterCallback in src/llamafactory/train/callbacks.py handles the wandb.init() call and automatically pushes model_args, data_args, and other configuration dictionaries to the dashboard.

Can I use MLflow with LlamaFactory's CLI interface?

Currently, MLflow appears in the Web UI dropdown but requires the training stage to support it beyond the UI selection. To enable full CLI support, you must extend the _verify_trackio_args function in src/llamafactory/hparams/parser.py to accept "mlflow" for your specific training stage, then add the corresponding initialization block in ReporterCallback.on_train_begin following the existing W&B pattern.

Where does LlamaFactory initialize the tracking callbacks during training?

The initialization occurs in src/llamafactory/train/callbacks.py within the ReporterCallback class. Specifically, the on_train_begin method checks the report_to list for supported strings like "wandb" or "trackio", imports the respective SDK, and executes configuration updates before the first training step begins.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}