How to Configure TRL Training Jobs on Hugging Face Jobs: A Complete Guide to SFT, DPO, and GRPO

Use the hf_jobs MCP helper to submit self-contained Python scripts that instantiate TRL trainers (SFTTrainer, DPOTrainer, GRPOTrainer) with algorithm-specific Config objects, GPU flavors like a10g-large, and Hub authentication via HF_TOKEN to run distributed fine-tuning on managed infrastructure.

The huggingface/skills repository provides drop-in training scripts under skills/hugging-face-model-trainer/scripts/ that demonstrate how to configure TRL training jobs on Hugging Face Jobs for three distinct fine-tuning paradigms. These scripts integrate dependency management, dataset loading, PEFT optimization, and real-time monitoring into a single file that executes on GPU instances without local setup.

TRL Training Paradigms Overview

The TRL (Training Reinforcement Learning) library implements three primary algorithms for aligning language models. Each paradigm uses a dedicated Trainer and Config class, targeting different data formats and optimization objectives:

• Supervised Fine-Tuning (SFT) — Uses SFTTrainer and SFTConfig for standard instruction following on prompt-response pairs. Best for adapting base models to specific domains or conversation styles.
• Direct Preference Optimization (DPO) — Uses DPOTrainer and DPOConfig to learn from preference pairs (chosen vs. rejected responses). Eliminates the need for a separate reward model by optimizing directly against human preferences.
• Group-Relative Policy Optimization (GRPO) — Uses GRPOTrainer and GRPOConfig for online reinforcement learning with group-based reward signals. Designed for tasks like mathematical reasoning where verification functions provide sparse rewards.

Job Submission Architecture

All training scripts in skills/hugging-face-model-trainer/scripts/ share a common execution pattern designed for the hf_jobs containerized runtime.

Script Structure Requirements

Each script begins with a dependency declaration block that pins required libraries:

# /// script

# dependencies = [

#   "trl>=0.12.0",

#   "transformers>=4.45.0",

#   "accelerate>=0.34.0",

#   "trackio>=0.3.0",

#   "peft>=0.13.0",

# ]

# ///

The uv container automatically installs these packages before execution, ensuring reproducible environments across runs.

GPU Flavors and Timeouts

Submit jobs via the hf_jobs helper with hardware specifications tailored to model size and algorithmic complexity:

Algorithm	Flavor	Timeout	VRAM
SFT	a10g-large	3h	24 GiB
DPO	a10g-large	3h	24 GiB
GRPO	a10g-large	4h	24 GiB

GRPO requires longer timeouts because online RL involves sampling multiple completions per prompt and computing group-relative advantages, increasing per-step latency.

SFT Configuration with LoRA

The train_sft_example.py script demonstrates memory-efficient supervised fine-tuning using Parameter-Efficient Fine-Tuning (PEFT) with LoRA adapters.

Key implementation details from skills/hugging-face-model-trainer/scripts/train_sft_example.py:

from datasets import load_dataset
from peft import LoraConfig
from trl import SFTTrainer, SFTConfig

# Load and split dataset

dataset = load_dataset("trl-lib/Capybara", split="train")
train, eval = dataset.train_test_split(test_size=0.1, seed=42).values()

# Configure LoRA for memory efficiency

peft_cfg = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules=["q_proj", "v_proj"]
)

# SFT-specific configuration

cfg = SFTConfig(
    output_dir="qwen-capybara-sft",
    push_to_hub=True,
    hub_model_id="username/qwen-capybara-sft",
    hub_strategy="every_save",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-5,
    lr_scheduler_type="cosine",
    warmup_ratio=0.1,
    logging_steps=10,
    save_strategy="steps",
    save_steps=100,
    eval_strategy="steps",
    eval_steps=100,
    report_to="trackio",
    project="my-sft-project",
    run_name="sft-run-01",
)

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-0.5B",
    train_dataset=train,
    eval_dataset=eval,
    args=cfg,
    peft_config=peft_cfg,
)

Submit this script using:

hf_jobs("uv", {
    "script": """<paste full script here>""",
    "flavor": "a10g-large",
    "timeout": "3h",
    "secrets": {"HF_TOKEN": "$HF_TOKEN"},
})

DPO Configuration for Preference Learning

The train_dpo_example.py script configures Direct Preference Optimization using paired preference data. Unlike SFT, DPO requires an instruct-tuned base model and does not use LoRA in the reference implementation.

From skills/hugging-face-model-trainer/scripts/train_dpo_example.py:

from datasets import load_dataset
from trl import DPOTrainer, DPOConfig

# Load preference pairs (chosen vs rejected)

pref = load_dataset("trl-lib/ultrafeedback_binarized", split="train")
train, eval = pref.train_test_split(test_size=0.1, seed=42).values()

cfg = DPOConfig(
    output_dir="qwen-dpo-aligned",
    push_to_hub=True,
    hub_model_id="username/qwen-dpo-aligned",
    hub_strategy="every_save",
    beta=0.1,  # KL-divergence penalty strength

    num_train_epochs=1,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=5e-7,
    lr_scheduler_type="cosine",
    warmup_ratio=0.1,
    logging_steps=10,
    save_strategy="steps",
    save_steps=100,
    eval_strategy="steps",
    eval_steps=100,
    report_to="trackio",
    project="my-dpo-project",
    run_name="dpo-run-01",
)

trainer = DPOTrainer(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    train_dataset=train,
    eval_dataset=eval,
    args=cfg,
)

The beta parameter controls the trade-off between preference optimization and retention of the base model's capabilities. Lower values prioritize alignment with preferences; higher values preserve the base distribution.

GRPO Configuration for Online RL

The train_grpo_example.py script implements Group-Relative Policy Optimization for scenarios requiring online reward computation, such as mathematical verification or code execution.

From skills/hugging-face-model-trainer/scripts/train_grpo_example.py:

from datasets import load_dataset
from trl import GRPOTrainer, GRPOConfig

# Load prompt-only dataset (no gold responses)

prompts = load_dataset("trl-lib/math_shepherd", split="train")

cfg = GRPOConfig(
    output_dir="qwen-grpo-math",
    push_to_hub=True,
    hub_model_id="username/qwen-grpo-math",
    hub_strategy="every_save",
    num_train_epochs=1,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=1e-6,
    lr_scheduler_type="cosine",
    warmup_ratio=0.1,
    logging_steps=10,
    save_strategy="steps",
    save_steps=100,
    report_to="trackio",
    project="my-grpo-project",
    run_name="grpo-run-01",
)

trainer = GRPOTrainer(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    train_dataset=prompts,
    args=cfg,
)

Submit with an extended timeout to accommodate online sampling:

hf_jobs("uv", {
    "script": """<paste full script here>""",
    "flavor": "a10g-large",
    "timeout": "4h",  # Extended for online RL sampling

    "secrets": {"HF_TOKEN": "$HF_TOKEN"},
})

GRPO differs from DPO in that it requires no pre-computed preference pairs; instead, it samples multiple completions per prompt during training and computes relative advantages within each group using external reward functions.

Monitoring and Hub Integration

All three configurations leverage Trackio for observability and automatic Hub synchronization for artifact management.

Real-Time Metrics Streaming

Setting report_to="trackio" in any Config object streams loss curves, learning rates, and GPU utilization metrics to a dedicated dashboard. View results at https://huggingface.co/spaces/{username}/trackio or embed the dashboard in your Spaces.

Checkpoint Management

The hub_strategy="every_save" parameter ensures that every checkpoint pushed to output_dir is automatically uploaded to the Hugging Face Hub under hub_model_id. This enables:

• Immediate model inference via the Inference API
• Collaborative fine-tuning workflows where team members resume from intermediate checkpoints
• Automatic model cards generation with training hyperparameters

Pre-Flight Utilities

Before submitting expensive GPU jobs, use the diagnostic scripts in the same directory to validate configurations.

Cost Estimation

skills/hugging-face-model-trainer/scripts/estimate_cost.py calculates expected runtime and GPU costs based on dataset size, model parameters, and training epochs. Run this locally to determine appropriate timeout values and select cost-effective flavors.

Dataset Validation

skills/hugging-face-model-trainer/scripts/dataset_inspector.py verifies that your dataset contains the required fields for your chosen algorithm (e.g., chosen/rejected columns for DPO, prompt only for GRPO). This prevents job failures due to schema mismatches after GPU allocation.

Summary

• Configure TRL training jobs on Hugging Face Jobs by wrapping algorithm-specific scripts in the hf_jobs("uv", {...}) helper with GPU flavors and authentication secrets.
• SFT requires SFTConfig with LoRA adapters for memory-efficient domain adaptation on prompt-response data.
• DPO uses DPOConfig with preference pairs and an instruct-tuned base model, optimizing directly against human judgments without reward models.
• GRPO employs GRPOConfig for online RL on prompt-only datasets, requiring extended timeouts to accommodate group sampling and reward computation.
• All scripts support Trackio monitoring via report_to="trackio" and automatic Hub pushes via push_to_hub=True and hub_strategy="every_save".
• Validate datasets and estimate costs using dataset_inspector.py and estimate_cost.py before committing GPU resources.

Frequently Asked Questions

What is the difference between DPO and GRPO in TRL?

DPO (Direct Preference Optimization) learns from static preference pairs labeled as chosen or rejected, optimizing the policy to increase the likelihood of preferred outputs. GRPO (Group-Relative Policy Optimization) is an online RL method that samples multiple completions per prompt during training and uses group-relative advantages computed from live reward functions, making it suitable for tasks with verifiable outcomes like math or coding where preferences cannot be pre-labeled.

Why does GRPO require a longer timeout than SFT or DPO?

GRPO performs online sampling during training: for each prompt, the model generates multiple completions (a "group"), computes rewards for each completion using external verifiers, then calculates relative advantages within that group. This sampling loop adds significant overhead per training step compared to the static forward passes of SFT or DPO, necessitating the 4-hour timeout versus 3 hours for the other methods.

Do I need to install TRL locally before submitting a job?

No. The dependency header block (# /// script) in each training script declares required packages including trl, transformers, and accelerate. The uv container automatically installs these dependencies in the remote GPU environment. You only need local TRL installation if you wish to run estimate_cost.py or dataset_inspector.py for pre-flight validation.

How do I resume training from a checkpoint on Hugging Face Jobs?

Set hub_strategy="every_save" and push_to_hub=True in your Config object to automatically upload checkpoints to the Hub. To resume, modify your script to load the model from the Hub checkpoint ID (e.g., model="username/qwen-sft/checkpoint-500") instead of the base model identifier, then resubmit via hf_jobs. The TRL trainers automatically detect incomplete training states and resume from the latest optimizer and scheduler checkpoints.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}