# How to Configure TRL Training Jobs on Hugging Face Jobs: A Complete Guide to SFT, DPO, and GRPO

> Configure TRL training jobs on Hugging Face Jobs using hf_jobs MCP helper. Run SFT, DPO, and GRPO fine-tuning with `a10g-large` GPUs and Hub authentication for efficient distributed training.

- Repository: [Hugging Face/skills](https://github.com/huggingface/skills)
- Tags: how-to-guide
- Published: 2026-03-08

---

**Use the `hf_jobs` MCP helper to submit self-contained Python scripts that instantiate TRL trainers (`SFTTrainer`, `DPOTrainer`, `GRPOTrainer`) with algorithm-specific Config objects, GPU flavors like `a10g-large`, and Hub authentication via `HF_TOKEN` to run distributed fine-tuning on managed infrastructure.**

The `huggingface/skills` repository provides drop-in training scripts under `skills/hugging-face-model-trainer/scripts/` that demonstrate how to configure TRL training jobs on Hugging Face Jobs for three distinct fine-tuning paradigms. These scripts integrate dependency management, dataset loading, PEFT optimization, and real-time monitoring into a single file that executes on GPU instances without local setup.

## TRL Training Paradigms Overview

The TRL (Training Reinforcement Learning) library implements three primary algorithms for aligning language models. Each paradigm uses a dedicated Trainer and Config class, targeting different data formats and optimization objectives:

- **Supervised Fine-Tuning (SFT)** — Uses `SFTTrainer` and `SFTConfig` for standard instruction following on prompt-response pairs. Best for adapting base models to specific domains or conversation styles.
- **Direct Preference Optimization (DPO)** — Uses `DPOTrainer` and `DPOConfig` to learn from preference pairs (chosen vs. rejected responses). Eliminates the need for a separate reward model by optimizing directly against human preferences.
- **Group-Relative Policy Optimization (GRPO)** — Uses `GRPOTrainer` and `GRPOConfig` for online reinforcement learning with group-based reward signals. Designed for tasks like mathematical reasoning where verification functions provide sparse rewards.

## Job Submission Architecture

All training scripts in `skills/hugging-face-model-trainer/scripts/` share a common execution pattern designed for the `hf_jobs` containerized runtime.

### Script Structure Requirements

Each script begins with a **dependency declaration block** that pins required libraries:

```python

# /// script

# dependencies = [

#   "trl>=0.12.0",

#   "transformers>=4.45.0",

#   "accelerate>=0.34.0",

#   "trackio>=0.3.0",

#   "peft>=0.13.0",

# ]

# ///

```

The `uv` container automatically installs these packages before execution, ensuring reproducible environments across runs.

### GPU Flavors and Timeouts

Submit jobs via the `hf_jobs` helper with hardware specifications tailored to model size and algorithmic complexity:

| Algorithm | Flavor | Timeout | VRAM |
|-----------|--------|---------|------|
| SFT | `a10g-large` | `3h` | 24 GiB |
| DPO | `a10g-large` | `3h` | 24 GiB |
| GRPO | `a10g-large` | `4h` | 24 GiB |

GRPO requires longer timeouts because online RL involves sampling multiple completions per prompt and computing group-relative advantages, increasing per-step latency.

## SFT Configuration with LoRA

The [`train_sft_example.py`](https://github.com/huggingface/skills/blob/main/train_sft_example.py) script demonstrates memory-efficient supervised fine-tuning using **Parameter-Efficient Fine-Tuning (PEFT)** with LoRA adapters.

Key implementation details from [`skills/hugging-face-model-trainer/scripts/train_sft_example.py`](https://github.com/huggingface/skills/blob/main/skills/hugging-face-model-trainer/scripts/train_sft_example.py):

```python
from datasets import load_dataset
from peft import LoraConfig
from trl import SFTTrainer, SFTConfig

# Load and split dataset

dataset = load_dataset("trl-lib/Capybara", split="train")
train, eval = dataset.train_test_split(test_size=0.1, seed=42).values()

# Configure LoRA for memory efficiency

peft_cfg = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules=["q_proj", "v_proj"]
)

# SFT-specific configuration

cfg = SFTConfig(
    output_dir="qwen-capybara-sft",
    push_to_hub=True,
    hub_model_id="username/qwen-capybara-sft",
    hub_strategy="every_save",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-5,
    lr_scheduler_type="cosine",
    warmup_ratio=0.1,
    logging_steps=10,
    save_strategy="steps",
    save_steps=100,
    eval_strategy="steps",
    eval_steps=100,
    report_to="trackio",
    project="my-sft-project",
    run_name="sft-run-01",
)

trainer = SFTTrainer(
    model="Qwen/Qwen2.5-0.5B",
    train_dataset=train,
    eval_dataset=eval,
    args=cfg,
    peft_config=peft_cfg,
)

```

Submit this script using:

```python
hf_jobs("uv", {
    "script": """<paste full script here>""",
    "flavor": "a10g-large",
    "timeout": "3h",
    "secrets": {"HF_TOKEN": "$HF_TOKEN"},
})

```

## DPO Configuration for Preference Learning

The [`train_dpo_example.py`](https://github.com/huggingface/skills/blob/main/train_dpo_example.py) script configures Direct Preference Optimization using paired preference data. Unlike SFT, DPO requires an **instruct-tuned base model** and does not use LoRA in the reference implementation.

From [`skills/hugging-face-model-trainer/scripts/train_dpo_example.py`](https://github.com/huggingface/skills/blob/main/skills/hugging-face-model-trainer/scripts/train_dpo_example.py):

```python
from datasets import load_dataset
from trl import DPOTrainer, DPOConfig

# Load preference pairs (chosen vs rejected)

pref = load_dataset("trl-lib/ultrafeedback_binarized", split="train")
train, eval = pref.train_test_split(test_size=0.1, seed=42).values()

cfg = DPOConfig(
    output_dir="qwen-dpo-aligned",
    push_to_hub=True,
    hub_model_id="username/qwen-dpo-aligned",
    hub_strategy="every_save",
    beta=0.1,  # KL-divergence penalty strength

    num_train_epochs=1,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=5e-7,
    lr_scheduler_type="cosine",
    warmup_ratio=0.1,
    logging_steps=10,
    save_strategy="steps",
    save_steps=100,
    eval_strategy="steps",
    eval_steps=100,
    report_to="trackio",
    project="my-dpo-project",
    run_name="dpo-run-01",
)

trainer = DPOTrainer(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    train_dataset=train,
    eval_dataset=eval,
    args=cfg,
)

```

The `beta` parameter controls the trade-off between preference optimization and retention of the base model's capabilities. Lower values prioritize alignment with preferences; higher values preserve the base distribution.

## GRPO Configuration for Online RL

The [`train_grpo_example.py`](https://github.com/huggingface/skills/blob/main/train_grpo_example.py) script implements Group-Relative Policy Optimization for scenarios requiring **online reward computation**, such as mathematical verification or code execution.

From [`skills/hugging-face-model-trainer/scripts/train_grpo_example.py`](https://github.com/huggingface/skills/blob/main/skills/hugging-face-model-trainer/scripts/train_grpo_example.py):

```python
from datasets import load_dataset
from trl import GRPOTrainer, GRPOConfig

# Load prompt-only dataset (no gold responses)

prompts = load_dataset("trl-lib/math_shepherd", split="train")

cfg = GRPOConfig(
    output_dir="qwen-grpo-math",
    push_to_hub=True,
    hub_model_id="username/qwen-grpo-math",
    hub_strategy="every_save",
    num_train_epochs=1,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=1e-6,
    lr_scheduler_type="cosine",
    warmup_ratio=0.1,
    logging_steps=10,
    save_strategy="steps",
    save_steps=100,
    report_to="trackio",
    project="my-grpo-project",
    run_name="grpo-run-01",
)

trainer = GRPOTrainer(
    model="Qwen/Qwen2.5-0.5B-Instruct",
    train_dataset=prompts,
    args=cfg,
)

```

Submit with an extended timeout to accommodate online sampling:

```python
hf_jobs("uv", {
    "script": """<paste full script here>""",
    "flavor": "a10g-large",
    "timeout": "4h",  # Extended for online RL sampling

    "secrets": {"HF_TOKEN": "$HF_TOKEN"},
})

```

GRPO differs from DPO in that it requires no pre-computed preference pairs; instead, it samples multiple completions per prompt during training and computes relative advantages within each group using external reward functions.

## Monitoring and Hub Integration

All three configurations leverage **Trackio** for observability and automatic Hub synchronization for artifact management.

### Real-Time Metrics Streaming

Setting `report_to="trackio"` in any Config object streams loss curves, learning rates, and GPU utilization metrics to a dedicated dashboard. View results at `https://huggingface.co/spaces/{username}/trackio` or embed the dashboard in your Spaces.

### Checkpoint Management

The `hub_strategy="every_save"` parameter ensures that every checkpoint pushed to `output_dir` is automatically uploaded to the Hugging Face Hub under `hub_model_id`. This enables:
- Immediate model inference via the Inference API
- Collaborative fine-tuning workflows where team members resume from intermediate checkpoints
- Automatic model cards generation with training hyperparameters

## Pre-Flight Utilities

Before submitting expensive GPU jobs, use the diagnostic scripts in the same directory to validate configurations.

### Cost Estimation

[`skills/hugging-face-model-trainer/scripts/estimate_cost.py`](https://github.com/huggingface/skills/blob/main/skills/hugging-face-model-trainer/scripts/estimate_cost.py) calculates expected runtime and GPU costs based on dataset size, model parameters, and training epochs. Run this locally to determine appropriate `timeout` values and select cost-effective flavors.

### Dataset Validation

[`skills/hugging-face-model-trainer/scripts/dataset_inspector.py`](https://github.com/huggingface/skills/blob/main/skills/hugging-face-model-trainer/scripts/dataset_inspector.py) verifies that your dataset contains the required fields for your chosen algorithm (e.g., `chosen`/`rejected` columns for DPO, `prompt` only for GRPO). This prevents job failures due to schema mismatches after GPU allocation.

## Summary

- **Configure TRL training jobs on Hugging Face Jobs** by wrapping algorithm-specific scripts in the `hf_jobs("uv", {...})` helper with GPU flavors and authentication secrets.
- **SFT** requires `SFTConfig` with LoRA adapters for memory-efficient domain adaptation on prompt-response data.
- **DPO** uses `DPOConfig` with preference pairs and an instruct-tuned base model, optimizing directly against human judgments without reward models.
- **GRPO** employs `GRPOConfig` for online RL on prompt-only datasets, requiring extended timeouts to accommodate group sampling and reward computation.
- All scripts support **Trackio monitoring** via `report_to="trackio"` and **automatic Hub pushes** via `push_to_hub=True` and `hub_strategy="every_save"`.
- Validate datasets and estimate costs using [`dataset_inspector.py`](https://github.com/huggingface/skills/blob/main/dataset_inspector.py) and [`estimate_cost.py`](https://github.com/huggingface/skills/blob/main/estimate_cost.py) before committing GPU resources.

## Frequently Asked Questions

### What is the difference between DPO and GRPO in TRL?

**DPO (Direct Preference Optimization)** learns from static preference pairs labeled as chosen or rejected, optimizing the policy to increase the likelihood of preferred outputs. **GRPO (Group-Relative Policy Optimization)** is an online RL method that samples multiple completions per prompt during training and uses group-relative advantages computed from live reward functions, making it suitable for tasks with verifiable outcomes like math or coding where preferences cannot be pre-labeled.

### Why does GRPO require a longer timeout than SFT or DPO?

GRPO performs **online sampling** during training: for each prompt, the model generates multiple completions (a "group"), computes rewards for each completion using external verifiers, then calculates relative advantages within that group. This sampling loop adds significant overhead per training step compared to the static forward passes of SFT or DPO, necessitating the 4-hour timeout versus 3 hours for the other methods.

### Do I need to install TRL locally before submitting a job?

No. The **dependency header block** (`# /// script`) in each training script declares required packages including `trl`, `transformers`, and `accelerate`. The `uv` container automatically installs these dependencies in the remote GPU environment. You only need local TRL installation if you wish to run [`estimate_cost.py`](https://github.com/huggingface/skills/blob/main/estimate_cost.py) or [`dataset_inspector.py`](https://github.com/huggingface/skills/blob/main/dataset_inspector.py) for pre-flight validation.

### How do I resume training from a checkpoint on Hugging Face Jobs?

Set `hub_strategy="every_save"` and `push_to_hub=True` in your Config object to automatically upload checkpoints to the Hub. To resume, modify your script to load the model from the Hub checkpoint ID (e.g., `model="username/qwen-sft/checkpoint-500"`) instead of the base model identifier, then resubmit via `hf_jobs`. The TRL trainers automatically detect incomplete training states and resume from the latest optimizer and scheduler checkpoints.