RF-DETR Dataset Formats for Training: COCO and YOLO Support Explained

RF-DETR automatically detects and trains on COCO and YOLO dataset formats by inspecting directory structure for _annotations.coco.json or data.yaml files.

When training computer vision models with the Roboflow RF-DETR repository, you can feed data in two industry-standard formats without manual configuration. The library determines which parser to invoke based on the presence of specific files in your dataset directory, supporting object detection, segmentation, and keypoint-preview tasks.

Supported Dataset Formats

RF-DETR accepts training data in COCO JSON or YOLO format. Both layouts support train, validation, and optional test splits through standardized folder structures.

COCO Format

The COCO loader expects a JSON annotation file following the standard COCO schema. When you point model.train(dataset_dir=...) at a directory, RF-DETR looks for train/_annotations.coco.json at the root of your dataset path. You may optionally include valid/_annotations.coco.json and test/_annotations.coco.json for validation and test splits.

The JSON must contain images, annotations, and categories fields at minimum. For segmentation models, each annotation record must also include a segmentation field containing polygon or RLE masks. The parsing logic resides in src/rfdetr/datasets/coco.py, which handles conversion to the internal training representation.

YOLO Format

YOLO datasets rely on a YAML configuration file and paired image-label directories. RF-DETR detects this format by searching for data.yaml in the dataset root alongside train/images/ and train/labels/ subdirectories. The YAML file defines class names and optionally declares splits.

For keypoint-preview training on YOLO pose datasets, the data.yaml must include a kpt_shape field specifying the number of keypoints and dimensions. The YOLO parser in src/rfdetr/datasets/yolo.py handles the data.yaml parsing and label file ingestion.

Automatic Format Detection

The detection logic requires no explicit format flags. As implemented in roboflow/rf-detr, the train() method inspects the provided dataset_dir for the characteristic files of each format:

• COCO trigger: Presence of train/_annotations.coco.json
• YOLO trigger: Presence of data.yaml with adjacent train/images/ folders

This automatic routing occurs before data loading begins, ensuring the correct pipeline—COCO via src/rfdetr/datasets/coco.py or YOLO via src/rfdetr/datasets/yolo.py—initializes for your training run. The validation logic in src/rfdetr/models/_defaults.py enforces these path requirements during model construction.

Training Configuration by Task Type

RF-DETR supports three model variants, each accepting these dataset formats with task-specific requirements.

Object Detection

Standard detection training accepts either COCO or YOLO formats. Bounding box annotations in COCO use the [x, y, width, height] format, while YOLO uses normalized [class_id, center_x, center_y, width, height] text files.

from rfdetr import RFDETRSmall

model = RFDETRSmall()
model.train(
    dataset_dir="path/to/my_coco_dataset",  # Requires train/_annotations.coco.json

    epochs=50,
)

Instance Segmentation

Segmentation models require mask annotations. In COCO format, provide the segmentation field in your JSON. YOLO segmentation datasets use polygon coordinates in the label files.

from rfdetr import RFDETRSegSmall

model = RFDETRSegSmall()
model.train(
    dataset_dir="path/to/my_yolo_dataset",  # Requires data.yaml + train/images/, train/labels/

    epochs=30,
)

Keypoint Detection

The keypoint-preview model accepts COCO keypoint JSON or YOLO pose datasets. YOLO pose datasets must declare kpt_shape in data.yaml to define the keypoint topology.

from rfdetr import RFDETRKeypointPreview

model = RFDETRKeypointPreview()
model.train(
    dataset_dir="path/to/my_yolo_pose_dataset",  # data.yaml must include kpt_shape

    epochs=20,
)

Key Implementation Files

Understanding the source structure helps debug format issues or extend support:

• src/rfdetr/datasets/coco.py – Implements COCO dataset loading, JSON parsing, and conversion utilities for bounding box, segmentation, and keypoint annotations.
• src/rfdetr/datasets/yolo.py – Handles YOLO dataset ingestion, including data.yaml parsing and coordinate normalization.
• src/rfdetr/models/_defaults.py – Contains validation logic for coco_path and other dataset-related constructor arguments.
• src/rfdetr/models/__init__.py – Exposes high-level model classes (RFDETRSmall, RFDETRSegSmall, RFDETRKeypointPreview) that provide the unified .train() interface.

Summary

• RF-DETR supports COCO JSON and YOLO dataset formats for all training tasks.
• Automatic detection occurs by checking for train/_annotations.coco.json (COCO) or data.yaml plus train/images/ (YOLO).
• COCO segmentation requires the segmentation field in annotation objects.
• YOLO pose datasets require the kpt_shape parameter in data.yaml for keypoint training.
• The dataset loaders in src/rfdetr/datasets/ handle format-specific parsing transparently.

Frequently Asked Questions

How does RF-DETR detect which dataset format I'm using?

RF-DETR inspects the directory structure provided to dataset_dir. If it finds train/_annotations.coco.json, it routes to the COCO loader. If it finds data.yaml alongside train/images/ and train/labels/ folders, it routes to the YOLO loader. This logic is enforced in src/rfdetr/models/_defaults.py and executed in the dataset factories.

Can I use a custom train/validation split with RF-DETR?

Yes. For COCO, place valid/_annotations.coco.json in the dataset root alongside the train file. For YOLO, define the val path in data.yaml pointing to your validation images and labels directories. You may also include a test split using the same naming conventions.

Does RF-DETR support YOLO segmentation datasets?

Yes. RF-DETR's YOLO parser in src/rfdetr/datasets/yolo.py supports segmentation masks encoded as polygon coordinates in YOLO label files. Use RFDETRSegSmall or other segmentation variants to train on these datasets.

What COCO fields are required for segmentation training?

Beyond the standard images, annotations, and categories fields, each annotation record must contain a segmentation field with polygon coordinates or RLE encoding. The COCO loader checks for this field during dataset initialization in src/rfdetr/datasets/coco.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}