How to Download and Preprocess the YFCC15M Dataset for RWKV-CLIP Training

Download the YFCC15M dataset using the provided aria2c-based script from Hugging Face, then convert the raw image-text pairs into MXNet .rec recordIO format using the data2rec.py utility before feeding them to the DALI data loader.

The RWKV-CLIP training pipeline expects the YFCC15M image-text corpus in a specific binary format optimized for NVIDIA DALI. According to the deepglint/rwkv-clip source code, the repository provides automated shell scripts for fast parallel downloads and Python utilities for recordIO serialization.

Install Parallel Download Tools

The download script automatically selects aria2c for multi-connection downloads if available, falling back to wget otherwise. Install aria2 for maximum throughput:

sudo apt update && sudo apt install -y aria2

Verify the installation by checking that aria2c is in your PATH. The training code itself requires MXNet and DALI, but the download phase only needs standard Unix utilities plus one of these downloaders.

Download YFCC15M from Hugging Face

Run the official wrapper script that fetches the dataset from the Kaichengalex/YFCC15M Hugging Face repository:

bash data/download_YFCC15M.sh

Under the hood, data/download_YFCC15M.sh invokes data/hfd.sh to perform three operations:

1. Query the Hugging Face API to enumerate all files in the dataset repository
2. Generate an aria2c-compatible manifest at .hfd/aria2c_urls.txt
3. Launch aria2c -x 10 to open ten parallel connections and stream files into the YFCC15M/ directory

If aria2 is not installed, the script transparently falls back to sequential wget requests. The complete dataset contains approximately 15,061,515 image-text pairs, so parallel downloading significantly reduces wait times.

Convert Raw Images to MXNet RecordIO Format

The training code in dataloaders/imagenet.py expects data packaged as MXNet .rec files (recordIO format) rather than loose JPEGs. Use the provided data2rec.py as a template to serialize the downloaded images and tokenized captions.

Step-by-Step Conversion Process

First, ensure you can import the repository's tokenizer:

from src.open_alip import tokenize
import mxnet as mx
import cv2
import numpy as np

For each image-text pair in YFCC15M/, pack the data into a binary record:

# Initialize the RecordIO writer

rec = mx.recordio.MXIndexedRecordIO('datarec.idx', 'datarec.rec', 'w')

# Example iteration over a single sample

img_path = 'YFCC15M/img/000000001.jpg'
caption_path = 'YFCC15M/captions/000000001.txt'

img = cv2.imread(img_path)
text = open(caption_path).read().strip()

# Tokenize the caption into integer IDs

txt_ids = tokenize(text).flatten().numpy()

# Create header: label field stores the token IDs

header = mx.recordio.IRHeader(flag=0, label=txt_ids, id=0, id2=0)

# Pack image bytes with the header

packed = mx.recordio.pack_img(header, img)

# Write to index 0 (increment for subsequent items)

rec.write_idx(0, packed)

In production, wrap this logic in a loop over all 15M samples. The script produces two critical outputs:

• datarec.idx — Index mapping for random access
• datarec.rec — Binary record data containing images and tokenized text arrays

These files replace the default train.rec and val.rec paths referenced in the training configuration.

Verify Data Loading with DALI

Before launching full training, validate that the .rec files are readable by the DALI pipeline. The repository provides dali.py as a convenience wrapper around dataloaders/imagenet.py:

python dali.py --data-dir /path/to/YFCC15M --rec-file datarec.rec

This instantiates a DALIClassificationIterator that reads the recordIO files and yields batches of (image_tensor, caption_ids) pairs. Successful execution confirms that the preprocessing pipeline aligns with the expectations of the RWKV-CLIP model input layers.

Launch Training

With the records in place, start distributed training by pointing the script to your data directory:

export DATA_ROOT=/path/to/YFCC15M
bash shell/train_RWKV_CLIP_B32_YFCC15M.sh

The training entry point searches for .rec files within DATA_ROOT (defaulting to ./YFCC15M if unset) and loads them via the DALI pipeline initialized in dataloaders/imagenet.py.

Summary

• Use aria2c via data/download_YFCC15M.sh to fetch 15M image-text pairs from Hugging Face in parallel
• Transform raw data into MXNet recordIO format using the pattern shown in data2rec.py, tokenizing captions with src.open_alip.tokenize
• Produce .rec and .idx files that the DALI loader in dataloaders/imagenet.py consumes for high-throughput training
• Set DATA_ROOT before calling shell/train_RWKV_CLIP_B32_YFCC15M.sh to point the trainer at your preprocessed records

Frequently Asked Questions

How large is the YFCC15M dataset when fully downloaded?

The dataset contains approximately 15 million image-text pairs. When stored as raw JPEGs and text files, expect roughly 300-400 GB of disk space. After conversion to MXNet .rec format, the binary records may occupy slightly more space due to indexing overhead, so allocate at least 500 GB total storage.

Can I use wget instead of aria2c for the download?

Yes. The data/download_YFCC15M.sh script detects whether aria2c is installed; if not present, it automatically falls back to wget for sequential downloading. However, aria2c is strongly recommended because the -x 10 parallel connection feature reduces download time from days to hours.

Why does RWKV-CLIP require MXNet recordIO format instead of standard image folders?

The training pipeline uses NVIDIA DALI (dataloaders/imagenet.py) for GPU-accelerated data loading. RecordIO format allows DALI to perform asynchronous I/O and automatic batching without Python GIL contention. The .rec files pack both images and tokenized caption arrays into a single binary stream, enabling the DALIClassificationIterator to feed the RWKV backbone efficiently.

What tokenizer does data2rec.py use for the captions?

The preprocessing script imports tokenize from src.open_alip, which implements the ALIP (Advanced Language-Image Pretraining) tokenization scheme. This tokenizer converts raw caption strings into integer ID sequences that the RWKV language model component expects as input targets during contrastive training.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}