# How to Download and Preprocess the YFCC15M Dataset for RWKV-CLIP Training

> Learn to download and preprocess the YFCC15M dataset for RWKV-CLIP training. Convert raw data to MXNet rec format using data2rec.py for efficient DALI loading.

- Repository: [DeepGlint/rwkv-clip](https://github.com/deepglint/rwkv-clip)
- Tags: how-to-guide
- Published: 2026-02-28

---

**Download the YFCC15M dataset using the provided `aria2c`-based script from Hugging Face, then convert the raw image-text pairs into MXNet `.rec` recordIO format using the [`data2rec.py`](https://github.com/deepglint/rwkv-clip/blob/main/data2rec.py) utility before feeding them to the DALI data loader.**

The RWKV-CLIP training pipeline expects the YFCC15M image-text corpus in a specific binary format optimized for NVIDIA DALI. According to the `deepglint/rwkv-clip` source code, the repository provides automated shell scripts for fast parallel downloads and Python utilities for recordIO serialization.

## Install Parallel Download Tools

The download script automatically selects **aria2c** for multi-connection downloads if available, falling back to **wget** otherwise. Install aria2 for maximum throughput:

```bash
sudo apt update && sudo apt install -y aria2

```

Verify the installation by checking that `aria2c` is in your PATH. The training code itself requires MXNet and DALI, but the download phase only needs standard Unix utilities plus one of these downloaders.

## Download YFCC15M from Hugging Face

Run the official wrapper script that fetches the dataset from the `Kaichengalex/YFCC15M` Hugging Face repository:

```bash
bash data/download_YFCC15M.sh

```

Under the hood, [`data/download_YFCC15M.sh`](https://github.com/deepglint/rwkv-clip/blob/main/data/download_YFCC15M.sh) invokes [`data/hfd.sh`](https://github.com/deepglint/rwkv-clip/blob/main/data/hfd.sh) to perform three operations:
1. Query the Hugging Face API to enumerate all files in the dataset repository
2. Generate an **aria2c**-compatible manifest at [`.hfd/aria2c_urls.txt`](https://github.com/deepglint/rwkv-clip/blob/main/.hfd/aria2c_urls.txt)
3. Launch `aria2c -x 10` to open ten parallel connections and stream files into the `YFCC15M/` directory

If aria2 is not installed, the script transparently falls back to sequential `wget` requests. The complete dataset contains approximately 15,061,515 image-text pairs, so parallel downloading significantly reduces wait times.

## Convert Raw Images to MXNet RecordIO Format

The training code in [`dataloaders/imagenet.py`](https://github.com/deepglint/rwkv-clip/blob/main/dataloaders/imagenet.py) expects data packaged as MXNet **.rec** files (recordIO format) rather than loose JPEGs. Use the provided [`data2rec.py`](https://github.com/deepglint/rwkv-clip/blob/main/data2rec.py) as a template to serialize the downloaded images and tokenized captions.

### Step-by-Step Conversion Process

First, ensure you can import the repository's tokenizer:

```python
from src.open_alip import tokenize
import mxnet as mx
import cv2
import numpy as np

```

For each image-text pair in `YFCC15M/`, pack the data into a binary record:

```python

# Initialize the RecordIO writer

rec = mx.recordio.MXIndexedRecordIO('datarec.idx', 'datarec.rec', 'w')

# Example iteration over a single sample

img_path = 'YFCC15M/img/000000001.jpg'
caption_path = 'YFCC15M/captions/000000001.txt'

img = cv2.imread(img_path)
text = open(caption_path).read().strip()

# Tokenize the caption into integer IDs

txt_ids = tokenize(text).flatten().numpy()

# Create header: label field stores the token IDs

header = mx.recordio.IRHeader(flag=0, label=txt_ids, id=0, id2=0)

# Pack image bytes with the header

packed = mx.recordio.pack_img(header, img)

# Write to index 0 (increment for subsequent items)

rec.write_idx(0, packed)

```

In production, wrap this logic in a loop over all 15M samples. The script produces two critical outputs:
- `datarec.idx` — Index mapping for random access
- `datarec.rec` — Binary record data containing images and tokenized text arrays

These files replace the default `train.rec` and `val.rec` paths referenced in the training configuration.

## Verify Data Loading with DALI

Before launching full training, validate that the `.rec` files are readable by the DALI pipeline. The repository provides [`dali.py`](https://github.com/deepglint/rwkv-clip/blob/main/dali.py) as a convenience wrapper around [`dataloaders/imagenet.py`](https://github.com/deepglint/rwkv-clip/blob/main/dataloaders/imagenet.py):

```bash
python dali.py --data-dir /path/to/YFCC15M --rec-file datarec.rec

```

This instantiates a `DALIClassificationIterator` that reads the recordIO files and yields batches of `(image_tensor, caption_ids)` pairs. Successful execution confirms that the preprocessing pipeline aligns with the expectations of the RWKV-CLIP model input layers.

## Launch Training

With the records in place, start distributed training by pointing the script to your data directory:

```bash
export DATA_ROOT=/path/to/YFCC15M
bash shell/train_RWKV_CLIP_B32_YFCC15M.sh

```

The training entry point searches for `.rec` files within `DATA_ROOT` (defaulting to `./YFCC15M` if unset) and loads them via the DALI pipeline initialized in [`dataloaders/imagenet.py`](https://github.com/deepglint/rwkv-clip/blob/main/dataloaders/imagenet.py).

## Summary

- **Use `aria2c`** via [`data/download_YFCC15M.sh`](https://github.com/deepglint/rwkv-clip/blob/main/data/download_YFCC15M.sh) to fetch 15M image-text pairs from Hugging Face in parallel
- **Transform raw data** into MXNet recordIO format using the pattern shown in [`data2rec.py`](https://github.com/deepglint/rwkv-clip/blob/main/data2rec.py), tokenizing captions with `src.open_alip.tokenize`
- **Produce `.rec` and `.idx` files** that the DALI loader in [`dataloaders/imagenet.py`](https://github.com/deepglint/rwkv-clip/blob/main/dataloaders/imagenet.py) consumes for high-throughput training
- **Set `DATA_ROOT`** before calling [`shell/train_RWKV_CLIP_B32_YFCC15M.sh`](https://github.com/deepglint/rwkv-clip/blob/main/shell/train_RWKV_CLIP_B32_YFCC15M.sh) to point the trainer at your preprocessed records

## Frequently Asked Questions

### How large is the YFCC15M dataset when fully downloaded?

The dataset contains approximately 15 million image-text pairs. When stored as raw JPEGs and text files, expect roughly 300-400 GB of disk space. After conversion to MXNet `.rec` format, the binary records may occupy slightly more space due to indexing overhead, so allocate at least 500 GB total storage.

### Can I use wget instead of aria2c for the download?

Yes. The [`data/download_YFCC15M.sh`](https://github.com/deepglint/rwkv-clip/blob/main/data/download_YFCC15M.sh) script detects whether `aria2c` is installed; if not present, it automatically falls back to `wget` for sequential downloading. However, aria2c is strongly recommended because the `-x 10` parallel connection feature reduces download time from days to hours.

### Why does RWKV-CLIP require MXNet recordIO format instead of standard image folders?

The training pipeline uses **NVIDIA DALI** ([`dataloaders/imagenet.py`](https://github.com/deepglint/rwkv-clip/blob/main/dataloaders/imagenet.py)) for GPU-accelerated data loading. RecordIO format allows DALI to perform asynchronous I/O and automatic batching without Python GIL contention. The `.rec` files pack both images and tokenized caption arrays into a single binary stream, enabling the `DALIClassificationIterator` to feed the RWKV backbone efficiently.

### What tokenizer does [`data2rec.py`](https://github.com/deepglint/rwkv-clip/blob/main/data2rec.py) use for the captions?

The preprocessing script imports `tokenize` from `src.open_alip`, which implements the ALIP (Advanced Language-Image Pretraining) tokenization scheme. This tokenizer converts raw caption strings into integer ID sequences that the RWKV language model component expects as input targets during contrastive training.