# How to Configure and Use NVIDIA DALI for Efficient Data Loading in RWKV-CLIP Training

> Configure NVIDIA DALI for efficient data loading in RWKV-CLIP training. Accelerate your workflow with GPU-side decoding and augmentation.

- Repository: [DeepGlint/rwkv-clip](https://github.com/deepglint/rwkv-clip)
- Tags: how-to-guide
- Published: 2026-02-28

---

**Use NVIDIA DALI with MXNet RecordIO files to enable GPU-side decoding and augmentation, eliminating CPU bottlenecks during RWKV-CLIP training on massive image-text datasets.**

Training vision-language models like RWKV-CLIP on web-scale corpora such as YFCC-15M requires feeding high-resolution images to the GPU at extreme throughput. The `deepglint/rwkv-clip` repository implements an NVIDIA DALI-based data loader that streams pre-packed MXNet RecordIO files directly to the GPU, bypassing CPU decoding entirely. This guide explains how to configure and use this pipeline for efficient distributed training.

## Why Use NVIDIA DALI for RWKV-CLIP Training?

Standard PyTorch `DataLoader` with `PIL` or `OpenCV` decoding creates a CPU bottleneck when loading millions of high-resolution images. The RWKV-CLIP repository solves this with **NVIDIA DALI**, a library for GPU-accelerated data loading and augmentation. Key advantages include:

- **GPU-side decoding and augmentation** – JPEGs are decoded on the GPU using `fn.decoders.image_random_crop`, avoiding expensive CPU-GPU transfer bottlenecks.
- **Parallel prefetching pipelines** – Multiple threads (`num_threads=args.workers`) feed the model while the previous batch is still being processed, with `prefetch_queue_depth=3` ensuring the GPU never starves.
- **Built-in augmentation fusion** – Random crops, resize, mirror, and mean-std normalization are performed in a single fused pipeline, maximizing throughput.

## Preparing MXNet RecordIO Data for DALI

DALI expects data in **MXNet RecordIO** format (`.rec` file with accompanying `.idx` index). Unlike standard image folders, this format allows DALI to perform sequential reads and sharding efficiently.

### Converting Image-Text Pairs to RecordIO Format

The repository provides [`data2rec.py`](https://github.com/deepglint/rwkv-clip/blob/main/data2rec.py) as a reference for packing image-text pairs. Each record stores a JPEG image and its tokenized text label (three caption variants concatenated). Here is the core logic for creating a single record:

```python

# data2rec.py – example of creating a RecordIO item

import mxnet as mx
import numpy as np
import cv2
from src.open_alip import tokenize

save_record = mx.recordio.MXIndexedRecordIO('datarec.idx', 'datarec.rec', 'w')
img = cv2.imread('img/architecture.jpg')
text_raw = 'Nice to meet you!'
label = tokenize(text_raw).flatten().numpy()

header = mx.recordio.IRHeader(flag=0, label=label, id=1, id2=0)
item = mx.recordio.pack_img(header, img)
save_record.write_idx(0, item)

```

Run a similar conversion over your entire training set (e.g., YFCC-15M), producing `<train_data>.rec` and `<train_data>.idx`. The path prefix (without extension) is passed to the training script as `args.train_data`.

## Configuring the DALI Pipeline

The [`dali.py`](https://github.com/deepglint/rwkv-clip/blob/main/dali.py) file implements the `dali_dataloader` function, which constructs a GPU-accelerated pipeline and returns a PyTorch-compatible iterator via `DALIWarper`.

### Distributed Training Environment Variables

Before launching training, export the standard PyTorch DistributedDataParallel (DDP) variables. DALI uses these to shard the dataset across GPUs:

```bash
export RANK=0               # global rank of this process

export LOCAL_RANK=0         # GPU id on the node

export WORLD_SIZE=1         # total number of processes

```

In multi-node training, ensure these variables match the process launcher (e.g., `torchrun` or `torch.distributed.launch`).

### The dali_dataloader Function and Pipeline Definition

The pipeline is defined in [`dali.py`](https://github.com/deepglint/rwkv-clip/blob/main/dali.py). It uses `fn.readers.mxnet` to read RecordIO files and `fn.decoders.image_random_crop` for GPU-side decoding and random cropping.

```python

# dali.py – core implementation

import os
from nvidia.dali import Pipeline, fn, types
from nvidia.dali.plugin.pytorch import DALIClassificationIterator

class DALIWarper(object):
    """Wraps DALIClassificationIterator to expose a PyTorch-style iterator."""
    def __init__(self, dali_iter):
        self.iter = dali_iter

    def __iter__(self):
        return self

    def __next__(self):
        batch = next(self.iter)
        # DALI returns list of dictionaries; extract tensors

        images = batch[0]["data"]
        labels = batch[0]["label"]
        return images, labels

    def reset(self):
        self.iter.reset()

def dali_dataloader(args):
    """Create a DALI pipeline and return a PyTorch-compatible iterator."""
    rec_file = f"{args.train_data}.rec"
    idx_file = f"{args.train_data}.idx"

    pipe = Pipeline(
        batch_size=args.batch_size,
        num_threads=args.workers,
        device_id=int(os.environ["LOCAL_RANK"]),
        prefetch_queue_depth=3,
        seed=int(os.environ["RANK"]) + 1467,
    )

    with pipe:
        # MXNet RecordIO reader – sharded for DDP

        jpegs, labels = fn.readers.mxnet(
            path=rec_file,
            index_path=idx_file,
            initial_fill=16384,
            num_shards=int(os.environ["WORLD_SIZE"]),
            shard_id=int(os.environ["RANK"]),
            random_shuffle=True,
            pad_last_batch=False,
            name="train",
        )

        # Training augmentations on GPU

        images = fn.decoders.image_random_crop(
            jpegs, device="mixed", output_type=types.RGB,
            random_aspect_ratio=[0.8, 1.25],
            random_area=[0.7, 1.0],
            num_attempts=100,
        )
        images = fn.resize(
            images, device="gpu",
            resize_x=args.input_size, resize_y=args.input_size,
            interp_type=types.INTERP_LINEAR,
        )
        mirror = fn.random.coin_flip(probability=0.5)

        # CLIP normalization constants

        mean = [0.48145466, 0.4578275, 0.40821073]
        std  = [0.26862954, 0.26130258, 0.27577711]

        images = fn.crop_mirror_normalize(
            images.gpu(),
            dtype=types.FLOAT,
            output_layout="CHW",
            crop=(args.input_size, args.input_size),
            mean=[x * 255 for x in mean],
            std=[x * 255 for x in std],
            mirror=mirror,
        )
        pipe.set_outputs(images, labels)

    pipe.build()
    return DALIWarper(
        DALIClassificationIterator(pipelines=[pipe], reader_name="train")
    )

```

Key configuration parameters in this pipeline include:

- **`num_threads=args.workers`** – Sets CPU threads for I/O and preprocessing.
- **`prefetch_queue_depth=3`** – Queues three batches ahead to hide latency.
- **`num_shards`/`shard_id`** – Ensures each DDP process reads a unique data partition.
- **Augmentation parameters** – `random_aspect_ratio=[0.8, 1.25]` and `random_area=[0.7, 1.0]` match the CLIP training recipe.

## Integrating DALI into the Training Loop

The [`train.py`](https://github.com/deepglint/rwkv-clip/blob/main/train.py) entry point consumes the DALI loader like a standard PyTorch DataLoader. Because DALI already places tensors on GPU, you avoid `to(device)` overhead.

```python

# train.py – data loading excerpt

from dali import dali_dataloader

def main(args):
    # ... model setup ...

    RWKV_CLIP_model = get_model_RWKV_CLIP(args)
    
    # Initialize DALI loader

    train_loader = dali_dataloader(args)

    for epoch in range(start_epoch, math.ceil(args.epochs)):
        for _, (img, text_token) in enumerate(train_loader):
            # img is already a GPU tensor (N, C, H, W)

            img = img.cuda()
            # text_token contains tokenized captions

            # ... training step ...

        
        # Critical: reset iterator for next epoch

        train_loader.reset()

```

**Critical requirement:** Call `train_loader.reset()` after each epoch. Unlike standard PyTorch DataLoader, DALI maintains internal state for shuffling and sharding. Without resetting, the next epoch may start with stale buffers or incorrect sharding offsets.

## Troubleshooting Common DALI Configuration Issues

| Issue | Cause | Solution |
|-------|-------|----------|
| **GPU Out-of-Memory** | Batch size too large for DALI’s prefetch queue | Reduce `args.batch_size` or lower `prefetch_queue_depth` from 3 to 2 |
| **Duplicate data across ranks** | `WORLD_SIZE` or `RANK` env vars mismatch DDP launcher | Verify `export WORLD_SIZE=${NGPUS}` matches `torchrun --nproc_per_node` |
| **FileNotFoundError for .rec/.idx** | `args.train_data` includes extension or wrong path | Ensure path is the base name without extension; DALI appends `.rec` and `.idx` |
| **Input size mismatch** | `args.input_size` differs from model config | Keep `args.input_size` consistent with `fn.resize` dimensions and model expected input |

## Summary

Configuring **NVIDIA DALI for efficient data loading in RWKV-CLIP training** involves four key steps:

- **Convert raw data to MXNet RecordIO** using [`data2rec.py`](https://github.com/deepglint/rwkv-clip/blob/main/data2rec.py) to create `.rec` and `.idx` files that DALI can stream efficiently.
- **Set DDP environment variables** (`RANK`, `LOCAL_RANK`, `WORLD_SIZE`) before launching to enable proper data sharding across GPUs.
- **Initialize the DALI pipeline** via `dali_dataloader()` in [`dali.py`](https://github.com/deepglint/rwkv-clip/blob/main/dali.py), configuring `batch_size`, `workers`, and `prefetch_queue_depth` to match your hardware.
- **Integrate into the training loop** by iterating over the `DALIWarper` iterator and calling `reset()` after each epoch to maintain correct shuffling and sharding.

This setup eliminates CPU decoding bottlenecks, keeps GPUs saturated with pre-processed batches, and scales efficiently to multi-node training.

## Frequently Asked Questions

### What data format does the RWKV-CLIP DALI loader require?

The loader requires **MXNet RecordIO** format, consisting of a `.rec` data file and a `.idx` index file. Unlike standard image folders, this format allows DALI to perform sequential reads, random shuffling, and efficient sharding across distributed processes. You can generate these files using the [`data2rec.py`](https://github.com/deepglint/rwkv-clip/blob/main/data2rec.py) utility script provided in the repository.

### How do I fix duplicate data across GPUs when using DALI?

Duplicate data occurs when the environment variables `RANK`, `LOCAL_RANK`, and `WORLD_SIZE` do not match your DistributedDataParallel (DDP) launcher configuration. Ensure you export these variables before starting training:

```bash
export RANK=$OMPI_COMM_WORLD_RANK  # or $SLURM_PROCID

export LOCAL_RANK=$OMPI_COMM_WORLD_LOCAL_RANK
export WORLD_SIZE=$OMPI_COMM_WORLD_SIZE

```

The `dali_dataloader` function uses `num_shards=WORLD_SIZE` and `shard_id=RANK` to partition the RecordIO file uniquely per GPU.

### Why must I call train_loader.reset() after each epoch?

Unlike PyTorch's standard DataLoader, DALI maintains internal state for shuffling buffers and sharding offsets. Calling `train_loader.reset()` at the end of each epoch re-initializes these buffers, ensuring that the next epoch starts with fresh random shuffling and correct shard offsets. Omitting this call can lead to deterministic repetition of batches or sharding errors in subsequent epochs.

### Can I use DALI with standard image folders instead of RecordIO?

The current [`dali.py`](https://github.com/deepglint/rwkv-clip/blob/main/dali.py) implementation is specifically designed for MXNet RecordIO files using `fn.readers.mxnet`. While DALI supports other readers like `fn.readers.file` for image folders, the RWKV-CLIP training scripts expect the RecordIO format to efficiently handle the image-text pairs and associated metadata (tokenized captions stored as labels). To use standard folders, you would need to modify [`dali.py`](https://github.com/deepglint/rwkv-clip/blob/main/dali.py) to use `fn.readers.file` and `fn.decoders.image`, and adjust the label handling logic accordingly.