How RWKV-CLIP Handles Raw, Synthetic, and Generated Text During Training

RWKV-CLIP randomly selects one of three 77-token text blocks—raw captions, synthetic captions from text-to-image models, or RWKV-generated captions—with equal probability during each training step to improve multimodal alignment.

The deepglint/rwkv-clip repository implements a unique text augmentation strategy for contrastive language-image pre-training (CLIP). Instead of training on fixed captions, the model processes image-text pairs where each sample contains three distinct caption sources concatenated into a single tensor. During the forward pass, the training loop randomly slices one of these three 77-token blocks, exposing the CLIP head to diverse textual distributions ranging from human annotations to synthetic generations.

The Three-Text Tensor Structure

For every image in a training batch, the dataloader provides a text_token tensor of length 231 tokens (77 × 3). This tensor contains three consecutive blocks that the training code randomly samples using random.choice([0, 1, 2]).

Raw Text (Original Dataset Captions)

The first block stores raw text—the original human-written captions from datasets like Flickr30K or COCO. In train.py, this corresponds to the slice text_token[i, :77]. These annotations provide high-quality, natural language descriptions that anchor the model to human linguistic patterns.

Synthetic Text (Diffusion Model Captions)

The second block contains synthetic text generated by external text-to-image models such as Stable Diffusion. Accessed via text_token[i, 77*1:77*2] (indices 77–154), these captions capture the distributional bias of diffusion-based generators, helping the CLIP model align images with machine-generated prompts commonly used in generative applications.

Generated Text (RWKV Language Model Captions)

The third block holds generated text produced by the RWKV language model itself. This slice text_token[i, 77*2:77*3] (indices 154–231) contains captions generated by the model's own text backbone, teaching the CLIP head to understand recursive, self-referential language generation while maintaining image-grounded semantics.

Random Selection Implementation in train.py

The core augmentation logic resides in train.py between lines 279–295. During each training iteration, the code iterates over the batch and randomly selects one of the three 77-token windows for every sample:

# train.py – random text-type selection (lines 284-294)

from random import choice

new_text_token = []
for i in range(text_token.size(0)):
    choose = choice([0, 1, 2])
    if choose == 0:                     # Raw Text

        new_text_token.append(text_token[i, :77].long().cuda())
    elif choose == 1:                   # Synthetic Text

        new_text_token.append(text_token[i, 77*1:77*2].long().cuda())
    elif choose == 2:                   # Generated Text

        new_text_token.append(text_token[i, 77*2:77*3].long().cuda())

text_token = torch.stack(new_text_token, dim=0)

After selection, the chosen slice is converted to long type, moved to the GPU via .cuda(), and restacked into a new tensor with shape [batch_size, 77]. This tensor is then fed into the CLIP text encoder defined in the model architecture.

Data Pipeline and Tokenization

The three-text concatenation happens in the dataloader before the training loop begins. Files such as dataloaders/flickr30k.py and dataloaders/coco.py load the raw, synthetic, and generated captions for each image and concatenate them into a single string sequence. The model/open_clip/tokenizer.py then tokenizes this concatenated string into the 231-token tensor using the Open-CLIP BPE tokenizer.

This preprocessing ensures that text_token arrives in the training loop already structured as [raw_77_tokens][synthetic_77_tokens][generated_77_tokens], allowing the simple slicing logic in train.py to operate efficiently without additional data loading overhead.

Contrastive Loss Computation

After text selection and encoding, the loss.py module computes the symmetric cross-entropy loss between image and text embeddings. Because each batch contains a random mixture of raw, synthetic, and generated captions, the contrastive objective forces the image encoder to produce embeddings that are invariant to the specific textual source while maintaining semantic alignment with the visual content. This diversity acts as a regularizer, improving zero-shot retrieval performance when the model encounters out-of-distribution captions during inference.

Inspecting Text Selection During Training

To verify which text types are being selected in real-time, you can instrument the training loop to log the random choices:

import random
import torch

# Inside the training loop after receiving text_token from dataloader

choices_log = []
augmented_texts = []

for i in range(text_token.size(0)):
    c = random.choice([0, 1, 2])
    choices_log.append(c)
    
    if c == 0:
        augmented_texts.append(text_token[i, :77])
    elif c == 1:
        augmented_texts.append(text_token[i, 77:154])
    else:
        augmented_texts.append(text_token[i, 154:231])

print(f"Text types selected (0=raw, 1=synthetic, 2=generated): {choices_log}")
text_token = torch.stack(augmented_texts, dim=0).long().cuda()

This approach helps debug distribution shifts and confirms that all three text sources are being utilized throughout training epochs.

Launching Training with Three-Text Augmentation

To start a training run that leverages this augmentation strategy on the Flickr30K dataset:

python train.py \
  --batch-size 256 \
  --epochs 32 \
  --train-data /path/to/flickr30k \
  --train-num-samples 100000 \
  --output ./rwkv_clip_checkpoints \
  --lr 0.1 \
  --precision bf16

The script automatically applies the random text-type selection for every batch, cycling between raw, synthetic, and generated captions as training progresses.

Summary

• RWKV-CLIP stores three caption types—raw, synthetic, and RWKV-generated—in a single 231-token tensor per image (77 tokens per block).
• Random uniform sampling (choice([0,1,2])) in train.py selects one block per training step, exposing the model to diverse textual distributions.
• Data preparation occurs in dataloaders/flickr30k.py and similar files, which concatenate captions before tokenization via model/open_clip/tokenizer.py.
• Implementation details include explicit tensor slicing (:77, 77:154, 154:231), GPU transfer with .cuda(), and restacking into the input batch.
• Robustness benefits arise from training the CLIP head to align images with both human-written and model-generated descriptions, improving zero-shot transfer to generative tasks.

Frequently Asked Questions

How does RWKV-CLIP decide which text type to use for each image?

The training loop in train.py calls random.choice([0, 1, 2]) for every sample in the batch, selecting raw text (0), synthetic text (1), or generated text (2) with equal probability. This decision happens on the CPU before the selected tensor slice is moved to the GPU, ensuring independent randomization across the batch.

What is the difference between synthetic and generated text in RWKV-CLIP?

Synthetic text refers to captions created by external text-to-image diffusion models like Stable Diffusion, while generated text refers to captions produced by the RWKV language model component within the RWKV-CLIP architecture itself. Synthetic text captures diffusion model priors, whereas generated text aligns the CLIP embeddings with the model's own recursive language generation capabilities.

Why concatenate all three text types instead of loading them separately?

Concatenating raw, synthetic, and generated captions into a single 231-token tensor in the dataloader (dataloaders/flickr30k.py, dataloaders/coco.py) minimizes data loading overhead and simplifies the training loop logic. The tokenization in model/open_clip/tokenizer.py processes the full string once, and train.py simply slices the pre-computed tensor indices without requiring complex branching in the data pipeline.

Which files handle the three-text format in the RWKV-CLIP repository?

The primary files are train.py (lines 279–295) for random selection logic, dataloaders/flickr30k.py and dataloaders/coco.py for concatenating the three caption sources during data loading, and model/open_clip/tokenizer.py for converting the concatenated strings into the token tensor consumed by the training loop.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}