How to Evaluate RWKV-CLIP on Standard Zero-Shot Classification Benchmarks

Evaluate RWKV-CLIP by building text prototypes with zero_shot_classifier in zero_shot.py, running inference with WarperCLIP_V_T_RWKV_method, and aggregating distributed results to compute top-1 accuracy or mAP on datasets like ImageNet, CIFAR-10, and CIFAR-100.

The deepglint/rwkv-clip repository provides a complete evaluation pipeline for assessing RWKV-CLIP's zero-shot classification capabilities without fine-tuning. This guide explains how to run standard benchmarks using the provided scripts and APIs.

Zero-Shot Evaluation Pipeline Overview

The evaluation process follows three distinct stages implemented in zero_shot.py:

1. Prepare a zero-shot classifier – For each target dataset, the model constructs class prototypes by encoding textual prompts. The zero_shot_classifier function calls WarperCLIP_V_T_RWKV_text_change_head from model/utils.py to generate normalized text embeddings.

2. Run inference on the image test split – Images are processed through the vision backbone (WarperCLIP_V_T_RWKV_method), normalized, and compared against class prototypes to obtain logits. The core inference loop resides in the run function of zero_shot.py.

3. Aggregate results across distributed ranks – When using multi-GPU setups, each rank stores partial logits and targets, which are collected via torch.distributed.all_gather_object. Final metrics (top-1 accuracy, mean-per-class accuracy, mAP, or ROC-AUC) are computed based on dataset-specific flags.

Step-by-Step Implementation Guide

Configuration and Model Loading

The evaluation script begins by loading dataset configurations and instantiating the model. Text prompts and class names are loaded from utils/template.json and utils/label.json, while the model architecture is defined in model_config/RWKV_CLIP_B32.json.

import json
import torch
from model import create_RWKV_Model
from model.utils import WarperCLIP_V_T_RWKV_text_change_head
from open_clip.transform import image_transform
from PIL import Image

# Load configuration

args = json.load(open('model_config/RWKV_CLIP_B32.json'))
model = create_RWKV_Model(args, model_weight_path='RWKV_CLIP_B32_YFCC15M.pt')
model.eval().cuda()

Building the Zero-Shot Classifier

The classifier matrix is constructed by encoding prompt templates for each class. For CIFAR-10, the system loads templates from utils/template.json and class names from utils/label.json, then averages the embeddings for each class.

# Prepare image transform

transform = image_transform(args['input_size'], False)

# Build zero-shot classifier for CIFAR-10

with open('utils/label.json') as f:
    labels = json.load(f)['cifar10']
with open('utils/template.json') as f:
    templates = json.load(f)['cifar10']

classifier = []
for classname in labels:
    texts = [tpl.format(classname) for tpl in templates]
    tokens = tokenize(texts).cuda()
    txt_emb = WarperCLIP_V_T_RWKV_text_change_head(model, tokens)
    cls_emb = torch.nn.functional.normalize(txt_emb, dim=-1).mean(0)
    classifier.append(cls_emb)

classifier = torch.stack(classifier).cuda()  # [num_classes, embed_dim]

Inference and Metric Aggregation

During evaluation, images are encoded using WarperCLIP_V_T_RWKV_method and compared against the classifier matrix. The run function in zero_shot.py handles distributed gathering of predictions and computes dataset-specific metrics.

# Inference loop (simplified)

with torch.no_grad():
    for images, targets in dataloader:
        images = images.cuda()
        image_features = WarperCLIP_V_T_RWKV_method(model, images)
        image_features = torch.nn.functional.normalize(image_features, dim=-1)
        logits = 100. * image_features @ classifier.T
        # Store logits and targets for metric computation

For distributed evaluation, the script uses torch.distributed.all_gather_object to collect tensors from all ranks before computing final accuracy or mAP scores.

Running the Evaluation

Command-Line Evaluation (Distributed)

The repository provides a shell script to launch distributed evaluation across multiple GPUs. The script shell/test_zero_shot_classificaiton.sh uses torchrun to manage processes.

# Launch evaluation on 8 GPUs

bash shell/test_zero_shot_classificaiton.sh \
    --model-weight path/to/RWKV_CLIP_B32_YFCC15M.pt \
    --dataset imagenet,cifar10,cifar100,food101,oxford_pets \
    --output-dir ./zero_shot_results.csv

Key arguments:

• --model-weight: Path to the pretrained checkpoint (e.g., RWKV_CLIP_B32_YFCC15M.pt)
• --dataset: Comma-separated list of dataset names
• --output-dir: File path for writing results (appended as CSV)

Python API for Custom Evaluation

For notebook-based experimentation, import the evaluation components directly from zero_shot.py and model/utils.py. This approach allows custom dataset integration without modifying the shell scripts.

from zero_shot import run, zero_shot_classifier
import torch

# Configure your model and datasets

model = create_RWKV_Model(args, model_weight_path='checkpoint.pt')
classifier = zero_shot_classifier(model, classnames, templates)

# Run evaluation

results = run(model, classifier, dataloader, args)

Interpreting Output Results

The evaluation script writes results in CSV format where each line corresponds to a model checkpoint and columns represent dataset scores. For example:

79.8,55.1,50.6,37.6,57.1,54.0,4.1,24.6,77.1,4.0,44.3,44.4

Columns follow the order specified in --dataset. The repository README provides reference numbers for each benchmark to verify your reproduction. Higher values indicate better zero-shot transfer performance.

Supported Benchmarks

The dataloaders directory contains loaders for all standard benchmarks:

• CIFAR-10 and CIFAR-100
• Food-101
• Oxford-IIIT Pets
• Flowers-102
• SUN397
• Stanford Cars
• DTD (Describable Textures)
• Caltech-101
• FGVC Aircraft
• ImageNet (ILSVRC-2012)

Each dataset automatically uses its designated evaluation metric: top-1 accuracy, mean-per-class accuracy, or mean average precision (mAP).

Summary

• RWKV-CLIP evaluates zero-shot classification by encoding text prompts into class prototypes via zero_shot_classifier in zero_shot.py and comparing them against image embeddings from WarperCLIP_V_T_RWKV_method.
• The evaluation pipeline supports distributed multi-GPU execution using torch.distributed.all_gather_object to aggregate predictions before computing metrics.
• Eleven standard benchmarks are supported through the dataloaders module, with metrics automatically selected based on dataset conventions.
• Results are output as CSV files containing accuracy or mAP scores for each dataset, allowing direct comparison with published baseline numbers.

Frequently Asked Questions

What datasets are supported for zero-shot evaluation?

RWKV-CLIP supports eleven standard image classification benchmarks: CIFAR-10, CIFAR-100, Food-101, Oxford-IIIT Pets, Flowers-102, SUN397, Stanford Cars, DTD (Describable Textures), Caltech-101, FGVC Aircraft, and ImageNet. Each dataset loader resides in the dataloaders directory and exposes a get_loader_test() method used by the evaluation script.

How does RWKV-CLIP handle distributed multi-GPU evaluation?

The run function in zero_shot.py uses torch.distributed.all_gather_object to collect logits and targets from all GPU ranks before computing final metrics. Each rank processes a shard of the test dataset, stores partial results, and participates in the all-gather operation to ensure the final accuracy calculation considers the entire test set.

What metrics are computed during evaluation?

The evaluation script automatically selects metrics based on dataset flags: top-1 accuracy for most datasets, mean-per-class accuracy for datasets with imbalanced classes, mean Average Precision (mAP) for multi-label scenarios, and ROC-AUC where specified. These metrics are computed after aggregating distributed results and written to the output CSV file.

Where are the prompt templates stored for zero-shot classification?

Text prompt templates are stored in utils/template.json, while human-readable class names are defined in utils/label.json. During evaluation, zero_shot_classifier loads these files to generate prompts (e.g., "a photo of a {}") for each class, which are then encoded by WarperCLIP_V_T_RWKV_text_change_head to create the classification weight matrix.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}