# How to Evaluate RWKV-CLIP on Standard Zero-Shot Classification Benchmarks

> Evaluate RWKV-CLIP on zero-shot classification benchmarks. Build text prototypes and run inference to compute top-1 accuracy or mAP on ImageNet and CIFAR datasets.

- Repository: [DeepGlint/rwkv-clip](https://github.com/deepglint/rwkv-clip)
- Tags: how-to-guide
- Published: 2026-02-28

---

**Evaluate RWKV-CLIP by building text prototypes with `zero_shot_classifier` in [`zero_shot.py`](https://github.com/deepglint/rwkv-clip/blob/main/zero_shot.py), running inference with `WarperCLIP_V_T_RWKV_method`, and aggregating distributed results to compute top-1 accuracy or mAP on datasets like ImageNet, CIFAR-10, and CIFAR-100.**

The `deepglint/rwkv-clip` repository provides a complete evaluation pipeline for assessing RWKV-CLIP's zero-shot classification capabilities without fine-tuning. This guide explains how to run standard benchmarks using the provided scripts and APIs.

## Zero-Shot Evaluation Pipeline Overview

The evaluation process follows three distinct stages implemented in [`zero_shot.py`](https://github.com/deepglint/rwkv-clip/blob/main/zero_shot.py):

1. **Prepare a zero-shot classifier** – For each target dataset, the model constructs class prototypes by encoding textual prompts. The `zero_shot_classifier` function calls `WarperCLIP_V_T_RWKV_text_change_head` from [`model/utils.py`](https://github.com/deepglint/rwkv-clip/blob/main/model/utils.py) to generate normalized text embeddings.

2. **Run inference on the image test split** – Images are processed through the vision backbone (`WarperCLIP_V_T_RWKV_method`), normalized, and compared against class prototypes to obtain logits. The core inference loop resides in the `run` function of [`zero_shot.py`](https://github.com/deepglint/rwkv-clip/blob/main/zero_shot.py).

3. **Aggregate results across distributed ranks** – When using multi-GPU setups, each rank stores partial logits and targets, which are collected via `torch.distributed.all_gather_object`. Final metrics (top-1 accuracy, mean-per-class accuracy, mAP, or ROC-AUC) are computed based on dataset-specific flags.

## Step-by-Step Implementation Guide

### Configuration and Model Loading

The evaluation script begins by loading dataset configurations and instantiating the model. Text prompts and class names are loaded from [`utils/template.json`](https://github.com/deepglint/rwkv-clip/blob/main/utils/template.json) and [`utils/label.json`](https://github.com/deepglint/rwkv-clip/blob/main/utils/label.json), while the model architecture is defined in [`model_config/RWKV_CLIP_B32.json`](https://github.com/deepglint/rwkv-clip/blob/main/model_config/RWKV_CLIP_B32.json).

```python
import json
import torch
from model import create_RWKV_Model
from model.utils import WarperCLIP_V_T_RWKV_text_change_head
from open_clip.transform import image_transform
from PIL import Image

# Load configuration

args = json.load(open('model_config/RWKV_CLIP_B32.json'))
model = create_RWKV_Model(args, model_weight_path='RWKV_CLIP_B32_YFCC15M.pt')
model.eval().cuda()

```

### Building the Zero-Shot Classifier

The classifier matrix is constructed by encoding prompt templates for each class. For CIFAR-10, the system loads templates from [`utils/template.json`](https://github.com/deepglint/rwkv-clip/blob/main/utils/template.json) and class names from [`utils/label.json`](https://github.com/deepglint/rwkv-clip/blob/main/utils/label.json), then averages the embeddings for each class.

```python

# Prepare image transform

transform = image_transform(args['input_size'], False)

# Build zero-shot classifier for CIFAR-10

with open('utils/label.json') as f:
    labels = json.load(f)['cifar10']
with open('utils/template.json') as f:
    templates = json.load(f)['cifar10']

classifier = []
for classname in labels:
    texts = [tpl.format(classname) for tpl in templates]
    tokens = tokenize(texts).cuda()
    txt_emb = WarperCLIP_V_T_RWKV_text_change_head(model, tokens)
    cls_emb = torch.nn.functional.normalize(txt_emb, dim=-1).mean(0)
    classifier.append(cls_emb)

classifier = torch.stack(classifier).cuda()  # [num_classes, embed_dim]

```

### Inference and Metric Aggregation

During evaluation, images are encoded using `WarperCLIP_V_T_RWKV_method` and compared against the classifier matrix. The `run` function in [`zero_shot.py`](https://github.com/deepglint/rwkv-clip/blob/main/zero_shot.py) handles distributed gathering of predictions and computes dataset-specific metrics.

```python

# Inference loop (simplified)

with torch.no_grad():
    for images, targets in dataloader:
        images = images.cuda()
        image_features = WarperCLIP_V_T_RWKV_method(model, images)
        image_features = torch.nn.functional.normalize(image_features, dim=-1)
        logits = 100. * image_features @ classifier.T
        # Store logits and targets for metric computation

```

For distributed evaluation, the script uses `torch.distributed.all_gather_object` to collect tensors from all ranks before computing final accuracy or mAP scores.

## Running the Evaluation

### Command-Line Evaluation (Distributed)

The repository provides a shell script to launch distributed evaluation across multiple GPUs. The script [`shell/test_zero_shot_classificaiton.sh`](https://github.com/deepglint/rwkv-clip/blob/main/shell/test_zero_shot_classificaiton.sh) uses `torchrun` to manage processes.

```bash

# Launch evaluation on 8 GPUs

bash shell/test_zero_shot_classificaiton.sh \
    --model-weight path/to/RWKV_CLIP_B32_YFCC15M.pt \
    --dataset imagenet,cifar10,cifar100,food101,oxford_pets \
    --output-dir ./zero_shot_results.csv

```

Key arguments:
- `--model-weight`: Path to the pretrained checkpoint (e.g., `RWKV_CLIP_B32_YFCC15M.pt`)
- `--dataset`: Comma-separated list of dataset names
- `--output-dir`: File path for writing results (appended as CSV)

### Python API for Custom Evaluation

For notebook-based experimentation, import the evaluation components directly from [`zero_shot.py`](https://github.com/deepglint/rwkv-clip/blob/main/zero_shot.py) and [`model/utils.py`](https://github.com/deepglint/rwkv-clip/blob/main/model/utils.py). This approach allows custom dataset integration without modifying the shell scripts.

```python
from zero_shot import run, zero_shot_classifier
import torch

# Configure your model and datasets

model = create_RWKV_Model(args, model_weight_path='checkpoint.pt')
classifier = zero_shot_classifier(model, classnames, templates)

# Run evaluation

results = run(model, classifier, dataloader, args)

```

### Interpreting Output Results

The evaluation script writes results in CSV format where each line corresponds to a model checkpoint and columns represent dataset scores. For example:

```

79.8,55.1,50.6,37.6,57.1,54.0,4.1,24.6,77.1,4.0,44.3,44.4

```

Columns follow the order specified in `--dataset`. The repository README provides reference numbers for each benchmark to verify your reproduction. Higher values indicate better zero-shot transfer performance.

## Supported Benchmarks

The `dataloaders` directory contains loaders for all standard benchmarks:

- **CIFAR-10** and **CIFAR-100**
- **Food-101**
- **Oxford-IIIT Pets**
- **Flowers-102**
- **SUN397**
- **Stanford Cars**
- **DTD** (Describable Textures)
- **Caltech-101**
- **FGVC Aircraft**
- **ImageNet** (ILSVRC-2012)

Each dataset automatically uses its designated evaluation metric: top-1 accuracy, mean-per-class accuracy, or mean average precision (mAP).

## Summary

- **RWKV-CLIP** evaluates zero-shot classification by encoding text prompts into class prototypes via `zero_shot_classifier` in [`zero_shot.py`](https://github.com/deepglint/rwkv-clip/blob/main/zero_shot.py) and comparing them against image embeddings from `WarperCLIP_V_T_RWKV_method`.
- The evaluation pipeline supports **distributed multi-GPU** execution using `torch.distributed.all_gather_object` to aggregate predictions before computing metrics.
- **Eleven standard benchmarks** are supported through the `dataloaders` module, with metrics automatically selected based on dataset conventions.
- Results are output as **CSV files** containing accuracy or mAP scores for each dataset, allowing direct comparison with published baseline numbers.

## Frequently Asked Questions

### What datasets are supported for zero-shot evaluation?

RWKV-CLIP supports eleven standard image classification benchmarks: CIFAR-10, CIFAR-100, Food-101, Oxford-IIIT Pets, Flowers-102, SUN397, Stanford Cars, DTD (Describable Textures), Caltech-101, FGVC Aircraft, and ImageNet. Each dataset loader resides in the `dataloaders` directory and exposes a `get_loader_test()` method used by the evaluation script.

### How does RWKV-CLIP handle distributed multi-GPU evaluation?

The `run` function in [`zero_shot.py`](https://github.com/deepglint/rwkv-clip/blob/main/zero_shot.py) uses `torch.distributed.all_gather_object` to collect logits and targets from all GPU ranks before computing final metrics. Each rank processes a shard of the test dataset, stores partial results, and participates in the all-gather operation to ensure the final accuracy calculation considers the entire test set.

### What metrics are computed during evaluation?

The evaluation script automatically selects metrics based on dataset flags: **top-1 accuracy** for most datasets, **mean-per-class accuracy** for datasets with imbalanced classes, **mean Average Precision (mAP)** for multi-label scenarios, and **ROC-AUC** where specified. These metrics are computed after aggregating distributed results and written to the output CSV file.

### Where are the prompt templates stored for zero-shot classification?

Text prompt templates are stored in [`utils/template.json`](https://github.com/deepglint/rwkv-clip/blob/main/utils/template.json), while human-readable class names are defined in [`utils/label.json`](https://github.com/deepglint/rwkv-clip/blob/main/utils/label.json). During evaluation, `zero_shot_classifier` loads these files to generate prompts (e.g., "a photo of a {}") for each class, which are then encoded by `WarperCLIP_V_T_RWKV_text_change_head` to create the classification weight matrix.