How to Perform Zero-Shot Image Classification with RWKV-CLIP on Custom Datasets

Zero-shot image classification with RWKV-CLIP works by encoding images through a Vision-RWKV backbone and textual class prompts through a Text-RWKV backbone, then computing similarity in a shared embedding space to predict labels without task-specific training.

RWKV-CLIP is an open-source multimodal model developed by DeepGlint that combines Vision-RWKV and Text-RWKV architectures to align image and text representations. Unlike traditional supervised learning, zero-shot classification allows you to categorize images into custom classes by simply providing text descriptions, eliminating the need for labeled training data. This guide walks you through the complete workflow using the deepglint/rwkv-clip repository.

Understanding the RWKV-CLIP Architecture for Zero-Shot Classification

RWKV-CLIP utilizes two parallel encoders to map images and text into a shared dimensional space. The Vision-RWKV backbone processes image patches through RWKV blocks (implemented in model/Image_rwkv.py), while the Text-RWKV backbone processes tokenized prompts (implemented in model/Text_rwkv.py).

Zero-shot classification leverages prompt templates to convert class names into rich textual descriptions. For each class, the model generates embeddings for multiple prompt variations (e.g., "a photo of a {class}", "a blurry photo of a {class}"), averages them to create robust class prototypes, and stores these as columns in a classifier weight matrix.

Preparing Your Custom Dataset for RWKV-CLIP

To run zero-shot classification on custom categories, you must provide two JSON configuration files that map dataset identifiers to class labels and prompt templates.

Creating the Label Mapping File (label.json)

The label file defines your class taxonomy. Create a JSON file where each key represents a dataset identifier and the value is a list of class names.

{
  "my_custom": ["cat", "dog", "car", "tree", "house"]
}

Store this as utils/label.json or specify a custom path via command-line arguments. The class names should be descriptive nouns that work naturally in sentence templates.

Defining Prompt Templates (template.json)

Prompt templates wrap class names into complete sentences to provide context for the text encoder. Each template should contain a {} placeholder where the class name inserts.

{
  "my_custom": [
    "a photo of a {}.",
    "a blurry photo of a {}.",
    "a black and white photo of a {}.",
    "a bright photo of a {}.",
    "a dark photo of a {}."
  ]
}

Multiple templates per class improve robustness by capturing different visual contexts. The zero_shot_classifier function in zero_shot.py averages embeddings across all templates for each class.

Building the Zero-Shot Classifier

The core logic for constructing the classifier resides in zero_shot.py. The function zero_shot_classifier (lines 59-73) tokenizes prompts, runs them through the Text-RWKV backbone, and normalizes embeddings.

import torch
import json
from model.utils import create_RWKV_Model
from zero_shot import zero_shot_classifier

# Initialize model with architecture parameters

args = {
    'input_size': 224,
    'image_patch_size': 16,
    'image_embed_dims': 384,
    'image_hidden_rate': 4,
    'image_depth': 12,
    'image_num_heads': 6,
    'image_output_cls_token': False,
    'image_with_cls_token': False,
    # Text model parameters

    'vocab_size': 49408,
    'n_embd': 384,
    'ctx_len': 77,
    'dim_att': 384,
    'dim_ffn': 384,
    'head_size': 64,
    'head_size_divisor': 8,
    'n_layer': 12,
    'dropout': 0.0,
    'text_initialization': True
}

model = create_RWKV_Model(args, model_weight_path='rwkv_clip.pth')
model.eval().cuda()

# Load custom dataset configuration

with open('utils/label.json') as f:
    class_names = json.load(f)['my_custom']
with open('utils/template.json') as f:
    templates = json.load(f)['my_custom']

# Build classifier matrix: shape [embed_dim, num_classes]

classifier = zero_shot_classifier(model, class_names, templates, args)
classifier = classifier.cuda()

The resulting classifier tensor contains normalized embeddings where each column represents a class prototype in the shared embedding space.

Running Inference on Custom Images

Once the classifier is built, you can classify new images by encoding them through the Vision-RWKV backbone and computing similarity scores.

Command-Line Interface

The repository provides zero_shot.py as a standalone script for quick evaluation:

python zero_shot.py \
  --model-weight path/to/rwkv_clip.pth \
  --output-dir results.txt \
  --batch-size 128 \
  --dataset my_custom \
  --input-size 224 \
  --image-patch-size 16 \
  --image-embed-dims 384 \
  --image-depth 12 \
  --image-num-heads 6 \
  --image-output-cls-token False \
  --image-with-cls-token False

Ensure that my_custom exists as a key in both utils/label.json and utils/template.json.

Python API for Custom Pipelines

For integration into existing workflows, use the WarperCLIP_V_T_RWKV_method function from model/utils.py to encode images:

import torch
import torch.nn.functional as F
from model.utils import WarperCLIP_V_T_RWKV_method

# Preprocess images to tensor shape [B, C, H, W]

images = preprocess_batch(pil_images).cuda()

# Extract image embeddings

image_features = WarperCLIP_V_T_RWKV_method(model, images)  # [B, embed_dim]

# Normalize and compute similarity

image_features = F.normalize(image_features, dim=-1)
logits = 100.0 * image_features @ classifier  # [B, num_classes]

predictions = logits.argmax(dim=1)

# Map indices to class names

predicted_classes = [class_names[idx] for idx in predictions]

The temperature scaling factor of 100.0 matches the CLIP training objective and ensures logits are in an appropriate range for softmax or argmax operations.

Summary

• RWKV-CLIP combines Vision-RWKV and Text-RWKV backbones to create aligned image-text embeddings without traditional attention mechanisms.
• Custom datasets require two JSON files: label.json for class names and template.json for prompt templates, stored in the utils/ directory.
• Classifier construction uses zero_shot_classifier in zero_shot.py to average text embeddings across multiple prompt templates per class.
• Image encoding is handled by WarperCLIP_V_T_RWKV_method in model/utils.py, which processes patches through the Vision-RWKV backbone.
• Inference computes dot-product similarity between normalized image embeddings and the classifier matrix, with logits scaled by 100.0 for consistency with CLIP training.

Frequently Asked Questions

What is the difference between RWKV-CLIP and standard CLIP models?

RWKV-CLIP replaces the standard Transformer attention mechanisms in both vision and text encoders with RWKV (Receptance Weighted Key Value) linear attention blocks. According to the source code in model/Image_rwkv.py and model/Text_rwkv.py, this reduces computational complexity from quadratic to linear with respect to sequence length while maintaining the contrastive learning objective that aligns image and text embeddings in a shared space.

How do I format custom prompt templates for better zero-shot accuracy?

Prompt templates should be stored in utils/template.json as a list of strings containing {} placeholders where class names insert. For optimal zero-shot performance, include diverse linguistic variations that capture different visual contexts—such as "a photo of a {}", "a blurry photo of a {}", "a black and white photo of a {}", and "a bright photo of a {}". The zero_shot_classifier function averages embeddings across all templates per class, so variety improves robustness to domain shifts.

Can I use RWKV-CLIP for few-shot learning instead of zero-shot?

While the repository is optimized for zero-shot classification via zero_shot.py, you can adapt the architecture for few-shot learning by freezing the RWKV-CLIP encoders and training a lightweight linear probe or adapter on top of the extracted embeddings. Use WarperCLIP_V_T_RWKV_method to generate image features and the text encoder to generate prototype embeddings for novel classes with limited examples, then compute nearest-neighbor or logistic regression classifiers in the embedding space.

What hardware requirements are needed to run RWKV-CLIP inference?

The repository supports CUDA-enabled GPUs for inference. Based on the model configurations in model/utils.py, the default architecture uses 384-dimensional embeddings with 12 layers for both vision and text encoders. For batch processing with zero_shot.py, a GPU with at least 8GB VRAM is recommended for batch sizes of 128 at 224×224 resolution. The model weights are loaded via load_model_weight in model/utils.py, which supports standard PyTorch checkpoint formats. CPU inference is possible but significantly slower due to the linear attention computations in the RWKV blocks.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}