How to Implement Zero-Shot Image-Text Retrieval Using RWKV-CLIP Embeddings

Zero-shot image-text retrieval with RWKV-CLIP works by encoding images through an Image-RWKV vision backbone and text prompts through a Text-RWKV language backbone, then computing cosine similarity between their normalized embeddings to rank cross-modal matches without task-specific training.

RWKV-CLIP is an open-source multimodal architecture from the deepglint/rwkv-clip repository that combines RWKV-based encoders with a CLIP-style projection head. This design enables zero-shot image-text retrieval by aligning vision and language embeddings in a shared latent space. In this guide, you will learn how to extract normalized embeddings using the provided utility functions and implement retrieval workflows using the text_image_retrieval.py pipeline.

Understanding the RWKV-CLIP Architecture

Image-RWKV Vision Encoder

Located in model/Image_rwkv.py, the Image_RWKV class processes images as sequences of patches. It begins with a patch embedding layer followed by learnable positional embeddings. The core computation happens through stacked Block_V6 modules containing VRWKV_SpatialMix_V6 and VRWKV_ChannelMix_V6 blocks. These replace traditional self-attention with a recurrent-style WKV kernel (WKV_6), achieving linear time complexity while preserving spatial mixing across image patches.

Text-RWKV Language Encoder

The language backbone in model/Text_rwkv.py implements Text_RWKV using token embeddings and stacked Block modules. Each block contains RWKV_Tmix_V6 (temporal-mix) and RWKV_CMix_V6 (channel-mix) layers that process token sequences bidirectionally using WKV_6_bidirectional. This architecture handles variable-length text inputs while maintaining the recurrent efficiency characteristic of RWKV models.

CLIP-Style Projection Head

The model/utils.py file defines the multimodal alignment mechanism through WarperCLIP_V_T_RWKV_method and WarperCLIP_V_T_RWKV_text_change_head. The get_model function wraps both encoders and adds a learnable logit_scale parameter. This head normalizes embeddings from both modalities into a shared space where cosine similarity directly measures cross-modal compatibility.

Extracting RWKV-CLIP Embeddings

Image Embedding Extraction

To extract image features, use the WarperCLIP_V_T_RWKV_method function in model/utils.py. The process differs based on the image_cls_token configuration:

• Global Average Pooling (default when image_cls_token=False): The vision encoder outputs a sequence of patch tokens, which are collapsed via model.avg_layer (lines 47-50 in utils.py).
• CLS Token (when image_cls_token=True): The model extracts the first token directly as the global representation (lines 51-52).

Both approaches produce L2-normalized vectors before similarity computation.

Text Embedding Extraction

For text, WarperCLIP_V_T_RWKV_text_change_head runs tokenized inputs through model.text_model. The implementation (lines 66-70 in utils.py) first L2-normalizes token-wise outputs, then aggregates across the sequence using adaptive-average-pool-1d via model.text_head. This produces a fixed-dimensional embedding that is normalized again to unit length.

Implementing Zero-Shot Retrieval

Step 1: Load Pre-trained Weights

Initialize the model architecture using configuration arguments matching the checkpoint, then load weights:

import torch
from model import get_model

# Initialize model with architecture parameters

model = get_model(image_encoder, text_encoder, 
                  image_cls_token=args.image_output_cls_token)
model.load_state_dict(torch.load(args.model_weight), strict=True)
model.eval().cuda()

Step 2: Build Text Classifier

Create zero-shot classifiers using prompt templates from utils/template.json:

from model.utils import tokenize, WarperCLIP_V_T_RWKV_text_change_head

classnames = ["cat", "dog", "car"]
templates = ["a photo of a {}.", "a blurry {}."]

# Format and tokenize prompts

texts = [t.format(c) for c in classnames for t in templates]
token_ids = tokenize(texts).cuda()

# Extract embeddings

text_emb = WarperCLIP_V_T_RWKV_text_change_head(model, token_ids)
text_emb = torch.nn.functional.normalize(text_emb, dim=-1)

# Average over templates per class

class_emb = text_emb.view(len(classnames), len(templates), -1).mean(dim=1)
class_emb = torch.nn.functional.normalize(class_emb, dim=-1)

Step 3: Encode Image Gallery

Process images through the vision encoder:

from model.utils import WarperCLIP_V_T_RWKV_method
from PIL import Image

# Preprocess image (using provided transforms)

img_tensor = preprocess(Image.open('sample.jpg')).unsqueeze(0).cuda()

# Extract and normalize

img_emb = WarperCLIP_V_T_RWKV_method(model, img_tensor)
img_emb = torch.nn.functional.normalize(img_emb, dim=-1)

Step 4: Compute Similarity and Rank

Calculate cosine similarity between image and text embeddings:

# Similarity matrix (num_images, num_classes)

similarity = img_emb @ class_emb.T

# Top-k retrieval

values, indices = similarity.topk(k=5, dim=-1)
predictions = [classnames[i] for i in indices[0]]

The text_image_retrieval.py script automates this workflow and reports R@1, R@5, and R@10 for both text-to-image (t2i) and image-to-text (i2t) directions (lines 90-125).

Code Examples

Running Zero-Shot Retrieval from Command Line

Execute the full pipeline on standard datasets:

python text_image_retrieval.py \
    --batch-size 128 \
    --dataset flickr \
    --model-weight path/to/rwkv_clip_weights.pth \
    --input-size 224 \
    --image-depth 12 \
    --image-embed-dims 384 \
    --image-patch-size 16 \
    --image-hidden-rate 4 \
    --image-num-heads 6 \
    --image-output-cls-token False \
    --image-with-cls-token False

The script outputs retrieval metrics:

Text retrieval {'r1': 42.3, 'r5': 71.8, 'r10': 83.5}
Image retrieval {'r1': 39.7, 'r5': 68.2, 'r10': 80.1}

Integrating into Custom Pipelines

For existing data loaders, replace feature extraction with the RWKV-CLIP helpers:

def encode_batch(images, texts, model):
    """Extract normalized embeddings for retrieval indexing."""
    img_feat = WarperCLIP_V_T_RWKV_method(model, images)
    txt_feat = WarperCLIP_V_T_RWKV_text_change_head(model, texts)
    
    return (
        torch.nn.functional.normalize(img_feat, dim=-1),
        torch.nn.functional.normalize(txt_feat, dim=-1)
    )

These embeddings can be indexed in FAISS or Annoy for million-scale retrieval.

Summary

• RWKV-CLIP combines Image_RWKV and Text_RWKV encoders with a CLIP-style projection head to enable zero-shot image-text retrieval.
• Image embeddings are extracted via WarperCLIP_V_T_RWKV_method in model/utils.py, using either global average pooling of patch tokens or a CLS token.
• Text embeddings are generated by WarperCLIP_V_T_RWKV_text_change_head, which applies adaptive average pooling over L2-normalized token sequences.
• The retrieval workflow involves encoding gallery images and text prompts, computing cosine similarity, and ranking matches—automated in text_image_retrieval.py with Recall@K metrics.
• RWKV's linear-complexity recurrent kernels (spatial-mix for vision, temporal-mix for text) provide efficient alternatives to quadratic self-attention while maintaining cross-modal alignment.

Frequently Asked Questions

How does RWKV-CLIP differ from standard CLIP models?

RWKV-CLIP replaces the traditional Transformer self-attention mechanisms in both vision and language encoders with RWKV blocks, which utilize recurrent-style WKV kernels. According to the deepglint/rwkv-clip source code, the vision encoder uses VRWKV_SpatialMix_V6 blocks (linear complexity over patches) while the text encoder uses RWKV_Tmix_V6 blocks (temporal mixing over tokens). This architecture reduces computational complexity from quadratic to linear while preserving the cross-modal alignment capabilities of CLIP-style contrastive learning.

What is the difference between using a CLS token and global average pooling for image embeddings?

In model/utils.py, the WarperCLIP_V_T_RWKV_method function handles two configurations based on the image_cls_token argument. When image_cls_token=False (default), the vision encoder outputs a sequence of patch tokens, which are collapsed via global average pooling (model.avg_layer) to produce a single vector representation (lines 47-50). When image_cls_token=True, the model extracts the first token directly as the global representation (lines 51-52). Both approaches produce L2-normalized vectors, with global average pooling generally providing more robust features for retrieval tasks.

How do I prepare text prompts for zero-shot classification?

Zero-shot retrieval requires formatting class names into descriptive prompts using templates from utils/template.json. For each class name in your dataset, create multiple prompt variations (e.g., "a photo of a {}.", "a blurry {}.") and tokenize them using the tokenize function. Feed these token IDs through WarperCLIP_V_T_RWKV_text_change_head to obtain embeddings, then average the embeddings across all templates for each class and L2-normalize the result. This process, implemented in text_image_retrieval.py, creates robust class prototypes that improve retrieval accuracy by capturing diverse linguistic descriptions of each category.

Can I use RWKV-CLIP for large-scale retrieval with millions of images?

Yes, the RWKV-CLIP architecture supports large-scale retrieval through its efficient embedding extraction and compatibility with approximate nearest neighbor libraries. Since both WarperCLIP_V_T_RWKV_method and WarperCLIP_V_T_RWKV_text_change_head produce fixed-dimensional, L2-normalized embeddings, you can index image galleries using FAISS, Annoy, or ScaNN. The linear complexity of RWKV blocks (spatial-mix for vision, temporal-mix for text) makes encoding large batches more efficient than quadratic-attention transformers. For million-scale datasets, extract embeddings in batches using the provided API, then build an index that supports cosine similarity search for sub-millisecond retrieval latency.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}