# How to Implement Zero-Shot Image-Text Retrieval Using RWKV-CLIP Embeddings

> Implement zero-shot image-text retrieval with RWKV-CLIP. Encode images and text with RWKV backbones and compute cosine similarity for cross-modal matching without task-specific training.

- Repository: [DeepGlint/rwkv-clip](https://github.com/deepglint/rwkv-clip)
- Tags: how-to-guide
- Published: 2026-02-28

---

**Zero-shot image-text retrieval with RWKV-CLIP works by encoding images through an Image-RWKV vision backbone and text prompts through a Text-RWKV language backbone, then computing cosine similarity between their normalized embeddings to rank cross-modal matches without task-specific training.**

RWKV-CLIP is an open-source multimodal architecture from the `deepglint/rwkv-clip` repository that combines RWKV-based encoders with a CLIP-style projection head. This design enables **zero-shot image-text retrieval** by aligning vision and language embeddings in a shared latent space. In this guide, you will learn how to extract normalized embeddings using the provided utility functions and implement retrieval workflows using the [`text_image_retrieval.py`](https://github.com/deepglint/rwkv-clip/blob/main/text_image_retrieval.py) pipeline.

## Understanding the RWKV-CLIP Architecture

### Image-RWKV Vision Encoder

Located in [`model/Image_rwkv.py`](https://github.com/deepglint/rwkv-clip/blob/main/model/Image_rwkv.py), the `Image_RWKV` class processes images as sequences of patches. It begins with a patch embedding layer followed by learnable positional embeddings. The core computation happens through stacked **Block_V6** modules containing **VRWKV_SpatialMix_V6** and **VRWKV_ChannelMix_V6** blocks. These replace traditional self-attention with a recurrent-style WKV kernel (`WKV_6`), achieving linear time complexity while preserving spatial mixing across image patches.

### Text-RWKV Language Encoder

The language backbone in [`model/Text_rwkv.py`](https://github.com/deepglint/rwkv-clip/blob/main/model/Text_rwkv.py) implements `Text_RWKV` using token embeddings and stacked **Block** modules. Each block contains **RWKV_Tmix_V6** (temporal-mix) and **RWKV_CMix_V6** (channel-mix) layers that process token sequences bidirectionally using `WKV_6_bidirectional`. This architecture handles variable-length text inputs while maintaining the recurrent efficiency characteristic of RWKV models.

### CLIP-Style Projection Head

The [`model/utils.py`](https://github.com/deepglint/rwkv-clip/blob/main/model/utils.py) file defines the multimodal alignment mechanism through `WarperCLIP_V_T_RWKV_method` and `WarperCLIP_V_T_RWKV_text_change_head`. The `get_model` function wraps both encoders and adds a learnable `logit_scale` parameter. This head normalizes embeddings from both modalities into a shared space where cosine similarity directly measures cross-modal compatibility.

## Extracting RWKV-CLIP Embeddings

### Image Embedding Extraction

To extract image features, use the `WarperCLIP_V_T_RWKV_method` function in [`model/utils.py`](https://github.com/deepglint/rwkv-clip/blob/main/model/utils.py). The process differs based on the `image_cls_token` configuration:

- **Global Average Pooling** (default when `image_cls_token=False`): The vision encoder outputs a sequence of patch tokens, which are collapsed via `model.avg_layer` (lines 47-50 in [`utils.py`](https://github.com/deepglint/rwkv-clip/blob/main/utils.py)).
- **CLS Token** (when `image_cls_token=True`): The model extracts the first token directly as the global representation (lines 51-52).

Both approaches produce L2-normalized vectors before similarity computation.

### Text Embedding Extraction

For text, `WarperCLIP_V_T_RWKV_text_change_head` runs tokenized inputs through `model.text_model`. The implementation (lines 66-70 in [`utils.py`](https://github.com/deepglint/rwkv-clip/blob/main/utils.py)) first L2-normalizes token-wise outputs, then aggregates across the sequence using **adaptive-average-pool-1d** via `model.text_head`. This produces a fixed-dimensional embedding that is normalized again to unit length.

## Implementing Zero-Shot Retrieval

### Step 1: Load Pre-trained Weights

Initialize the model architecture using configuration arguments matching the checkpoint, then load weights:

```python
import torch
from model import get_model

# Initialize model with architecture parameters

model = get_model(image_encoder, text_encoder, 
                  image_cls_token=args.image_output_cls_token)
model.load_state_dict(torch.load(args.model_weight), strict=True)
model.eval().cuda()

```

### Step 2: Build Text Classifier

Create zero-shot classifiers using prompt templates from [`utils/template.json`](https://github.com/deepglint/rwkv-clip/blob/main/utils/template.json):

```python
from model.utils import tokenize, WarperCLIP_V_T_RWKV_text_change_head

classnames = ["cat", "dog", "car"]
templates = ["a photo of a {}.", "a blurry {}."]

# Format and tokenize prompts

texts = [t.format(c) for c in classnames for t in templates]
token_ids = tokenize(texts).cuda()

# Extract embeddings

text_emb = WarperCLIP_V_T_RWKV_text_change_head(model, token_ids)
text_emb = torch.nn.functional.normalize(text_emb, dim=-1)

# Average over templates per class

class_emb = text_emb.view(len(classnames), len(templates), -1).mean(dim=1)
class_emb = torch.nn.functional.normalize(class_emb, dim=-1)

```

### Step 3: Encode Image Gallery

Process images through the vision encoder:

```python
from model.utils import WarperCLIP_V_T_RWKV_method
from PIL import Image

# Preprocess image (using provided transforms)

img_tensor = preprocess(Image.open('sample.jpg')).unsqueeze(0).cuda()

# Extract and normalize

img_emb = WarperCLIP_V_T_RWKV_method(model, img_tensor)
img_emb = torch.nn.functional.normalize(img_emb, dim=-1)

```

### Step 4: Compute Similarity and Rank

Calculate cosine similarity between image and text embeddings:

```python

# Similarity matrix (num_images, num_classes)

similarity = img_emb @ class_emb.T

# Top-k retrieval

values, indices = similarity.topk(k=5, dim=-1)
predictions = [classnames[i] for i in indices[0]]

```

The [`text_image_retrieval.py`](https://github.com/deepglint/rwkv-clip/blob/main/text_image_retrieval.py) script automates this workflow and reports **R@1, R@5, and R@10** for both text-to-image (t2i) and image-to-text (i2t) directions (lines 90-125).

## Code Examples

### Running Zero-Shot Retrieval from Command Line

Execute the full pipeline on standard datasets:

```bash
python text_image_retrieval.py \
    --batch-size 128 \
    --dataset flickr \
    --model-weight path/to/rwkv_clip_weights.pth \
    --input-size 224 \
    --image-depth 12 \
    --image-embed-dims 384 \
    --image-patch-size 16 \
    --image-hidden-rate 4 \
    --image-num-heads 6 \
    --image-output-cls-token False \
    --image-with-cls-token False

```

The script outputs retrieval metrics:

```

Text retrieval {'r1': 42.3, 'r5': 71.8, 'r10': 83.5}
Image retrieval {'r1': 39.7, 'r5': 68.2, 'r10': 80.1}

```

### Integrating into Custom Pipelines

For existing data loaders, replace feature extraction with the RWKV-CLIP helpers:

```python
def encode_batch(images, texts, model):
    """Extract normalized embeddings for retrieval indexing."""
    img_feat = WarperCLIP_V_T_RWKV_method(model, images)
    txt_feat = WarperCLIP_V_T_RWKV_text_change_head(model, texts)
    
    return (
        torch.nn.functional.normalize(img_feat, dim=-1),
        torch.nn.functional.normalize(txt_feat, dim=-1)
    )

```

These embeddings can be indexed in FAISS or Annoy for million-scale retrieval.

## Summary

- **RWKV-CLIP** combines `Image_RWKV` and `Text_RWKV` encoders with a CLIP-style projection head to enable **zero-shot image-text retrieval**.
- **Image embeddings** are extracted via `WarperCLIP_V_T_RWKV_method` in [`model/utils.py`](https://github.com/deepglint/rwkv-clip/blob/main/model/utils.py), using either global average pooling of patch tokens or a CLS token.
- **Text embeddings** are generated by `WarperCLIP_V_T_RWKV_text_change_head`, which applies adaptive average pooling over L2-normalized token sequences.
- The **retrieval workflow** involves encoding gallery images and text prompts, computing cosine similarity, and ranking matches—automated in [`text_image_retrieval.py`](https://github.com/deepglint/rwkv-clip/blob/main/text_image_retrieval.py) with Recall@K metrics.
- RWKV's **linear-complexity recurrent kernels** (spatial-mix for vision, temporal-mix for text) provide efficient alternatives to quadratic self-attention while maintaining cross-modal alignment.

## Frequently Asked Questions

### How does RWKV-CLIP differ from standard CLIP models?

RWKV-CLIP replaces the traditional Transformer self-attention mechanisms in both vision and language encoders with **RWKV blocks**, which utilize recurrent-style WKV kernels. According to the `deepglint/rwkv-clip` source code, the vision encoder uses **VRWKV_SpatialMix_V6** blocks (linear complexity over patches) while the text encoder uses **RWKV_Tmix_V6** blocks (temporal mixing over tokens). This architecture reduces computational complexity from quadratic to linear while preserving the cross-modal alignment capabilities of CLIP-style contrastive learning.

### What is the difference between using a CLS token and global average pooling for image embeddings?

In [`model/utils.py`](https://github.com/deepglint/rwkv-clip/blob/main/model/utils.py), the `WarperCLIP_V_T_RWKV_method` function handles two configurations based on the `image_cls_token` argument. When `image_cls_token=False` (default), the vision encoder outputs a sequence of patch tokens, which are collapsed via **global average pooling** (`model.avg_layer`) to produce a single vector representation (lines 47-50). When `image_cls_token=True`, the model extracts the first token directly as the global representation (lines 51-52). Both approaches produce L2-normalized vectors, with global average pooling generally providing more robust features for retrieval tasks.

### How do I prepare text prompts for zero-shot classification?

Zero-shot retrieval requires formatting class names into descriptive prompts using templates from [`utils/template.json`](https://github.com/deepglint/rwkv-clip/blob/main/utils/template.json). For each class name in your dataset, create multiple prompt variations (e.g., "a photo of a {}.", "a blurry {}.") and tokenize them using the `tokenize` function. Feed these token IDs through `WarperCLIP_V_T_RWKV_text_change_head` to obtain embeddings, then average the embeddings across all templates for each class and L2-normalize the result. This process, implemented in [`text_image_retrieval.py`](https://github.com/deepglint/rwkv-clip/blob/main/text_image_retrieval.py), creates robust class prototypes that improve retrieval accuracy by capturing diverse linguistic descriptions of each category.

### Can I use RWKV-CLIP for large-scale retrieval with millions of images?

Yes, the RWKV-CLIP architecture supports large-scale retrieval through its efficient embedding extraction and compatibility with approximate nearest neighbor libraries. Since both `WarperCLIP_V_T_RWKV_method` and `WarperCLIP_V_T_RWKV_text_change_head` produce fixed-dimensional, L2-normalized embeddings, you can index image galleries using FAISS, Annoy, or ScaNN. The linear complexity of RWKV blocks (spatial-mix for vision, temporal-mix for text) makes encoding large batches more efficient than quadratic-attention transformers. For million-scale datasets, extract embeddings in batches using the provided API, then build an index that supports cosine similarity search for sub-millisecond retrieval latency.