# Architectural Differences Between VisionRWKV and RWKV-CLIP: A Deep Dive into the RWKV-CLIP Repository

> Explore the architectural differences between VisionRWKV and RWKV-CLIP. Understand how VisionRWKV adapts ViT self-attention with RWKV layers for contrastive learning.

- Repository: [DeepGlint/rwkv-clip](https://github.com/deepglint/rwkv-clip)
- Tags: deep-dive
- Published: 2026-02-28

---

**VisionRWKV is a pure visual backbone that replaces standard ViT self-attention with recurrent RWKV layers, while RWKV-CLIP combines VisionRWKV with a text encoder, pooling heads, and a learnable temperature to create a complete CLIP-style dual-encoder for contrastive learning.**

The `deepglint/rwkv-clip` repository implements a novel vision-language architecture using the RWKV (Receptance Weighted Key Value) mechanism. Understanding the architectural differences between VisionRWKV and RWKV-CLIP is essential for researchers adapting these models for downstream tasks, as the former serves as a modular component while the latter represents a complete training system.

## What Is VisionRWKV? The Pure Visual Backbone

VisionRWKV functions as a standalone visual encoder that processes images through a stack of recurrent-attention blocks. According to the source code in [`model/Image_rwkv.py`](https://github.com/deepglint/rwkv-clip/blob/main/model/Image_rwkv.py), it mirrors a Vision Transformer (ViT) structure but replaces the quadratic self-attention mechanism with an efficient linear-complexity RWKV kernel.

### Input Processing and Patch Embedding

The architecture begins with **PatchEmbed** from MMCV, which splits input images into non-overlapping patches of shape `[B, N, C]` where `N = H·W` represents the total number of patches. In the constructor of `Image_RWKV` at `model/Image_rwkv.py#L53-L99`, the system initializes a learnable **positional embedding** (`pos_embed`) of shape `[1, num_patches, embed_dims]` that is dynamically resized to the current resolution using `resize_pos_embed` (see `model/Image_rwkv.py#L50-L56`).

### Core Architecture: SpatialMix and ChannelMix Blocks

Each transformer block (`Block_V6` defined at `model/Image_rwkv.py#L99-L151`) contains two critical components:

- **VRWKV_SpatialMix_V6**: Implements spatial mixing through the "q-shift" trick that shifts patches across spatial dimensions before applying the RWKV recurrent kernel. The shift operation is performed by `q_shift_multihead` (see `model/Image_rwkv.py#L79-L99`).
- **VRWKV_ChannelMix_V6**: A lightweight MLP that mixes channel information using linear layers and ReLU-squared activation (see `model/Image_rwkv.py#L124-L152`).

The stack includes LayerNorm layers, optional pre-norm/post-norm configurations, and stochastic depth (`DropPath`) for regularization.

### The RWKV Recurrent Mechanism

Unlike standard transformers, VisionRWKV processes spatial information through a custom CUDA kernel `wkv6_image` implemented in [`model/cuda_image/wkv6_op.cpp`](https://github.com/deepglint/rwkv-clip/blob/main/model/cuda_image/wkv6_op.cpp). This kernel executes the RWKV recurrence—a time-mixing operation that computes attention weights using receptance, key, and value states with linear complexity relative to sequence length.

### Output Formats and CLS Token Support

VisionRWKV supports an optional learnable **CLS token** controlled by the `with_cls_token` flag. When `output_cls_token=True`, the forward pass returns both patch features and the classification token as a tuple `[patch_token, cls_token]`. By default, the model returns a 4-D tensor of shape `[B, C, H, W]` representing the final patch feature map, suitable for dense prediction tasks.

## What Is RWKV-CLIP? The Dual-Encoder System

RWKV-CLIP represents the complete architecture that wraps VisionRWKV into a contrastive learning framework. Defined primarily in [`model/utils.py`](https://github.com/deepglint/rwkv-clip/blob/main/model/utils.py), this system combines visual and textual encoders with projection heads to produce normalized embeddings for image-text matching.

### Composition of the Two Towers

The `get_model` class at `model/utils.py#L75-L90` instantiates two distinct backbones:

1. **Image_RWKV**: The VisionRWKV backbone described above.
2. **Text_RWKV**: A separate RWKV-based language model (defined in [`model/Text_rwkv.py`](https://github.com/deepglint/rwkv-clip/blob/main/model/Text_rwkv.py)) that processes token sequences using time-mix and channel-mix operations with bidirectional shift mechanisms.

The factory function `get_model_RWKV_CLIP` in `train.py#L18-L32` orchestrates the construction of both components, passing critical flags such as `with_cls_token` to configure the visual backbone.

### Pooling and Projection Heads

RWKV-CLIP adds several architectural layers not present in the base VisionRWKV:

- **Image Head**: The `WarperCLIP_V_T_RWKV_method` applies `GlobalAveragePooling` (implemented as `nn.AdaptiveAvgPool2d` in `model/utils.py#L7-L23`) to collapse spatial dimensions into a single vector. When a CLS token is present, the system uses that embedding directly instead of pooling (see `model/utils.py#L46-L52`).
- **Text Head**: The text backbone output undergoes L2 normalization followed by `nn.AdaptiveAvgPool1d` to compress the token sequence into a fixed-dimensional representation (see `model/utils.py#L66-L70`).

### Logit Scaling and Normalization

A critical architectural addition is the **learnable temperature parameter** `logit_scale`, initialized to `log(1/0.07)` (approximately 1.0) as defined in `get_model.__init__` at `model/utils.py#L81`. During the forward pass (`model/utils.py#L85-L90`), both image and text embeddings undergo L2 normalization via `F.normalize` before returning the temperature-scaled similarity metric. The output signature follows the standard CLIP API: `forward(image, text) → (image_embedding, text_embedding, logit_scale.exp())`.

## Key Architectural Differences

| Feature | VisionRWKV | RWKV-CLIP |
|---------|------------|-----------|
| **Primary Purpose** | Pure visual feature extraction for downstream tasks | Joint image-text encoding for contrastive learning |
| **Core Components** | Patch embedding → positional embed → `Block_V6` stack (SpatialMix + ChannelMix) | `Image_RWKV` + `Text_RWKV` + pooling heads + temperature parameter |
| **Output Tensor Shape** | `[B, C, H, W]` feature maps or `[patch, cls]` tuples | Two 1-D embeddings `[B, D]` ready for cosine similarity calculation |
| **Additional Layers** | None beyond optional dropout and drop-path | `GlobalAveragePooling`, `AdaptiveAvgPool1d`, learnable `nn.Parameter` for temperature |
| **Training Objective** | Backbone pre-training or feature extraction | End-to-end CLIP contrastive loss (image-text matching) |
| **Key Files** | [`model/Image_rwkv.py`](https://github.com/deepglint/rwkv-clip/blob/main/model/Image_rwkv.py), `model/cuda_image/wkv6_cuda.cu` | [`model/utils.py`](https://github.com/deepglint/rwkv-clip/blob/main/model/utils.py), [`train.py`](https://github.com/deepglint/rwkv-clip/blob/main/train.py) |

## Implementation Examples

### Instantiating VisionRWKV Directly

For computer vision tasks requiring spatial feature maps, instantiate the backbone directly from [`model/Image_rwkv.py`](https://github.com/deepglint/rwkv-clip/blob/main/model/Image_rwkv.py):

```python
import torch
from model import Image_RWKV

# Configure the visual backbone

vision = Image_RWKV(
    img_size=224,
    patch_size=16,
    embed_dims=384,
    num_heads=8,
    depth=12,
    hidden_rate=4,
    output_cls_token=False,
    with_cls_token=False,
)

# Generate spatial features

dummy_img = torch.randn(2, 3, 224, 224)  # Batch size 2

patch_feats = vision(dummy_img)          # Output: [B, C, H, W]

print(patch_feats.shape)  # torch.Size([2, 384, 14, 14])

```

This configuration produces a 14×14 feature map with 384 channels, suitable for dense prediction tasks or as input to custom heads.

### Building the Complete RWKV-CLIP Model

For multimodal contrastive training, use the factory function in [`train.py`](https://github.com/deepglint/rwkv-clip/blob/main/train.py) to construct both encoders with proper argument parsing:

```python
import argparse
import torch
from train import get_model_RWKV_CLIP

# Configure arguments

parser = argparse.ArgumentParser()
parser.add_argument("--input-size", type=int, default=224)
parser.add_argument("--image-patch-size", type=int, default=16)
parser.add_argument("--image-embed-dims", type=int, default=384)
parser.add_argument("--image-depth", type=int, default=12)
parser.add_argument("--image-num-heads", type=int, default=8)

# Add vocab_size and other text parameters as needed

args = parser.parse_args([])

# Build dual-encoder

model = get_model_RWKV_CLIP(args)

# Forward pass

imgs = torch.randn(2, 3, 224, 224)
txt = torch.randint(0, 32000, (2, 77))  # Token IDs

img_emb, txt_emb, logit_scale = model(imgs, txt)
print(img_emb.shape, txt_emb.shape, logit_scale)

# torch.Size([2, 384]) torch.Size([2, 384]) tensor(1.0)

```

The `get_model_RWKV_CLIP` function handles the instantiation of both towers, pooling configurations, and the learnable temperature parameter required for contrastive learning.

## Summary

- **VisionRWKV** serves as a drop-in replacement for Vision Transformers, using `VRWKV_SpatialMix_V6` and `VRWKV_ChannelMix_V6` blocks with custom CUDA kernels (`wkv6_image`) to achieve linear-time attention complexity.
- **RWKV-CLIP** combines VisionRWKV with a text RWKV encoder, adding `GlobalAveragePooling`, `AdaptiveAvgPool1d`, and a learnable `logit_scale` parameter to create a complete CLIP-style architecture.
- The visual backbone outputs spatial feature maps `[B, C, H, W]` or patch/CLS tokens, while the full RWKV-CLIP system outputs normalized 1-D embeddings `[B, D]` designed for cosine similarity computation.
- Key implementation files include [`model/Image_rwkv.py`](https://github.com/deepglint/rwkv-clip/blob/main/model/Image_rwkv.py) for the visual backbone and [`model/utils.py`](https://github.com/deepglint/rwkv-clip/blob/main/model/utils.py) for the dual-encoder wrapper and pooling heads.

## Frequently Asked Questions

### When should I use VisionRWKV instead of RWKV-CLIP?

Use **VisionRWKV** when you need a standalone visual backbone for tasks like object detection, segmentation, or image classification where you will attach custom heads to the spatial feature maps. Use **RWKV-CLIP** only when you require a complete image-text dual-encoder system for zero-shot classification or multimodal retrieval tasks that rely on contrastive learning between visual and textual representations.

### Does RWKV-CLIP use the same RWKV kernel for both modalities?

No. According to the source code in `model/cuda_image/` and `model/cuda_text/`, the repository implements **separate CUDA kernels** for each modality: `wkv6_image` processes spatial patches in the visual encoder, while `wkv6_text` handles sequential tokens in the language model. Both implement the same RWKV recurrence mathematically but are optimized for their respective data structures (2-D spatial grids vs. 1-D sequences).

### What output format does VisionRWKV produce?

VisionRWKV produces either a **4-D tensor** of shape `[B, C, H, W]` representing the final patch feature map (the default behavior), or a tuple containing `[patch_tokens, cls_token]` when configured with `output_cls_token=True`. This differs from RWKV-CLIP, which always returns pooled 1-D vectors of shape `[B, D]` after applying global average pooling to the visual features.

### How does the learnable temperature work in RWKV-CLIP?

The **learnable temperature** (`logit_scale`) is a scalar parameter initialized to `log(1/0.07)` (approximately 1.0) as defined in `model/utils.py#L81`. During training, the model learns to adjust this temperature to optimize the contrastive loss. The forward pass returns `logit_scale.exp()`, which scales the cosine similarity between image and text embeddings. This learnable scaling factor allows the network to automatically find the optimal temperature for the contrastive objective rather than using a fixed hyperparameter.