RWKV-CLIP-B/16 vs RWKV-CLIP-B/32: Architectural Differences and Selection Guide

RWKV-CLIP-B/16 and RWKV-CLIP-B/32 are vision-language models from the deepglint/rwkv-clip repository that share identical RWKV backbones but differ only in image patch size—16 pixels yielding 196 tokens for high-resolution detail versus 32 pixels yielding 49 tokens for computational efficiency.

The deepglint/rwkv-clip repository provides CLIP-style contrastive learning models built on the RWKV architecture. Both RWKV-CLIP-B/16 (also referenced as B/I6) and RWKV-CLIP-B/32 variants offer strong zero-shot capabilities, but selecting between them requires understanding how patch size affects tokenization, memory usage, and downstream accuracy.

Core Architectural Distinctions

Image Patch Size and Tokenization

The only architectural difference between the two variants is defined in their JSON configuration files. In model_config/RWKV_CLIP_B16.json, line 8 specifies "image_patch_size": 16, while model_config/RWKV_CLIP_B32.json sets "image_patch_size": 32.

For the standard 224×224 pixel input resolution used by both models:

• RWKV-CLIP-B/16: Divides the image into 16×16 pixel patches, producing (224 / 16)² = 14 × 14 = **196** image tokens
• RWKV-CLIP-B/32: Uses 32×32 pixel patches, resulting in (224 / 32)² = 7 × 7 = **49** image tokens

This 4× difference in sequence length directly impacts the model's spatial granularity and computational requirements.

Identical Backbone Configuration

All other hyper-parameters remain constant between variants, ensuring the RWKV architecture itself does not change:

• image_depth: 12 layers
• image_embed_dims: 640 dimensions
• image_hidden_rate: 5
• image_num_heads: 8 attention heads
• n_layer: 6 RWKV blocks
• n_embd: 640 embedding dimensions

As implemented in deepglint/rwkv-clip, both configuration files specify identical text encoders and RWKV backbones; only the visual tokenization strategy differs.

Performance and Computational Trade-offs

Accuracy on Fine-Grained Tasks

RWKV-CLIP-B/16 captures finer spatial details due to its higher token density, typically yielding stronger zero-shot retrieval and classification performance on tasks requiring precise visual reasoning. The smaller 16-pixel patches preserve more local information, benefiting fine-grained recognition.

RWKV-CLIP-B/32 provides a coarser visual representation with larger receptive fields per token. While slightly less precise on detail-intensive tasks, it maintains competitive accuracy on general vision-language benchmarks.

Memory and Inference Speed

RWKV-CLIP-B/16 requires approximately 4× the FLOPs and GPU memory compared to B/32 because it processes 196 versus 49 image tokens. This results in slower training throughput and higher latency during inference.

RWKV-CLIP-B/32 minimizes memory consumption and maximizes processing speed, making it advantageous for resource-constrained environments, batch processing at scale, or real-time applications where latency matters more than pixel-level precision.

Loading and Running Inference with Both Variants

To instantiate either variant, use the configuration loader from model_config/utils_notebook.py and the model builder from model/utils.py. The following example demonstrates loading both models and performing zero-shot classification:

import torch
import clip
from PIL import Image
from open_clip.transform import image_transform
from model_config.utils_notebook import load_model_configs
from model.utils import create_RWKV_Model

# Load RWKV-CLIP-B/16 (high-resolution variant)

cfg_b16 = load_model_configs('model_config/RWKV_CLIP_B16.json')
model_b16 = create_RWKV_Model(cfg_b16, model_weight_path='path/to/B16_weight.pt')
model_b16.eval().to('cuda' if torch.cuda.is_available() else 'cpu')

# Load RWKV-CLIP-B/32 (efficient variant)  

cfg_b32 = load_model_configs('model_config/RWKV_CLIP_B32.json')
model_b32 = create_RWKV_Model(cfg_b32, model_weight_path='path/to/B32_weight.pt')
model_b32.eval().to('cuda' if torch.cuda.is_available() else 'cpu')

# Preprocess inputs (224x224 for both variants)

transform = image_transform(cfg_b16.input_size, False)
image = transform(Image.open('example.jpg')).unsqueeze(0)
text = clip.tokenize(['a diagram', 'a dog', 'a cat'])

device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
image, text = image.to(device), text.to(device)

# Inference with B/16

with torch.no_grad():
    img_feat_b16, txt_feat_b16, _ = model_b16(image, text)
    img_feat_b16 = img_feat_b16 / img_feat_b16.norm(dim=-1, keepdim=True)
    txt_feat_b16 = txt_feat_b16 / txt_feat_b16.norm(dim=-1, keepdim=True)
    probs_b16 = (100.0 * img_feat_b16 @ txt_feat_b16.T).softmax(dim=-1)
    print('B/16 probabilities:', probs_b16.squeeze().cpu().numpy())

# Inference with B/32

with torch.no_grad():
    img_feat_b32, txt_feat_b32, _ = model_b32(image, text)
    img_feat_b32 = img_feat_b32 / img_feat_b32.norm(dim=-1, keepdim=True)
    txt_feat_b32 = txt_feat_b32 / txt_feat_b32.norm(dim=-1, keepdim=True)
    probs_b32 = (100.0 * img_feat_b32 @ txt_feat_b32.T).softmax(dim=-1)
    print('B/32 probabilities:', probs_b32.squeeze().cpu().numpy())

Both configurations share the same input_size of 224×224 pixels and use identical preprocessing pipelines, ensuring consistent image normalization across variants.

Summary

• Patch size is the sole architectural difference: 16×16 pixels for B/16 versus 32×32 pixels for B/32, configured via "image_patch_size" in model_config/RWKV_CLIP_B16.json and model_config/RWKV_CLIP_B32.json.
• Token count scales quadratically: B/16 processes 196 image tokens while B/32 processes 49 tokens for the same 224×224 input.
• B/16 excels at fine-grained tasks requiring detailed spatial understanding but demands approximately 4× more memory and computation.
• B/32 prioritizes efficiency with lower memory requirements and faster inference while maintaining strong baseline zero-shot performance.
• Backbone architecture is identical: Both variants use 12-layer image encoders with 640-dimensional embeddings and 6 RWKV blocks as defined in their respective configuration files.

Frequently Asked Questions

What is the main difference between RWKV-CLIP-B/16 and B/32?

The primary distinction is the image patch size defined in their configuration files. RWKV-CLIP-B/16 uses 16×16 pixel patches producing 196 tokens, while B/32 uses 32×32 patches producing 49 tokens. All other architectural parameters—including the RWKV backbone depth, embedding dimensions, and text encoder settings—remain identical between the two variants.

Which variant should I use for fine-grained image classification?

Choose RWKV-CLIP-B/16 when your application requires detecting subtle visual details or precise spatial relationships. The smaller patch size generates a longer token sequence that preserves finer-grained information, typically resulting in higher zero-shot accuracy on detailed classification and retrieval tasks according to the model configuration analysis.

Do both variants use the same pretrained weights?

No, each variant requires its own specific pretrained checkpoint due to the different patch embedding dimensions caused by the patch size disparity. You must load weights matching your configuration—B/16 weights for the 16-pixel config and B/32 weights for the 32-pixel config—when calling create_RWKV_Model() from model/utils.py.

How do I switch between B/16 and B/32 in my existing codebase?

Switching requires only two changes: (1) update the configuration file path passed to load_model_configs()—use model_config/RWKV_CLIP_B16.json for B/16 or model_config/RWKV_CLIP_B32.json for B/32; (2) ensure you load the corresponding pretrained weights specific to that patch size. The inference code, preprocessing transforms, and model API remain identical between variants.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}