Key Hyperparameters in RWKV-CLIP model_config JSON Files: Complete Tuning Guide

The RWKV-CLIP repository stores critical model architecture and training settings in JSON configuration files under model_config/, where hyperparameters like image_patch_size, n_embd, and precision control the visual-text encoder capacity, memory usage, and convergence behavior.

Tuning these values correctly allows you to balance accuracy against GPU memory constraints when training or fine-tuning the RWKV-CLIP vision-language models. The repository provides two baseline configurations—RWKV_CLIP_B32.json and RWKV_CLIP_B16.json—that differ primarily in visual patch resolution but share the same hyperparameter schema.

Overview of model_config JSON Files

The model_config/ directory contains the primary configuration files that define model behavior:

• model_config/RWKV_CLIP_B32.json – Default configuration for the B32-patch visual encoder (32×32 pixel patches).
• model_config/RWKV_CLIP_B16.json – Configuration for the B16-patch visual encoder (16×16 pixel patches).
• model_config/utils_notebook.py – Helper utilities such as load_model_configs() for programmatically loading these JSON files into Python dictionaries.

Both JSON files expose identical hyperparameter keys; only the values for image-specific fields (e.g., image_patch_size, input_size) differ between the B16 and B32 variants.

Core Hyperparameters and Tuning Guidelines

The configuration schema divides hyperparameters into hardware efficiency, visual encoder architecture, text encoder architecture, and regularization groups.

Hardware and Training Efficiency

These parameters determine GPU memory footprint and training throughput:

• precision – Numeric format for model tensors (bf16, fp16, or fp32).
  Set to bf16 or fp16 on modern NVIDIA GPUs (Ampere or newer) to halve memory usage and accelerate matrix operations. Use fp32 only when debugging numerical instabilities.

• workers – Number of CPU processes spawned for data loading.
  Scale this to 4–8 times the number of GPUs, capped at the physical CPU core count. Increasing workers reduces GPU starvation during training but raises CPU memory pressure.

• with_cp – Boolean flag enabling gradient checkpointing.
  Set true when training deeper models (e.g., image_depth > 12) on GPUs with limited VRAM. This trades ~20% training speed for the ability to fit larger batch sizes or deeper architectures.

Visual Encoder Architecture

These keys define the Vision Transformer (ViT) backbone that processes image patches:

• image_patch_size – Spatial resolution of each patch (16 or 32).
  Smaller values (16) yield finer-grained features and longer sequence lengths, improving retrieval accuracy on detailed imagery. Larger values (32) reduce sequence length, lowering memory and compute requirements.

• input_size – Height and width of the resized input image (square).
  Must match the training resolution (commonly 224 or 336). Increasing this improves downstream task performance but quadratically increases compute.

• image_depth – Number of transformer blocks in the visual encoder.
  Typical values range from 12 to 24. Increase this when under-fitting; decrease if GPU memory is constrained.

• image_embed_dims – Dimensionality of patch embeddings.
  Common values are 640 or 1024. This must align with n_embd in the text encoder to enable cross-modal contrastive learning.

• image_num_heads – Attention heads per visual block.
  Must evenly divide image_embed_dims. Increasing heads improves parallelization but can dilute per-head representational capacity.

• image_hidden_rate – Expansion factor for the MLP hidden dimension (embed_dims * hidden_rate).
  Standard values are 4.0–6.0. Higher rates increase model capacity at the cost of memory.

• image_output_cls_token and image_with_cls_token – Booleans controlling the CLS token output.
  Set both to false when using mean-pooled patch features; set to true when you require a single classification vector.

Text Encoder Architecture

These parameters configure the RWKV-based language model that processes text:

• n_embd – Embedding dimension for the text encoder.
  Must match image_embed_dims to ensure compatible contrastive loss computation.

• n_layer – Number of RWKV blocks in the text encoder.
  Typical range is 6–12. Increase for complex linguistic tasks; decrease for faster inference.

• ctx_len – Maximum sequence length (tokens).
  Default is 77 for CLIP-style training. Increasing this allows longer captions but linearly increases memory.

• vocab_size – Tokenizer vocabulary size.
  Must align with the tokenizer checkpoint used during preprocessing.

• head_size and head_size_divisor – Control the per-head dimension calculation.
  Usually head_size is derived as n_embd // text_num_head.

• text_num_head – Number of attention heads in the text encoder.
  Must divide evenly into n_embd.

• pos_emb – Positional encoding type (0 = sinusoidal, 1 = learned).
  Keep at 0 for standard CLIP initialization; switch to 1 for fine-tuning with learnable positions.

• text_initialization – Boolean indicating whether to load pretrained text weights.
  Set true when using the provided CLIP text encoder checkpoints; set false for ablation studies.

Regularization and Stochastic Depth

• dropout – Standard dropout rate applied to dense layers.
  Range 0.0–0.3. Increase when overfitting is observed on small datasets.

• drop_path_rate – Stochastic depth rate for dropping entire transformer layers during training.
  Effective for deep models (depth > 12). Values 0.0–0.3 improve generalization.

Practical Tuning Workflow

Follow this systematic approach when adjusting the JSON configurations:

1. Establish Hardware Baseline – Set precision to bf16 (if supported) and workers to 4× GPU count. Enable with_cp only if you encounter out-of-memory errors.

2. Select Patch Resolution – Choose image_patch_size (16 vs. 32) based on your dataset granularity. Use B16 for fine-grained recognition; B32 for faster experimentation.

3. Align Embedding Dimensions – Ensure image_embed_dims equals n_embd. Mismatches will cause contrastive loss computation errors.

4. Scale Depth and Width – Increase image_depth and n_layer together when moving from small to large datasets. Monitor GPU memory; if usage exceeds 80%, enable with_cp or reduce image_hidden_rate.

5. Adjust Regularization – Start with dropout = 0.0 and drop_path_rate = 0.0. Raise dropout to 0.1–0.2 if validation loss diverges from training loss. Apply drop_path_rate only when image_depth > 12.

6. Validate Sequence Length – Set ctx_len to 77 for standard CLIP-style training. Only increase if your captions consistently exceed this length, as memory usage scales linearly with ctx_len.

Code Examples

Loading and Modifying Configurations Programmatically

import json
from pathlib import Path

# Load the B32 configuration

config_path = Path("model_config/RWKV_CLIP_B32.json")
cfg = json.loads(config_path.read_text())

# Hardware optimization: switch to fp16 precision

cfg["precision"] = "fp16"

# Architecture scaling: increase visual depth for larger datasets

cfg["image_depth"] = 16  # default is 12

# Memory management: enable gradient checkpointing for deeper models

cfg["with_cp"] = True

# Regularization: add dropout to prevent overfitting

cfg["dropout"] = 0.1

# Save the tuned configuration

tuned_path = Path("model_config/RWKV_CLIP_B32_tuned.json")
tuned_path.write_text(json.dumps(cfg, indent=4))
print(f"Tuned config saved to {tuned_path}")

Using the Repository Helper Utility

# Alternative: use the provided notebook utility

from model_config.utils_notebook import load_model_configs

# Load all available configurations

configs = load_model_configs("model_config")

# Access specific model settings

b32_cfg = configs["RWKV_CLIP_B32"]
b16_cfg = configs["RWKV_CLIP_B16"]

# Verify cross-modal alignment

assert b32_cfg["image_embed_dims"] == b32_cfg["n_embd"], \
    "Embedding dimensions must match between vision and text encoders"

Shell-Based Configuration Tuning

# One-liner to create a memory-optimized variant of B16 config

python -c "
import json, pathlib, sys
p = pathlib.Path('model_config/RWKV_CLIP_B16.json')
cfg = json.loads(p.read_text())
cfg['precision'] = 'bf16'
cfg['with_cp'] = True
cfg['dropout'] = 0.05
p.with_name('RWKV_CLIP_B16_memory_optimized.json').write_text(json.dumps(cfg, indent=2))
" 

Summary

• Configuration Location: All hyperparameters reside in model_config/RWKV_CLIP_B32.json and model_config/RWKV_CLIP_B16.json, with helper utilities in model_config/utils_notebook.py.

• Critical Alignment: Always ensure image_embed_dims equals n_embd to maintain compatible vision-text embedding spaces for contrastive learning.

• Hardware Optimization: Set precision to bf16 or fp16 for modern GPUs, adjust workers to CPU core count, and enable with_cp (gradient checkpointing) when training deeper models on memory-constrained hardware.

• Architecture Scaling: Increase image_depth and n_layer for higher capacity; reduce image_patch_size (switching from B32 to B16) for finer-grained visual features at the cost of increased compute.

• Regularization: Apply dropout (0.0–0.3) for dense layer regularization and drop_path_rate (0.0–0.3) for stochastic depth in deep transformers, particularly when overfitting occurs on small datasets.

Frequently Asked Questions

What is the difference between RWKV_CLIP_B32.json and RWKV_CLIP_B16.json?

The primary distinction is the image_patch_size parameter: B32 uses 32×32 pixel patches while B16 uses 16×16 patches. The B16 configuration produces longer visual sequences and captures finer spatial details, making it suitable for high-resolution image understanding, whereas B32 offers faster training and lower memory consumption. Both files share identical hyperparameter schemas, allowing you to switch between resolutions by adjusting only the patch-related fields.

How do I enable gradient checkpointing to train larger models?

Set the with_cp (with checkpointing) boolean to true in your JSON configuration. This setting triggers gradient checkpointing in the visual and text encoders, trading approximately 20% training speed for the ability to fit models with increased image_depth or larger ctx_len into limited GPU memory. This is essential when scaling image_depth beyond 12 layers or when using precision: fp32 on consumer-grade GPUs.

Why must image_embed_dims match n_embd?

These parameters define the dimensionalities of the visual encoder output (image_embed_dims) and the text encoder output (n_embd). The RWKV-CLIP contrastive loss computation requires both modalities to project into the same embedding space to calculate similarity scores. A mismatch will raise dimensionality errors during the forward pass of the multimodal fusion layers. When modifying either value, always update the other to maintain alignment.

What is the recommended starting configuration for fine-tuning on a small dataset?

Begin with the RWKV_CLIP_B16.json template for better feature granularity, but immediately increase regularization to prevent overfitting. Set dropout to 0.1–0.2 and drop_path_rate to 0.1 if image_depth exceeds 12. Keep precision at bf16 for efficiency, enable with_cp only if memory is constrained, and maintain the default ctx_len of 77 unless your text captions are consistently longer. Monitor validation loss closely; if it diverges early, reduce image_depth or increase dropout further.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}