# Key Hyperparameters in RWKV-CLIP model_config JSON Files: Complete Tuning Guide

> Unlock RWKV-CLIP performance by tuning model_config JSON hyperparameters like image_patch_size and n_embd. Optimize your visual-text encoder for better capacity and convergence today.

- Repository: [DeepGlint/rwkv-clip](https://github.com/deepglint/rwkv-clip)
- Tags: deep-dive
- Published: 2026-02-28

---

**The RWKV-CLIP repository stores critical model architecture and training settings in JSON configuration files under `model_config/`, where hyperparameters like `image_patch_size`, `n_embd`, and `precision` control the visual-text encoder capacity, memory usage, and convergence behavior.**

Tuning these values correctly allows you to balance accuracy against GPU memory constraints when training or fine-tuning the RWKV-CLIP vision-language models. The repository provides two baseline configurations—[`RWKV_CLIP_B32.json`](https://github.com/deepglint/rwkv-clip/blob/main/RWKV_CLIP_B32.json) and [`RWKV_CLIP_B16.json`](https://github.com/deepglint/rwkv-clip/blob/main/RWKV_CLIP_B16.json)—that differ primarily in visual patch resolution but share the same hyperparameter schema.

## Overview of model_config JSON Files

The `model_config/` directory contains the primary configuration files that define model behavior:

- **[`model_config/RWKV_CLIP_B32.json`](https://github.com/deepglint/rwkv-clip/blob/main/model_config/RWKV_CLIP_B32.json)** – Default configuration for the B32-patch visual encoder (32×32 pixel patches).
- **[`model_config/RWKV_CLIP_B16.json`](https://github.com/deepglint/rwkv-clip/blob/main/model_config/RWKV_CLIP_B16.json)** – Configuration for the B16-patch visual encoder (16×16 pixel patches).
- **[`model_config/utils_notebook.py`](https://github.com/deepglint/rwkv-clip/blob/main/model_config/utils_notebook.py)** – Helper utilities such as `load_model_configs()` for programmatically loading these JSON files into Python dictionaries.

Both JSON files expose identical hyperparameter keys; only the values for image-specific fields (e.g., `image_patch_size`, `input_size`) differ between the B16 and B32 variants.

## Core Hyperparameters and Tuning Guidelines

The configuration schema divides hyperparameters into hardware efficiency, visual encoder architecture, text encoder architecture, and regularization groups.

### Hardware and Training Efficiency

These parameters determine GPU memory footprint and training throughput:

- **`precision`** – Numeric format for model tensors (`bf16`, `fp16`, or `fp32`).  
  Set to `bf16` or `fp16` on modern NVIDIA GPUs (Ampere or newer) to halve memory usage and accelerate matrix operations. Use `fp32` only when debugging numerical instabilities.

- **`workers`** – Number of CPU processes spawned for data loading.  
  Scale this to 4–8 times the number of GPUs, capped at the physical CPU core count. Increasing `workers` reduces GPU starvation during training but raises CPU memory pressure.

- **`with_cp`** – Boolean flag enabling gradient checkpointing.  
  Set `true` when training deeper models (e.g., `image_depth` > 12) on GPUs with limited VRAM. This trades ~20% training speed for the ability to fit larger batch sizes or deeper architectures.

### Visual Encoder Architecture

These keys define the Vision Transformer (ViT) backbone that processes image patches:

- **`image_patch_size`** – Spatial resolution of each patch (16 or 32).  
  Smaller values (16) yield finer-grained features and longer sequence lengths, improving retrieval accuracy on detailed imagery. Larger values (32) reduce sequence length, lowering memory and compute requirements.

- **`input_size`** – Height and width of the resized input image (square).  
  Must match the training resolution (commonly 224 or 336). Increasing this improves downstream task performance but quadratically increases compute.

- **`image_depth`** – Number of transformer blocks in the visual encoder.  
  Typical values range from 12 to 24. Increase this when under-fitting; decrease if GPU memory is constrained.

- **`image_embed_dims`** – Dimensionality of patch embeddings.  
  Common values are 640 or 1024. This must align with `n_embd` in the text encoder to enable cross-modal contrastive learning.

- **`image_num_heads`** – Attention heads per visual block.  
  Must evenly divide `image_embed_dims`. Increasing heads improves parallelization but can dilute per-head representational capacity.

- **`image_hidden_rate`** – Expansion factor for the MLP hidden dimension (`embed_dims * hidden_rate`).  
  Standard values are 4.0–6.0. Higher rates increase model capacity at the cost of memory.

- **`image_output_cls_token`** and **`image_with_cls_token`** – Booleans controlling the CLS token output.  
  Set both to `false` when using mean-pooled patch features; set to `true` when you require a single classification vector.

### Text Encoder Architecture

These parameters configure the RWKV-based language model that processes text:

- **`n_embd`** – Embedding dimension for the text encoder.  
  Must match `image_embed_dims` to ensure compatible contrastive loss computation.

- **`n_layer`** – Number of RWKV blocks in the text encoder.  
  Typical range is 6–12. Increase for complex linguistic tasks; decrease for faster inference.

- **`ctx_len`** – Maximum sequence length (tokens).  
  Default is 77 for CLIP-style training. Increasing this allows longer captions but linearly increases memory.

- **`vocab_size`** – Tokenizer vocabulary size.  
  Must align with the tokenizer checkpoint used during preprocessing.

- **`head_size`** and **`head_size_divisor`** – Control the per-head dimension calculation.  
  Usually `head_size` is derived as `n_embd // text_num_head`.

- **`text_num_head`** – Number of attention heads in the text encoder.  
  Must divide evenly into `n_embd`.

- **`pos_emb`** – Positional encoding type (0 = sinusoidal, 1 = learned).  
  Keep at `0` for standard CLIP initialization; switch to `1` for fine-tuning with learnable positions.

- **`text_initialization`** – Boolean indicating whether to load pretrained text weights.  
  Set `true` when using the provided CLIP text encoder checkpoints; set `false` for ablation studies.

### Regularization and Stochastic Depth

- **`dropout`** – Standard dropout rate applied to dense layers.  
  Range 0.0–0.3. Increase when overfitting is observed on small datasets.

- **`drop_path_rate`** – Stochastic depth rate for dropping entire transformer layers during training.  
  Effective for deep models (depth > 12). Values 0.0–0.3 improve generalization.

## Practical Tuning Workflow

Follow this systematic approach when adjusting the JSON configurations:

1. **Establish Hardware Baseline** – Set `precision` to `bf16` (if supported) and `workers` to 4× GPU count. Enable `with_cp` only if you encounter out-of-memory errors.

2. **Select Patch Resolution** – Choose `image_patch_size` (16 vs. 32) based on your dataset granularity. Use B16 for fine-grained recognition; B32 for faster experimentation.

3. **Align Embedding Dimensions** – Ensure `image_embed_dims` equals `n_embd`. Mismatches will cause contrastive loss computation errors.

4. **Scale Depth and Width** – Increase `image_depth` and `n_layer` together when moving from small to large datasets. Monitor GPU memory; if usage exceeds 80%, enable `with_cp` or reduce `image_hidden_rate`.

5. **Adjust Regularization** – Start with `dropout` = 0.0 and `drop_path_rate` = 0.0. Raise `dropout` to 0.1–0.2 if validation loss diverges from training loss. Apply `drop_path_rate` only when `image_depth` > 12.

6. **Validate Sequence Length** – Set `ctx_len` to 77 for standard CLIP-style training. Only increase if your captions consistently exceed this length, as memory usage scales linearly with `ctx_len`.

## Code Examples

### Loading and Modifying Configurations Programmatically

```python
import json
from pathlib import Path

# Load the B32 configuration

config_path = Path("model_config/RWKV_CLIP_B32.json")
cfg = json.loads(config_path.read_text())

# Hardware optimization: switch to fp16 precision

cfg["precision"] = "fp16"

# Architecture scaling: increase visual depth for larger datasets

cfg["image_depth"] = 16  # default is 12

# Memory management: enable gradient checkpointing for deeper models

cfg["with_cp"] = True

# Regularization: add dropout to prevent overfitting

cfg["dropout"] = 0.1

# Save the tuned configuration

tuned_path = Path("model_config/RWKV_CLIP_B32_tuned.json")
tuned_path.write_text(json.dumps(cfg, indent=4))
print(f"Tuned config saved to {tuned_path}")

```

### Using the Repository Helper Utility

```python

# Alternative: use the provided notebook utility

from model_config.utils_notebook import load_model_configs

# Load all available configurations

configs = load_model_configs("model_config")

# Access specific model settings

b32_cfg = configs["RWKV_CLIP_B32"]
b16_cfg = configs["RWKV_CLIP_B16"]

# Verify cross-modal alignment

assert b32_cfg["image_embed_dims"] == b32_cfg["n_embd"], \
    "Embedding dimensions must match between vision and text encoders"

```

### Shell-Based Configuration Tuning

```bash

# One-liner to create a memory-optimized variant of B16 config

python -c "
import json, pathlib, sys
p = pathlib.Path('model_config/RWKV_CLIP_B16.json')
cfg = json.loads(p.read_text())
cfg['precision'] = 'bf16'
cfg['with_cp'] = True
cfg['dropout'] = 0.05
p.with_name('RWKV_CLIP_B16_memory_optimized.json').write_text(json.dumps(cfg, indent=2))
" 

```

## Summary

- **Configuration Location**: All hyperparameters reside in [`model_config/RWKV_CLIP_B32.json`](https://github.com/deepglint/rwkv-clip/blob/main/model_config/RWKV_CLIP_B32.json) and [`model_config/RWKV_CLIP_B16.json`](https://github.com/deepglint/rwkv-clip/blob/main/model_config/RWKV_CLIP_B16.json), with helper utilities in [`model_config/utils_notebook.py`](https://github.com/deepglint/rwkv-clip/blob/main/model_config/utils_notebook.py).

- **Critical Alignment**: Always ensure `image_embed_dims` equals `n_embd` to maintain compatible vision-text embedding spaces for contrastive learning.

- **Hardware Optimization**: Set `precision` to `bf16` or `fp16` for modern GPUs, adjust `workers` to CPU core count, and enable `with_cp` (gradient checkpointing) when training deeper models on memory-constrained hardware.

- **Architecture Scaling**: Increase `image_depth` and `n_layer` for higher capacity; reduce `image_patch_size` (switching from B32 to B16) for finer-grained visual features at the cost of increased compute.

- **Regularization**: Apply `dropout` (0.0–0.3) for dense layer regularization and `drop_path_rate` (0.0–0.3) for stochastic depth in deep transformers, particularly when overfitting occurs on small datasets.

## Frequently Asked Questions

### What is the difference between RWKV_CLIP_B32.json and RWKV_CLIP_B16.json?

The primary distinction is the `image_patch_size` parameter: B32 uses 32×32 pixel patches while B16 uses 16×16 patches. The B16 configuration produces longer visual sequences and captures finer spatial details, making it suitable for high-resolution image understanding, whereas B32 offers faster training and lower memory consumption. Both files share identical hyperparameter schemas, allowing you to switch between resolutions by adjusting only the patch-related fields.

### How do I enable gradient checkpointing to train larger models?

Set the `with_cp` (with checkpointing) boolean to `true` in your JSON configuration. This setting triggers gradient checkpointing in the visual and text encoders, trading approximately 20% training speed for the ability to fit models with increased `image_depth` or larger `ctx_len` into limited GPU memory. This is essential when scaling `image_depth` beyond 12 layers or when using `precision: fp32` on consumer-grade GPUs.

### Why must image_embed_dims match n_embd?

These parameters define the dimensionalities of the visual encoder output (`image_embed_dims`) and the text encoder output (`n_embd`). The RWKV-CLIP contrastive loss computation requires both modalities to project into the same embedding space to calculate similarity scores. A mismatch will raise dimensionality errors during the forward pass of the multimodal fusion layers. When modifying either value, always update the other to maintain alignment.

### What is the recommended starting configuration for fine-tuning on a small dataset?

Begin with the [`RWKV_CLIP_B16.json`](https://github.com/deepglint/rwkv-clip/blob/main/RWKV_CLIP_B16.json) template for better feature granularity, but immediately increase regularization to prevent overfitting. Set `dropout` to 0.1–0.2 and `drop_path_rate` to 0.1 if `image_depth` exceeds 12. Keep `precision` at `bf16` for efficiency, enable `with_cp` only if memory is constrained, and maintain the default `ctx_len` of 77 unless your text captions are consistently longer. Monitor validation loss closely; if it diverges early, reduce `image_depth` or increase `dropout` further.