Optimal Learning Rate Schedules and Warmup Strategies for RWKV-CLIP

The RWKV-CLIP training pipeline implements a cosine annealing schedule with a built-in 10% linear warmup phase via OneCycleLR, while also offering a standard linear decay option without warmup.

Choosing the right learning rate schedule is critical for stabilizing vision-language pretraining in the deepglint/rwkv-clip repository. The official training script provides two scheduler implementations that control how the optimizer's step size evolves across epochs, directly impacting convergence speed and final model performance.

Built-in Learning Rate Schedulers

The train.py file defines two distinct scheduling strategies through command-line arguments, each suited to different training scenarios.

Cosine Annealing with Linear Warmup (OneCycleLR)

The default and recommended scheduler employs PyTorch's OneCycleLR (lines 60-67 in train.py). This approach combines a linear warmup phase with a cosine decay phase in a single cycle:

lr_scheduler = optim.lr_scheduler.OneCycleLR(
    optimizer=opt,
    max_lr=[args.lr],
    steps_per_epoch=steps_per_epoch,
    epochs=args.epochs,
    pct_start=0.1,  # Warmup for first 10% of training

)

The pct_start=0.1 parameter hard-codes the warmup duration to 10% of total training steps. During this phase, the learning rate linearly increases from an initial fraction of max_lr up to the peak learning rate specified by --lr. After the warmup completes, the schedule follows a cosine curve decaying to near-zero.

Linear Decay Schedule (LinearLR)

For experiments requiring simple monotonic decay without warmup, the script implements LinearLR (lines 68-71):

lr_scheduler = optim.lr_scheduler.LinearLR(
    optimizer=opt,
    start_factor=1.0,
    end_factor=0.0,
    total_iters=total_steps
)

This scheduler gradually reduces the learning rate from its initial value to zero across the entire training run. Select this option by passing --lr-scheduler linear when launching training.

Understanding the Warmup Configuration

Despite the presence of a --warmup CLI argument in the argument parser, this parameter is currently unused in the training logic. The effective warmup proportion is controlled exclusively by the pct_start value in the OneCycleLR initialization.

To adjust the warmup duration, you must modify the source code in train.py:

# For a 20% warmup instead of 10%

pct_start=0.2

Alternatively, replace the entire scheduler block with a custom implementation using CosineAnnealingLR or CosineAnnealingWarmRestarts paired with LambdaLR for manual warmup control.

Implementation Examples

Default Cosine Schedule with 10% Warmup

Run training with the recommended default configuration:

python train.py \
    --output ./output_dir \
    --train-data ./data/rec \
    --train-num-samples 1000000 \
    --batch-size 256 \
    --lr 0.1 \
    --lr-scheduler cosine \
    --epochs 32

Linear Decay Without Warmup

For straightforward linear decay:

python train.py \
    --output ./output_dir \
    --train-data ./data/rec \
    --lr 0.1 \
    --lr-scheduler linear \
    --epochs 32

Custom Warmup Duration (20%)

To extend the warmup to 20% of training steps, edit train.py before execution:

lr_scheduler = optim.lr_scheduler.OneCycleLR(
    optimizer=opt,
    max_lr=[args.lr],
    steps_per_epoch=steps_per_epoch,
    epochs=args.epochs,
    pct_start=0.2,  # Modified: 20% warmup

)

Summary

• Default behavior: The OneCycleLR scheduler in train.py automatically applies a 10% linear warmup via pct_start=0.1, followed by cosine decay.
• Unused parameter: The --warmup CLI argument is defined but ignored; warmup control requires modifying the pct_start value directly in the source code.
• Alternative option: Use --lr-scheduler linear for pure linear decay without any warmup phase.
• Customization: Replace the scheduler instantiation block to implement exotic schedules like warm restarts or polynomial decay.

Frequently Asked Questions

What is the default warmup strategy in RWKV-CLIP?

The default strategy uses OneCycleLR with pct_start=0.1, which linearly increases the learning rate from a small initial value to the maximum learning rate over the first 10% of training steps. Afterward, it cosine-decays the learning rate for the remaining 90% of steps. This implementation is located in train.py lines 60-67.

Why does the --warmup argument not affect training?

Although the argument parser defines --warmup with a default value of 0.1, the training script never references this variable when constructing the learning rate scheduler. The warmup proportion is hard-coded as pct_start=0.1 in the OneCycleLR constructor call. To change the warmup duration, you must manually edit the pct_start parameter in train.py.

How do I implement a custom warmup duration?

To use a custom warmup proportion (e.g., 5% or 20%), modify line 65 in train.py where OneCycleLR is instantiated. Change pct_start=0.1 to your desired decimal fraction (e.g., pct_start=0.05 for 5%). For more complex warmup curves, replace the OneCycleLR block with a LambdaLR scheduler that implements your custom warmup function followed by your preferred decay schedule.

Which scheduler performs better for vision-language pretraining?

According to the RWKV-CLIP source code, the cosine schedule with warmup is the recommended default for most experiments. The 10% warmup helps stabilize early training when optimizing large vision-language models, while the cosine decay provides smooth convergence. The linear scheduler serves as a baseline for ablation studies or when comparing against other architectures that use simple decay.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}