# Optimal Learning Rate Schedules and Warmup Strategies for RWKV-CLIP

> Discover optimal learning rate schedules and warmup strategies for RWKV-CLIP. Explore cosine annealing with OneCycleLR or standard linear decay to boost your model's performance.

- Repository: [DeepGlint/rwkv-clip](https://github.com/deepglint/rwkv-clip)
- Tags: performance
- Published: 2026-02-28

---

**The RWKV-CLIP training pipeline implements a cosine annealing schedule with a built-in 10% linear warmup phase via OneCycleLR, while also offering a standard linear decay option without warmup.**

Choosing the right learning rate schedule is critical for stabilizing vision-language pretraining in the `deepglint/rwkv-clip` repository. The official training script provides two scheduler implementations that control how the optimizer's step size evolves across epochs, directly impacting convergence speed and final model performance.

## Built-in Learning Rate Schedulers

The [`train.py`](https://github.com/deepglint/rwkv-clip/blob/main/train.py) file defines two distinct scheduling strategies through command-line arguments, each suited to different training scenarios.

### Cosine Annealing with Linear Warmup (OneCycleLR)

The default and recommended scheduler employs PyTorch's `OneCycleLR` (lines 60-67 in [`train.py`](https://github.com/deepglint/rwkv-clip/blob/main/train.py)). This approach combines a **linear warmup** phase with a **cosine decay** phase in a single cycle:

```python
lr_scheduler = optim.lr_scheduler.OneCycleLR(
    optimizer=opt,
    max_lr=[args.lr],
    steps_per_epoch=steps_per_epoch,
    epochs=args.epochs,
    pct_start=0.1,  # Warmup for first 10% of training

)

```

The `pct_start=0.1` parameter hard-codes the warmup duration to 10% of total training steps. During this phase, the learning rate linearly increases from an initial fraction of `max_lr` up to the peak learning rate specified by `--lr`. After the warmup completes, the schedule follows a cosine curve decaying to near-zero.

### Linear Decay Schedule (LinearLR)

For experiments requiring simple monotonic decay without warmup, the script implements `LinearLR` (lines 68-71):

```python
lr_scheduler = optim.lr_scheduler.LinearLR(
    optimizer=opt,
    start_factor=1.0,
    end_factor=0.0,
    total_iters=total_steps
)

```

This scheduler gradually reduces the learning rate from its initial value to zero across the entire training run. Select this option by passing `--lr-scheduler linear` when launching training.

## Understanding the Warmup Configuration

Despite the presence of a `--warmup` CLI argument in the argument parser, **this parameter is currently unused** in the training logic. The effective warmup proportion is controlled exclusively by the `pct_start` value in the OneCycleLR initialization.

To adjust the warmup duration, you must modify the source code in [`train.py`](https://github.com/deepglint/rwkv-clip/blob/main/train.py):

```python

# For a 20% warmup instead of 10%

pct_start=0.2

```

Alternatively, replace the entire scheduler block with a custom implementation using `CosineAnnealingLR` or `CosineAnnealingWarmRestarts` paired with `LambdaLR` for manual warmup control.

## Implementation Examples

### Default Cosine Schedule with 10% Warmup

Run training with the recommended default configuration:

```bash
python train.py \
    --output ./output_dir \
    --train-data ./data/rec \
    --train-num-samples 1000000 \
    --batch-size 256 \
    --lr 0.1 \
    --lr-scheduler cosine \
    --epochs 32

```

### Linear Decay Without Warmup

For straightforward linear decay:

```bash
python train.py \
    --output ./output_dir \
    --train-data ./data/rec \
    --lr 0.1 \
    --lr-scheduler linear \
    --epochs 32

```

### Custom Warmup Duration (20%)

To extend the warmup to 20% of training steps, edit [`train.py`](https://github.com/deepglint/rwkv-clip/blob/main/train.py) before execution:

```python
lr_scheduler = optim.lr_scheduler.OneCycleLR(
    optimizer=opt,
    max_lr=[args.lr],
    steps_per_epoch=steps_per_epoch,
    epochs=args.epochs,
    pct_start=0.2,  # Modified: 20% warmup

)

```

## Summary

- **Default behavior**: The `OneCycleLR` scheduler in [`train.py`](https://github.com/deepglint/rwkv-clip/blob/main/train.py) automatically applies a 10% linear warmup via `pct_start=0.1`, followed by cosine decay.
- **Unused parameter**: The `--warmup` CLI argument is defined but ignored; warmup control requires modifying the `pct_start` value directly in the source code.
- **Alternative option**: Use `--lr-scheduler linear` for pure linear decay without any warmup phase.
- **Customization**: Replace the scheduler instantiation block to implement exotic schedules like warm restarts or polynomial decay.

## Frequently Asked Questions

### What is the default warmup strategy in RWKV-CLIP?

The default strategy uses `OneCycleLR` with `pct_start=0.1`, which linearly increases the learning rate from a small initial value to the maximum learning rate over the first 10% of training steps. Afterward, it cosine-decays the learning rate for the remaining 90% of steps. This implementation is located in [`train.py`](https://github.com/deepglint/rwkv-clip/blob/main/train.py) lines 60-67.

### Why does the --warmup argument not affect training?

Although the argument parser defines `--warmup` with a default value of `0.1`, the training script never references this variable when constructing the learning rate scheduler. The warmup proportion is hard-coded as `pct_start=0.1` in the `OneCycleLR` constructor call. To change the warmup duration, you must manually edit the `pct_start` parameter in [`train.py`](https://github.com/deepglint/rwkv-clip/blob/main/train.py).

### How do I implement a custom warmup duration?

To use a custom warmup proportion (e.g., 5% or 20%), modify line 65 in [`train.py`](https://github.com/deepglint/rwkv-clip/blob/main/train.py) where `OneCycleLR` is instantiated. Change `pct_start=0.1` to your desired decimal fraction (e.g., `pct_start=0.05` for 5%). For more complex warmup curves, replace the OneCycleLR block with a `LambdaLR` scheduler that implements your custom warmup function followed by your preferred decay schedule.

### Which scheduler performs better for vision-language pretraining?

According to the RWKV-CLIP source code, the **cosine schedule with warmup** is the recommended default for most experiments. The 10% warmup helps stabilize early training when optimizing large vision-language models, while the cosine decay provides smooth convergence. The linear scheduler serves as a baseline for ablation studies or when comparing against other architectures that use simple decay.