# RIFE Learning Rate Schedule: Warm-Up and Cosine Annealing Explained

> Discover RIFE's learning rate schedule: a 2k step warm-up to 3e-4 then cosine annealing to 3e-6. Understand this key training technique from hzwer/eccv2022-rife.

- Repository: [hzwer/eccv2022-rife](https://github.com/hzwer/eccv2022-rife)
- Tags: deep-dive
- Published: 2026-03-03

---

**RIFE implements a two-phase learning rate schedule featuring a 2,000-step linear warm-up from 0 to 3e-4, followed by cosine annealing decay down to 3e-6 for the remainder of training.**

The `hzwer/eccv2022-rife` repository uses this specific learning rate schedule to stabilize early training while ensuring smooth convergence during the final epochs. Understanding this schedule is essential for reproducing RIFE's results or fine-tuning the model on custom video interpolation datasets.

## The Two-Phase Learning Rate Schedule in RIFE

RIFE's training script defines the schedule in the `get_learning_rate(step)` function within [`train.py`](https://github.com/hzwer/eccv2022-rife/blob/main/train.py). The implementation divides training into distinct warm-up and decay phases to optimize convergence behavior.

### Phase 1: Linear Warm-Up (Steps 0–2,000)

During the initial 2,000 steps, the learning rate increases linearly from 0 to the base value of **3e-4**. This gradual ramp-up prevents early training instability that can occur when using large learning rates on randomly initialized weights.

```python
if step < 2000:
    mul = step / 2000.
    return 3e-4 * mul

```

### Phase 2: Cosine Annealing (Remaining Steps)

After completing the warm-up, the schedule transitions to cosine annealing. The learning rate follows a cosine curve decaying from **3e-4** down to **3e-6** over the remaining training steps. The decay calculation uses the total training duration (`args.epoch * args.step_per_epoch`) minus the 2,000-step warm-up period.

```python
else:
    mul = np.cos(
        (step - 2000) /
        (args.epoch * args.step_per_epoch - 2000.) *
        math.pi) * 0.5 + 0.5
    return (3e-4 - 3e-6) * mul + 3e-6

```

## Implementation Details in train.py

The `get_learning_rate(step)` function returns the raw learning rate value, which the training loop then scales according to the distributed training configuration. Specifically, the effective learning rate passed to the model's `update` method is multiplied by `args.world_size / 4`:

```python
learning_rate = get_learning_rate(step) * args.world_size / 4

```

This scaling ensures consistent optimization behavior across different GPU configurations, automatically adjusting the learning rate based on the number of available GPUs relative to the baseline 4-GPU setup.

## Code Examples

### Computing Learning Rates for Specific Steps

You can inspect the learning rate schedule without running full training by importing the schedule function:

```python
import numpy as np
import math
from train import get_learning_rate, args

# Configure training parameters for inspection

args.epoch = 100
args.step_per_epoch = 1000
args.world_size = 4

def compute_lr(step):
    # Apply world size scaling as in the actual training loop

    return get_learning_rate(step) * args.world_size / 4

# Inspect learning rates at key milestones

milestones = [0, 1000, 2000, 5000, args.epoch * args.step_per_epoch]
for step in milestones:
    print(f"Step {step:>6}: LR = {compute_lr(step):.2e}")

```

### Integrating the Schedule into Custom Training Loops

When adapting RIFE's schedule for other optimizers or frameworks, manually update the learning rate each step:

```python
import torch
import torch.optim as optim

# Initialize optimizer with zero LR (will be set manually)

optimizer = optim.Adam(model.parameters(), lr=0)

for step, batch in enumerate(train_loader):
    # Calculate current learning rate using RIFE's schedule

    current_lr = get_learning_rate(step) * world_size / 4
    
    # Update optimizer learning rate

    for param_group in optimizer.param_groups:
        param_group['lr'] = current_lr
    
    # Standard training step

    optimizer.zero_grad()
    loss = model(batch)
    loss.backward()
    optimizer.step()

```

## Summary

- RIFE uses a **two-phase learning rate schedule** defined in [`train.py`](https://github.com/hzwer/eccv2022-rife/blob/main/train.py)'s `get_learning_rate(step)` function.
- **Phase 1** implements a 2,000-step linear warm-up from 0 to **3e-4**.
- **Phase 2** applies cosine annealing decay from **3e-4** down to **3e-6** for the remaining training steps.
- The learning rate is **scaled by world size** (`args.world_size / 4`) to support distributed training across multiple GPUs.

## Frequently Asked Questions

### What is the base learning rate in RIFE?

The base learning rate is **3e-4** (0.0003). This value represents the maximum learning rate reached at the end of the 2,000-step warm-up phase and serves as the starting point for the subsequent cosine annealing decay.

### How long is the warm-up period?

RIFE uses a **2,000-step warm-up period**. During these initial steps, the learning rate increases linearly from 0 to the base value of 3e-4. This duration is hardcoded in the `get_learning_rate(step)` function within [`train.py`](https://github.com/hzwer/eccv2022-rife/blob/main/train.py).

### Does RIFE use learning rate scaling for distributed training?

Yes, RIFE implements **learning rate scaling based on world size**. The effective learning rate passed to the optimizer is calculated as `get_learning_rate(step) * args.world_size / 4`. This scaling ensures that the optimization process remains stable when training across different numbers of GPUs, automatically adjusting the step size relative to the baseline 4-GPU configuration.

### What is the minimum learning rate at the end of training?

The minimum learning rate is **3e-6** (0.000003). This value represents the floor of the cosine annealing curve reached at the final training step. The schedule linearly interpolates between the base rate (3e-4) and this minimum value using the cosine multiplier.