RIFE Learning Rate Schedule: Warm-Up and Cosine Annealing Explained

RIFE implements a two-phase learning rate schedule featuring a 2,000-step linear warm-up from 0 to 3e-4, followed by cosine annealing decay down to 3e-6 for the remainder of training.

The hzwer/eccv2022-rife repository uses this specific learning rate schedule to stabilize early training while ensuring smooth convergence during the final epochs. Understanding this schedule is essential for reproducing RIFE's results or fine-tuning the model on custom video interpolation datasets.

The Two-Phase Learning Rate Schedule in RIFE

RIFE's training script defines the schedule in the get_learning_rate(step) function within train.py. The implementation divides training into distinct warm-up and decay phases to optimize convergence behavior.

Phase 1: Linear Warm-Up (Steps 0–2,000)

During the initial 2,000 steps, the learning rate increases linearly from 0 to the base value of 3e-4. This gradual ramp-up prevents early training instability that can occur when using large learning rates on randomly initialized weights.

if step < 2000:
    mul = step / 2000.
    return 3e-4 * mul

Phase 2: Cosine Annealing (Remaining Steps)

After completing the warm-up, the schedule transitions to cosine annealing. The learning rate follows a cosine curve decaying from 3e-4 down to 3e-6 over the remaining training steps. The decay calculation uses the total training duration (args.epoch * args.step_per_epoch) minus the 2,000-step warm-up period.

else:
    mul = np.cos(
        (step - 2000) /
        (args.epoch * args.step_per_epoch - 2000.) *
        math.pi) * 0.5 + 0.5
    return (3e-4 - 3e-6) * mul + 3e-6

Implementation Details in train.py

The get_learning_rate(step) function returns the raw learning rate value, which the training loop then scales according to the distributed training configuration. Specifically, the effective learning rate passed to the model's update method is multiplied by args.world_size / 4:

learning_rate = get_learning_rate(step) * args.world_size / 4

This scaling ensures consistent optimization behavior across different GPU configurations, automatically adjusting the learning rate based on the number of available GPUs relative to the baseline 4-GPU setup.

Code Examples

Computing Learning Rates for Specific Steps

You can inspect the learning rate schedule without running full training by importing the schedule function:

import numpy as np
import math
from train import get_learning_rate, args

# Configure training parameters for inspection

args.epoch = 100
args.step_per_epoch = 1000
args.world_size = 4

def compute_lr(step):
    # Apply world size scaling as in the actual training loop

    return get_learning_rate(step) * args.world_size / 4

# Inspect learning rates at key milestones

milestones = [0, 1000, 2000, 5000, args.epoch * args.step_per_epoch]
for step in milestones:
    print(f"Step {step:>6}: LR = {compute_lr(step):.2e}")

Integrating the Schedule into Custom Training Loops

When adapting RIFE's schedule for other optimizers or frameworks, manually update the learning rate each step:

import torch
import torch.optim as optim

# Initialize optimizer with zero LR (will be set manually)

optimizer = optim.Adam(model.parameters(), lr=0)

for step, batch in enumerate(train_loader):
    # Calculate current learning rate using RIFE's schedule

    current_lr = get_learning_rate(step) * world_size / 4
    
    # Update optimizer learning rate

    for param_group in optimizer.param_groups:
        param_group['lr'] = current_lr
    
    # Standard training step

    optimizer.zero_grad()
    loss = model(batch)
    loss.backward()
    optimizer.step()

Summary

• RIFE uses a two-phase learning rate schedule defined in train.py's get_learning_rate(step) function.
• Phase 1 implements a 2,000-step linear warm-up from 0 to 3e-4.
• Phase 2 applies cosine annealing decay from 3e-4 down to 3e-6 for the remaining training steps.
• The learning rate is scaled by world size (args.world_size / 4) to support distributed training across multiple GPUs.

Frequently Asked Questions

What is the base learning rate in RIFE?

The base learning rate is 3e-4 (0.0003). This value represents the maximum learning rate reached at the end of the 2,000-step warm-up phase and serves as the starting point for the subsequent cosine annealing decay.

How long is the warm-up period?

RIFE uses a 2,000-step warm-up period. During these initial steps, the learning rate increases linearly from 0 to the base value of 3e-4. This duration is hardcoded in the get_learning_rate(step) function within train.py.

Does RIFE use learning rate scaling for distributed training?

Yes, RIFE implements learning rate scaling based on world size. The effective learning rate passed to the optimizer is calculated as get_learning_rate(step) * args.world_size / 4. This scaling ensures that the optimization process remains stable when training across different numbers of GPUs, automatically adjusting the step size relative to the baseline 4-GPU configuration.

What is the minimum learning rate at the end of training?

The minimum learning rate is 3e-6 (0.000003). This value represents the floor of the cosine annealing curve reached at the final training step. The schedule linearly interpolates between the base rate (3e-4) and this minimum value using the cosine multiplier.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}