How to Run Benchmarks on Vimeo90K, UCF101, and MiddleBury Datasets with RIFE

You can evaluate the RIFE model on Vimeo90K, UCF101, and MiddleBury datasets by executing the dedicated Python scripts in the benchmark/ directory after placing the downloaded datasets in the expected folder structure.

The hzwer/eccv2022-rife repository provides standardized evaluation scripts that compute PSNR, SSIM, and Interpolation Error (IE) to measure the quality of frame interpolation. These benchmarks allow you to reproduce the paper's results or validate custom model modifications against established video interpolation datasets.

Benchmark Overview and Metrics

The RIFE benchmark suite targets three distinct test sets, each emphasizing different aspects of temporal consistency and motion handling:

Dataset	Primary Metrics	Benchmark Script
Vimeo90K	PSNR, SSIM	benchmark/Vimeo90K.py
UCF101	PSNR, SSIM	benchmark/UCF101.py
Middlebury (Other)	Interpolation Error (IE)	benchmark/MiddleBury_Other.py

PSNR (Peak Signal-to-Noise Ratio) calculates (-10 \log_{10}(\text{MSE})) on raw RGB values to quantify pixel-level accuracy. SSIM (Structural Similarity Index) uses the MATLAB-compatible implementation in model.pytorch_msssim.ssim_matlab to assess perceptual quality. IE (Interpolation Error) for Middlebury computes the mean absolute error against ground-truth intermediate frames.

Dataset Preparation and Directory Structure

Before running benchmarks, download and extract the datasets to match the hardcoded paths in the evaluation scripts:

Dataset	Download Source	Target Directory (Relative to Repo Root)
Vimeo90K	toflow.csail.mit.edu	vimeo_interp_test/ containing tri_testlist.txt and target/ subfolders
UCF101	liuziwei7.github.io/projects/VoxelFlow	UCF101/ucf101_interp_ours/ with subfolders containing frame_00.png, frame_01_gt.png, frame_02.png
Middlebury	vision.middlebury.edu/flow/data	other-data/ (input frames) and other-gt-interp/ (ground-truth interpolations)

Ensure the directory structures match exactly; the benchmark scripts use glob patterns and specific filename conventions to load image triplets.

Running the Benchmarks

Execute the following commands from the repository root to evaluate the pretrained model stored in train_log/:

Vimeo90K Benchmark

python3 benchmark/Vimeo90K.py

Expected output format:

Avg PSNR: 35.615 SSIM: 0.9779

The script in benchmark/Vimeo90K.py instantiates model.RIFE.Model, loads weights from train_log/, and iterates through the Vimeo90K triplet list to compute average PSNR and SSIM.

UCF101 Benchmark

python3 benchmark/UCF101.py

Expected output format:

Avg PSNR: 35.282 SSIM: 0.9688

The benchmark/UCF101.py script follows the same pattern as Vimeo90K but adapts the data loading for the UCF101 directory structure, which organizes interpolated frames into action categories.

Middlebury Other Benchmark

python3 benchmark/MiddleBury_Other.py

Expected output format:

1.956

The benchmark/MiddleBury_Other.py script computes the Interpolation Error (mean absolute error) against the ground-truth middle frames provided in the Middlebury dataset. This metric is specific to the Middlebury evaluation protocol.

Hardware Configuration

The scripts automatically detect and utilize CUDA if available. To force CPU execution, set the environment variable:

CUDA_VISIBLE_DEVICES= python3 benchmark/Vimeo90K.py

Understanding the Evaluation Pipeline

All benchmark scripts share a common evaluation pattern implemented in the repository's core modules:

1. Model Initialization: The scripts import model.RIFE and instantiate the Model class. They load pretrained checkpoints from train_log/ using torch.load() and set the model to evaluation mode with model.eval()【/cache/repos/github.com/hzwer/eccv2022-rife/main/benchmark/Vimeo90K.py#L12-L18】.

2. Inference: For each image pair (I0, I2), the scripts call model.inference(I0, I2) to generate the intermediate frame. This method handles the optical flow estimation and frame synthesis internally.

3. Metric Calculation:

   • PSNR is calculated as (-10 \times \log_{10}(\text{MSE})) between the predicted and ground-truth frames.
   • SSIM utilizes model.pytorch_msssim.ssim_matlab, a PyTorch implementation designed to match MATLAB's SSIM calculation for consistent academic comparison.
   • IE (Middlebury) computes the mean absolute error between the synthesized frame and the ground-truth interpolation.

4. Aggregation: Results are accumulated in Python lists during iteration, and the script prints the arithmetic mean across the entire test set upon completion.

Summary

• Vimeo90K, UCF101, and Middlebury benchmarks are executed via benchmark/Vimeo90K.py, benchmark/UCF101.py, and benchmark/MiddleBury_Other.py respectively.
• Ensure datasets are downloaded and extracted to the specific directory structures expected by the scripts (vimeo_interp_test/, UCF101/ucf101_interp_ours/, other-data/).
• The evaluation pipeline automatically handles model loading, inference via model.inference(), and metric calculation (PSNR/SSIM for Vimeo90K/UCF101, IE for Middlebury).
• Use CUDA_VISIBLE_DEVICES= to force CPU execution if GPU memory is constrained.

Frequently Asked Questions

How do I download the Vimeo90K dataset for RIFE benchmarking?

Download the Vimeo90K dataset from the toflow.csail.mit.edu website. Extract the archive so that you have a vimeo_interp_test/ directory in the repository root containing tri_testlist.txt and the target/ subfolders with the test triplets.

What metrics does the Middlebury benchmark calculate for RIFE?

The Middlebury benchmark script (benchmark/MiddleBury_Other.py) calculates the Interpolation Error (IE), which is the mean absolute error between the synthesized intermediate frame and the ground-truth interpolation provided in the other-gt-interp/ directory. Unlike Vimeo90K and UCF101, Middlebury does not report PSNR or SSIM in the standard RIFE evaluation protocol.

Can I run RIFE benchmarks on CPU-only machines?

Yes, the benchmark scripts automatically detect CUDA availability and fall back to CPU execution if no GPU is present. You can also force CPU execution by setting CUDA_VISIBLE_DEVICES= before the Python command, which ensures the model runs on the CPU even if CUDA drivers are installed.

Where are the pretrained weights loaded from during benchmarking?

The benchmark scripts load pretrained weights from the train_log/ directory relative to the repository root. Specifically, they instantiate model.RIFE.Model and load the checkpoint using torch.load() to restore the trained parameters before switching the model to evaluation mode with model.eval().

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}