# How to Run Benchmarks on Vimeo90K, UCF101, and MiddleBury Datasets with RIFE

> Learn how to run benchmarks on Vimeo90K, UCF101, and MiddleBury datasets using RIFE. Follow simple steps with dedicated Python scripts to evaluate model performance.

- Repository: [hzwer/eccv2022-rife](https://github.com/hzwer/eccv2022-rife)
- Tags: how-to-guide
- Published: 2026-03-03

---

**You can evaluate the RIFE model on Vimeo90K, UCF101, and MiddleBury datasets by executing the dedicated Python scripts in the `benchmark/` directory after placing the downloaded datasets in the expected folder structure.**

The `hzwer/eccv2022-rife` repository provides standardized evaluation scripts that compute **PSNR**, **SSIM**, and **Interpolation Error (IE)** to measure the quality of frame interpolation. These benchmarks allow you to reproduce the paper's results or validate custom model modifications against established video interpolation datasets.

## Benchmark Overview and Metrics

The RIFE benchmark suite targets three distinct test sets, each emphasizing different aspects of temporal consistency and motion handling:

| Dataset | Primary Metrics | Benchmark Script |
|---------|----------------|------------------|
| **Vimeo90K** | PSNR, SSIM | [`benchmark/Vimeo90K.py`](https://github.com/hzwer/eccv2022-rife/blob/main/benchmark/Vimeo90K.py) |
| **UCF101** | PSNR, SSIM | [`benchmark/UCF101.py`](https://github.com/hzwer/eccv2022-rife/blob/main/benchmark/UCF101.py) |
| **Middlebury (Other)** | Interpolation Error (IE) | [`benchmark/MiddleBury_Other.py`](https://github.com/hzwer/eccv2022-rife/blob/main/benchmark/MiddleBury_Other.py) |

**PSNR** (Peak Signal-to-Noise Ratio) calculates \(-10 \log_{10}(\text{MSE})\) on raw RGB values to quantify pixel-level accuracy. **SSIM** (Structural Similarity Index) uses the MATLAB-compatible implementation in `model.pytorch_msssim.ssim_matlab` to assess perceptual quality. **IE** (Interpolation Error) for Middlebury computes the mean absolute error against ground-truth intermediate frames.

## Dataset Preparation and Directory Structure

Before running benchmarks, download and extract the datasets to match the hardcoded paths in the evaluation scripts:

| Dataset | Download Source | Target Directory (Relative to Repo Root) |
|---------|----------------|------------------------------------------|
| **Vimeo90K** | [toflow.csail.mit.edu](http://toflow.csail.mit.edu/) | `vimeo_interp_test/` containing [`tri_testlist.txt`](https://github.com/hzwer/eccv2022-rife/blob/main/tri_testlist.txt) and `target/` subfolders |
| **UCF101** | [liuziwei7.github.io/projects/VoxelFlow](https://liuziwei7.github.io/projects/VoxelFlow) | `UCF101/ucf101_interp_ours/` with subfolders containing `frame_00.png`, `frame_01_gt.png`, `frame_02.png` |
| **Middlebury** | [vision.middlebury.edu/flow/data](https://vision.middlebury.edu/flow/data/) | `other-data/` (input frames) and `other-gt-interp/` (ground-truth interpolations) |

Ensure the directory structures match exactly; the benchmark scripts use glob patterns and specific filename conventions to load image triplets.

## Running the Benchmarks

Execute the following commands from the repository root to evaluate the pretrained model stored in `train_log/`:

### Vimeo90K Benchmark

```bash
python3 benchmark/Vimeo90K.py

```

**Expected output format:**

```

Avg PSNR: 35.615 SSIM: 0.9779

```

The script in [`benchmark/Vimeo90K.py`](https://github.com/hzwer/eccv2022-rife/blob/main/benchmark/Vimeo90K.py) instantiates `model.RIFE.Model`, loads weights from `train_log/`, and iterates through the Vimeo90K triplet list to compute average PSNR and SSIM.

### UCF101 Benchmark

```bash
python3 benchmark/UCF101.py

```

**Expected output format:**

```

Avg PSNR: 35.282 SSIM: 0.9688

```

The [`benchmark/UCF101.py`](https://github.com/hzwer/eccv2022-rife/blob/main/benchmark/UCF101.py) script follows the same pattern as Vimeo90K but adapts the data loading for the UCF101 directory structure, which organizes interpolated frames into action categories.

### Middlebury Other Benchmark

```bash
python3 benchmark/MiddleBury_Other.py

```

**Expected output format:**

```

1.956

```

The [`benchmark/MiddleBury_Other.py`](https://github.com/hzwer/eccv2022-rife/blob/main/benchmark/MiddleBury_Other.py) script computes the Interpolation Error (mean absolute error) against the ground-truth middle frames provided in the Middlebury dataset. This metric is specific to the Middlebury evaluation protocol.

### Hardware Configuration

The scripts automatically detect and utilize CUDA if available. To force CPU execution, set the environment variable:

```bash
CUDA_VISIBLE_DEVICES= python3 benchmark/Vimeo90K.py

```

## Understanding the Evaluation Pipeline

All benchmark scripts share a common evaluation pattern implemented in the repository's core modules:

1. **Model Initialization**: The scripts import `model.RIFE` and instantiate the `Model` class. They load pretrained checkpoints from `train_log/` using `torch.load()` and set the model to evaluation mode with `model.eval()`【/cache/repos/github.com/hzwer/eccv2022-rife/main/benchmark/Vimeo90K.py#L12-L18】.

2. **Inference**: For each image pair `(I0, I2)`, the scripts call `model.inference(I0, I2)` to generate the intermediate frame. This method handles the optical flow estimation and frame synthesis internally.

3. **Metric Calculation**: 
   - **PSNR** is calculated as \(-10 \times \log_{10}(\text{MSE})\) between the predicted and ground-truth frames.
   - **SSIM** utilizes `model.pytorch_msssim.ssim_matlab`, a PyTorch implementation designed to match MATLAB's SSIM calculation for consistent academic comparison.
   - **IE** (Middlebury) computes the mean absolute error between the synthesized frame and the ground-truth interpolation.

4. **Aggregation**: Results are accumulated in Python lists during iteration, and the script prints the arithmetic mean across the entire test set upon completion.

## Summary

- **Vimeo90K**, **UCF101**, and **Middlebury** benchmarks are executed via [`benchmark/Vimeo90K.py`](https://github.com/hzwer/eccv2022-rife/blob/main/benchmark/Vimeo90K.py), [`benchmark/UCF101.py`](https://github.com/hzwer/eccv2022-rife/blob/main/benchmark/UCF101.py), and [`benchmark/MiddleBury_Other.py`](https://github.com/hzwer/eccv2022-rife/blob/main/benchmark/MiddleBury_Other.py) respectively.
- Ensure datasets are downloaded and extracted to the specific directory structures expected by the scripts (`vimeo_interp_test/`, `UCF101/ucf101_interp_ours/`, `other-data/`).
- The evaluation pipeline automatically handles model loading, inference via `model.inference()`, and metric calculation (PSNR/SSIM for Vimeo90K/UCF101, IE for Middlebury).
- Use `CUDA_VISIBLE_DEVICES=` to force CPU execution if GPU memory is constrained.

## Frequently Asked Questions

### How do I download the Vimeo90K dataset for RIFE benchmarking?

Download the Vimeo90K dataset from the [toflow.csail.mit.edu](http://toflow.csail.mit.edu/) website. Extract the archive so that you have a `vimeo_interp_test/` directory in the repository root containing [`tri_testlist.txt`](https://github.com/hzwer/eccv2022-rife/blob/main/tri_testlist.txt) and the `target/` subfolders with the test triplets.

### What metrics does the Middlebury benchmark calculate for RIFE?

The Middlebury benchmark script ([`benchmark/MiddleBury_Other.py`](https://github.com/hzwer/eccv2022-rife/blob/main/benchmark/MiddleBury_Other.py)) calculates the **Interpolation Error (IE)**, which is the mean absolute error between the synthesized intermediate frame and the ground-truth interpolation provided in the `other-gt-interp/` directory. Unlike Vimeo90K and UCF101, Middlebury does not report PSNR or SSIM in the standard RIFE evaluation protocol.

### Can I run RIFE benchmarks on CPU-only machines?

Yes, the benchmark scripts automatically detect CUDA availability and fall back to CPU execution if no GPU is present. You can also force CPU execution by setting `CUDA_VISIBLE_DEVICES=` before the Python command, which ensures the model runs on the CPU even if CUDA drivers are installed.

### Where are the pretrained weights loaded from during benchmarking?

The benchmark scripts load pretrained weights from the `train_log/` directory relative to the repository root. Specifically, they instantiate `model.RIFE.Model` and load the checkpoint using `torch.load()` to restore the trained parameters before switching the model to evaluation mode with `model.eval()`.