How to Integrate RIFE with VapourSynth for AI Video Frame Interpolation

You can integrate RIFE with VapourSynth by loading the pretrained Model class from model/RIFE.py, wrapping the inference() method to handle tensor conversion, and using std.ModifyFrame or std.Interleave to insert interpolated frames into your processing pipeline.

The hzwer/eccv2022-rife repository provides a high-performance PyTorch implementation of RIFE (Real-Time Intermediate Flow Estimation) for synthesizing intermediate frames between video frames. While the repository ships with a command-line driver, integrating RIFE directly into VapourSynth pipelines requires bridging the PyTorch model with VS's frame-based processing graph.

Understanding the RIFE Architecture

Core Model Components

The interpolation logic resides in model/RIFE.py, which defines the Model class. This class encapsulates the IFNet architecture for optical flow estimation and the fusion network for frame synthesis. The two primary methods for integration are:

• load_model(path, rank): Loads pretrained weights from flownet.pkl and associated checkpoint files in the specified directory.
• inference(im0, im1, scale): Accepts two PyTorch tensors of shape (1, 3, H, W) with values in [0, 1] and returns the interpolated middle frame.

Reference Implementation

The inference_video.py script demonstrates practical usage of the Model class. It handles video I/O, iterates through frame pairs, and contains the make_inference() helper function (lines 78-88) for recursive interpolation when generating multiple intermediate frames for 4× or 8× slow-motion.

Step-by-Step VapourSynth Integration

Step 1: Load the Pretrained Model

First, instantiate the RIFE model and load the pretrained weights. The model directory should contain flownet.pkl and related checkpoint files downloaded from the repository releases.

from model.RIFE import Model
import torch
from pathlib import Path

model_dir = Path('train_log')  # Directory containing flownet.pkl

rife = Model()
rife.load_model(str(model_dir), rank=0)
rife.eval()
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
rife.flownet.to(device)

Step 2: Wrap Model Inference for VS Frames

VapourSynth frames use the VideoFrame object with plane-based storage. Create a helper function that converts VS frames to PyTorch tensors, calls rife.inference(), and returns the result as a numpy array.

import vapoursynth as vs
import numpy as np

core = vs.core

def interpolate_pair(frame0: vs.VideoFrame, frame1: vs.VideoFrame) -> np.ndarray:
    """Generate intermediate frame between two VS frames."""
    # Convert VS planes to HxWxC numpy array

    img0 = np.stack([np.array(frame0.get_read_array(i), copy=False) 
                     for i in range(frame0.format.num_planes)], axis=2)
    img1 = np.stack([np.array(frame1.get_read_array(i), copy=False) 
                     for i in range(frame1.format.num_planes)], axis=2)
    
    # Convert to torch tensors (1,3,H,W) in [0,1]

    t0 = torch.from_numpy(img0).permute(2, 0, 1).unsqueeze(0).float().to(device) / 255.0
    t1 = torch.from_numpy(img1).permute(2, 0, 1).unsqueeze(0).float().to(device) / 255.0
    
    with torch.no_grad():
        middle = rife.inference(t0, t1, scale=1)
    
    # Convert back to numpy HxWx3 uint8

    out = (middle[0].cpu().numpy().transpose(1, 2, 0) * 255.0).clip(0, 255).astype(np.uint8)
    return out

Step 3: Inject Interpolated Frames into the Pipeline

Use VapourSynth's std.ModifyFrame or std.Interleave to insert the generated frames between original frames. For 2× interpolation, alternate between original and interpolated frames.

def rife_double_fps(clip: vs.VideoNode) -> vs.VideoNode:
    """Double the frame rate using RIFE interpolation."""
    # Cache frames to allow look-ahead

    frame_list = [clip.get_frame(i) for i in range(clip.num_frames)]
    new_frames = []
    
    for i in range(len(frame_list) - 1):
        new_frames.append(frame_list[i])
        mid_np = interpolate_pair(frame_list[i], frame_list[i+1])
        # Create VS frame from numpy (simplified - in production use proper frame creation)

        mid_frame = core.std.BlankClip(width=clip.width, height=clip.height, 
                                       length=1, format=vs.RGB24)
        new_frames.append(mid_frame)
    
    new_frames.append(frame_list[-1])
    
    # Rebuild clip

    def get_frame(n):
        return new_frames[n]
    
    return core.std.ModifyFrame(
        template=core.std.BlankClip(width=clip.width, height=clip.height, 
                                    length=len(new_frames), fpsnum=clip.fps_num*2, 
                                    fpsden=clip.fps_den, format=clip.format),
        clip=clip,
        selector=get_frame
    )

# Usage

src = core.ffms2.Source('input.mp4')
src = core.resize.Bicubic(src, format=vs.RGB24)  # RIFE requires RGB

out = rife_double_fps(src)
out.set_output()

Achieving Higher Interpolation Factors

For 4× or 8× slow-motion, implement recursive interpolation using the make_inference pattern from inference_video.py. This function generates intermediate frames between already-interpolated results.

def make_inference(I0, I1, n):
    """Recursively generate n intermediate frames between I0 and I1."""
    if n == 0:
        return []
    middle = rife.inference(I0, I1, scale=1)
    if n == 1:
        return [middle]
    # Recursively interpolate between I0-middle and middle-I1

    left = make_inference(I0, middle, n // 2)
    right = make_inference(middle, I1, n // 2)
    return left + [middle] + right

Set n to 2^exp - 1 where exp is the desired exponent (2 for 4×, 3 for 8×).

Alternative: Community VapourSynth Plugins

If you prefer not to maintain custom glue code, community plugins provide ready-made VS integration:

• vs-rife: A Python-based plugin that wraps the official implementation. Install via pip install vsrife and invoke as core.rife.RIFE(clip, model='4.6', num_frames=2).

• VapourSynth-RIFE-ncnn-Vulkan: A Vulkan-accelerated port using ncnn for GPU inference without PyTorch dependencies, ideal for systems lacking CUDA support.

Summary

• The hzwer/eccv2022-rife repository provides the Model class in model/RIFE.py for frame interpolation, with load_model() and inference() as the primary integration points.
• To integrate with VapourSynth, convert VS VideoFrame objects to PyTorch tensors, run rife.inference(), and convert the output back to numpy arrays for frame reconstruction.
• Use std.ModifyFrame or std.Interleave to inject interpolated frames into the pipeline, effectively doubling or quadrupling frame rates.
• For higher-order interpolation (4×, 8×), implement recursive inference using the make_inference helper pattern from inference_video.py.
• Community plugins like vs-rife and VapourSynth-RIFE-ncnn-Vulkan offer drop-in alternatives if you prefer not to write custom Python glue code.

Frequently Asked Questions

Can I use RIFE with VapourSynth without installing PyTorch?

While the official hzwer/eccv2022-rife implementation requires PyTorch for the Model class, you can use the VapourSynth-RIFE-ncnn-Vulkan community plugin instead. This port uses the ncnn inference engine and Vulkan compute shaders, eliminating the PyTorch and CUDA dependencies while maintaining real-time performance on compatible GPUs.

What input format does RIFE expect when integrated into VapourSynth?

RIFE expects RGB24 planar format with pixel values normalized to the range [0, 1] as PyTorch tensors of shape (1, 3, H, W). In your VapourSynth script, convert your source clip using core.resize.Bicubic(src, format=vs.RGB24) before processing. The inference() method in model/RIFE.py handles the tensor operations internally.

How do I handle memory management for long videos in VapourSynth?

For long videos, avoid caching all frames in a Python list simultaneously as shown in the basic example. Instead, implement a sliding window buffer using std.FrameEval to process frame pairs on-demand, or use the vs-rife plugin which handles frame caching internally. When using the raw Python approach, ensure you wrap inference calls in torch.no_grad() and manually delete intermediate tensors (del tensor_name) to prevent GPU memory accumulation.

Where can I find pretrained model weights for the integration?

Pretrained weights are available in the releases section of the hzwer/eccv2022-rife repository. Download the flownet.pkl and associated checkpoint files, then place them in a directory such as train_log/. In your integration script, point the load_model() method to this directory. The inference_video.py reference implementation demonstrates this pattern by defaulting to the train_log directory for checkpoint loading.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}