How to Use RIFE-ncnn-vulkan for Faster CPU Inference

RIFE-ncnn-vulkan delivers 2-3× faster CPU inference compared to the PyTorch implementation by eliminating Python overhead and using NCNN's optimized fixed-point quantization and operator fusion.

The ECCV2022-RIFE repository provides the official PyTorch implementation of Real-Time Intermediate Flow Estimation, with core model logic in model/RIFE.py and high-level inference scripts in inference_video.py and inference_img.py. While these Python scripts work well on CUDA GPUs, CPU inference is significantly slower due to the model size and interpreter overhead. For production deployments without NVIDIA hardware, the community-maintained RIFE-ncnn-vulkan project offers a C++/NCNN alternative that runs optimized inference on CPU via the NCNN backend or on GPU via Vulkan.

Why RIFE-ncnn-vulkan Outperforms PyTorch on CPU

The original implementation in model/RIFE.py loads full 32-bit weights and processes tensors through multiple Python layers, which creates substantial overhead on CPUs. In contrast, RIFE-ncnn-vulkan compiles to native C++ using the NCNN inference engine, which applies operator fusion and fixed-point quantization specifically optimized for ARM and x86 CPUs.

This architecture eliminates the Python interpreter bottleneck and reduces memory bandwidth, typically yielding a 2-3× speedup on modern 8-core processors compared to running inference_video.py with --device cpu.

Building RIFE-ncnn-vulkan from Source

The RIFE-ncnn-vulkan repository is maintained separately at nihui/rife-ncnn-vulkan. To compile the binary with CPU support:

1. Clone the repository:

git clone https://github.com/nihui/rife-ncnn-vulkan.git
cd rife-ncnn-vulkan

2. Build with CMake. The NCNN backend automatically detects CPU capabilities; Vulkan SDK is only required for GPU support:

mkdir build && cd build
cmake ..
make -j$(nproc)

This produces the rife-ncnn-vulkan executable. Pre-trained weights in NCNN format (rife-v3-hd-model.bin and rife-v3-hd-model.param) are typically available in the repository's release assets.

Running RIFE-ncnn-vulkan on CPU

By default, the binary runs on CPU when the -g (GPU) flag is omitted. NCNN automatically utilizes all available CPU cores with optimized intrinsics.

Single Image Pair Interpolation

To generate an intermediate frame at timestep t=0.5 between two input images:

./rife-ncnn-vulkan -i img0.png img1.png -o out.png -t 0.5

Batch Video Processing

For full video interpolation, extract frames with ffmpeg, process consecutive pairs, and reassemble:

# Extract frames from input video

ffmpeg -i input.mp4 -qscale:v 2 frame_%05d.png

# Interpolate between each frame pair

for i in $(seq -f "%05g" 0 $((N-2))); do
    ./rife-ncnn-vulkan \
        -i frame_${i}.png frame_$((i+1)).png \
        -o out_${i}.png \
        -t 0.5
done

# Reassemble into video

ffmpeg -framerate 60 -i out_%05d.png -c:v libx264 -pix_fmt yuv420p output.mp4

Integrating RIFE-ncnn-vulkan with Python

You can retain the high-level workflow of inference_video.py while offloading computation to the optimized binary using Python's subprocess module:

import subprocess
from pathlib import Path

BIN = Path("rife-ncnn-vulkan/build/rife-ncnn-vulkan")

def interpolate_frames(img0: Path, img1: Path, output: Path, timestep: float = 0.5):
    """Call RIFE-ncnn-vulkan binary for frame interpolation."""
    subprocess.run([
        str(BIN),
        "-i", str(img0), str(img1),
        "-o", str(output),
        "-t", str(timestep)
    ], check=True)

# Example usage inside a video processing loop

for i in range(num_frames - 1):
    interpolate_frames(
        Path(f"frame_{i:05d}.png"),
        Path(f"frame_{i+1:05d}.png"),
        Path(f"interpolated_{i:05d}.png")
    )

This hybrid approach leverages the ECCV2022-RIFE model weights and logic while achieving the CPU performance benefits of the NCNN implementation.

Summary

• RIFE-ncnn-vulkan provides a C++/NCNN implementation of the RIFE model that eliminates Python overhead and runs 2-3× faster on CPU than the PyTorch version in inference_video.py.
• Build the binary from the nihui/rife-ncnn-vulkan repository using CMake; it requires no GPU dependencies for CPU-only operation.
• Run inference by omitting the -g flag to use the optimized NCNN CPU backend with automatic multi-core utilization.
• For video workflows, combine the binary with ffmpeg for frame extraction and reassembly, or call it from Python via subprocess to maintain compatibility with existing inference_video.py pipelines.

Frequently Asked Questions

What is the difference between RIFE-ncnn-vulkan and the PyTorch implementation in ECCV2022-RIFE?

The PyTorch implementation in model/RIFE.py and inference_video.py provides the full training and inference pipeline in Python, optimized for GPU acceleration via CUDA. RIFE-ncnn-vulkan is a separate inference-only project that converts trained weights to NCNN format and implements the forward pass in C++, enabling highly optimized CPU inference through operator fusion and quantization without Python overhead.

Can RIFE-ncnn-vulkan run on macOS or Windows?

Yes. The project uses CMake for building and compiles on Linux, macOS, and Windows. On macOS, the binary can utilize Apple Silicon CPU backends via NCNN's ARM optimizations. On Windows, build with Visual Studio or MinGW. Pre-built binaries are often available in the GitHub Releases section of the nihui/rife-ncnn-vulkan repository.

How do I convert my own trained RIFE PyTorch models to NCNN format?

To use custom weights with the NCNN binary, export your PyTorch .pth checkpoint to ONNX format using torch.onnx.export, then convert the ONNX model to NCNN's .param and .bin format using the onnx2ncnn tool provided by the NCNN project. The nihui/rife-ncnn-vulkan repository typically includes documentation or scripts for converting the official RIFE-v3 weights.

Is GPU inference with RIFE-ncnn-vulkan faster than the PyTorch version?

When using the -g flag to enable Vulkan GPU acceleration, RIFE-ncnn-vulkan can achieve comparable or better performance than the PyTorch CUDA implementation on supported GPUs, often exceeding 30 FPS on laptop GPUs. However, the primary advantage of the NCNN version is its efficiency on CPU and its ability to run on devices without NVIDIA CUDA support, making it ideal for edge deployment and non-NVIDIA hardware.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}