How RIFE Implements Arbitrary-Timestep Video Frame Interpolation Using Recursive Inference

RIFE achieves arbitrary-timestep interpolation by conditioning its flow-estimation network on a continuous time-offset input and recursively bisecting the interval between two frames to generate any number of intermediate frames.

The RIFE (Realtime Intermediate Flow Estimation) framework, available in the hzwer/eccv2022-rife repository, enables high-quality video frame interpolation at any temporal position between two input frames. Unlike fixed midpoint interpolation, arbitrary-timestep video frame interpolation requires the model to predict motion and appearance for continuous time values. RIFE solves this through a time-aware network architecture combined with a recursive inference strategy that optimizes both quality and computational efficiency.

Time-Offset Conditioning in IFNet_m

When initialized with arbitrary=True, the RIFE model loads the IFNet_m architecture from model/IFNet_m.py instead of the standard IFNet. This variant accepts a continuous timestep parameter that conditions the entire flow estimation process.

Continuous Time Encoding

The network receives the normalized time offset t ∈ [0, 1] as an additional channel broadcast to the full spatial dimensions. In the forward method of IFNet_m, the timestep tensor is concatenated with the input frames:

# IFNet_m.forward (lines 64-66)

timestep = (x[:, :1].clone() * 0 + 1) * timestep      # broadcast scalar to full-size map

img0 = x[:, :3]
img1 = x[:, 3:6]
...

# First block receives the time map together with the two input frames

flow, mask = stu[i](torch.cat((img0, img1, timestep), 1), None, scale=scale[i])

This conditioning allows the network to learn a continuous mapping from (I₀, I₁, t) → (optical flow, blending mask), enabling direct prediction for any intermediate moment rather than just the midpoint.

The high-level Model class in model/RIFE.py forwards this timestep parameter during inference:

# Model.inference (model/RIFE.py, lines 56-66)

flow, mask, merged, flow_teacher, merged_teacher, loss_distill = \
    self.flownet(imgs, scale_list, timestep=timestep)

When timestep is omitted, the network defaults to 0.5 for standard midpoint interpolation. Any other value produces an arbitrary-timestep frame directly.

Recursive Bisection for Multiple Arbitrary Frames

For generating multiple intermediate frames (e.g., 2ⁿ–1 frames for 2ⁿ-fold slow-motion), RIFE employs a recursive bisection strategy rather than invoking the network separately for each timestamp. This approach, implemented in inference_video.py, minimizes forward passes while maintaining temporal coherence.

The Recursive Inference Strategy

The make_inference function recursively subdivides the temporal interval:

# make_inference (inference_video.py, lines 78-88)

def make_inference(I0, I1, n):
    middle = model.inference(I0, I1, args.scale)   # default t=0.5

    if n == 1:
        return [middle]
    first_half  = make_inference(I0, middle, n=n//2)
    second_half = make_inference(middle, I1, n=n//2)
    if n % 2:
        return [*first_half, middle, *second_half]
    else:
        return [*first_half, *second_half]

The algorithm follows these steps:

1. Base case: Generate the midpoint frame using t = 0.5.
2. Recursive step: Split the interval [I₀, I₁] into two sub-intervals and repeat the process on each half.
3. Composition: Merge the results from both halves, inserting the middle frame when necessary.

Because each recursive call leverages the time-offset-conditioned network, the final sequence corresponds to exactly the requested arbitrary timestamps (e.g., ¼, ¾, ⅛). This method requires only O(log₂ N) forward passes for N frames, significantly reducing computational overhead compared to direct per-frame inference.

End-to-End Implementation Flow

The complete arbitrary-timestep pipeline involves three distinct stages:

1. Model initialization: model = Model(arbitrary=True) selects IFNet_m and enables time conditioning.
2. Single-frame inference: model.inference(I0, I1, timestep=0.3) returns the frame at 30% of the temporal interval.
3. Multi-frame generation: make_inference(I0, I1, n) recursively constructs sequences of arbitrary length.

This architecture ensures that RIFE can handle both single arbitrary-timestep queries and complex multi-frame interpolation tasks efficiently.

Complete Working Example

The following implementation demonstrates loading the arbitrary-timestep model and generating both single and multiple intermediate frames:

# -------------------------------------------------

# 1. Load the arbitrary-timestep model

# -------------------------------------------------

from model.RIFE import Model
model = Model(arbitrary=True)        # internally selects IFNet_m

model.load_model('train_log', -1)
model.eval()
model.device()

# -------------------------------------------------

# 2. Interpolate a single arbitrary frame (t = 0.2)

# -------------------------------------------------

I0 = torch.randn(1, 3, 720, 1280).to(device)   # first frame tensor

I1 = torch.randn(1, 3, 720, 1280).to(device)   # second frame tensor

mid = model.inference(I0, I1, timestep=0.2)      # frame at 20% of the interval

# mid is a tensor of shape (1, 3, H, W)

# -------------------------------------------------

# 3. Generate 7 intermediate frames (2³-fold slow-motion)

# -------------------------------------------------

def make_inference(I0, I1, n):
    middle = model.inference(I0, I1)            # default t=0.5

    if n == 1:
        return [middle]
    first = make_inference(I0, middle, n // 2)
    second = make_inference(middle, I1, n // 2)
    return [*first, middle, *second] if n % 2 else [*first, *second]

frames = make_inference(I0, I1, n=7)   # returns a list of 7 tensors

Summary

• Time-offset conditioning: The IFNet_m architecture in model/IFNet_m.py accepts a continuous timestep parameter that broadcasts to a full spatial map, enabling direct prediction of optical flow and blending masks at any temporal position.
• Recursive bisection: The make_inference function in inference_video.py recursively halves temporal intervals, generating multiple arbitrary-timestep frames with O(log₂ N) forward passes rather than linear complexity.
• Flexible API: Setting arbitrary=True during model initialization switches from standard midpoint interpolation to the time-aware architecture, supporting both single-frame queries (timestep=0.3) and complex sequence generation.

Frequently Asked Questions

What is the difference between IFNet and IFNet_m in RIFE?

The standard IFNet architecture performs fixed midpoint interpolation between two frames. IFNet_m, located in model/IFNet_m.py, extends this capability by accepting a continuous timestep parameter that conditions the flow estimation process. When arbitrary=True is passed to the Model constructor, RIFE automatically loads IFNet_m to enable arbitrary-timestep interpolation.

How does the timestep parameter affect interpolation quality?

The timestep parameter represents the normalized temporal position between two input frames, where 0.0 corresponds to the first frame and 1.0 to the second. The network learns to predict optical flow and blending masks conditioned on this continuous value, allowing it to generate physically plausible intermediate frames at any sub-frame precision. The quality depends on the network's ability to generalize across the continuous time domain during training.

Why does RIFE use recursive bisection instead of direct inference for multiple frames?

Recursive bisection reduces the computational complexity from O(N) to O(log₂ N) forward passes for generating N intermediate frames. By recursively interpolating the midpoint and then processing sub-intervals, RIFE reuses intermediate results and maintains temporal consistency across the sequence. This approach is implemented in inference_video.py and is particularly efficient for generating 2ⁿ–1 intermediate frames (e.g., 7 frames for 8× slow-motion).

What is the computational complexity of generating N arbitrary frames?

Using the recursive strategy in make_inference, generating N intermediate frames requires approximately ⌈log₂(N+1)⌉ forward passes through the network. For example, producing 7 intermediate frames (8× slow-motion) requires only 3 recursive levels (midpoint, then quarters, then eighths), totaling 7 forward passes rather than 7 separate inferences. This logarithmic scaling makes high-frame-rate interpolation computationally feasible.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}