PTS Shift Mechanism for Overlay Animations in video-use: Aligning Frame 0 with Window Start

The PTS shift mechanism resets overlay timestamps to zero and adds the window start time, ensuring frame 0 of an animation appears exactly when the overlay window begins in the final output.

The browser-use/video-use repository composites final videos through extract, concat, overlay, and subtitle stages. When overlay animations are introduced, their internal timelines start at 0 seconds while the target window in the output may begin later. The PTS shift mechanism for overlay animations solves this temporal misalignment by mathematically adjusting presentation timestamps before the overlay filter processes the streams.

Why PTS Shifting Is Required for Overlay Synchronization

Without timestamp adjustment, FFmpeg would display the middle of an animation at the window start because the overlay's original PTS begins at 0. The overlay filter places incoming frames at their specified timestamps. If an animation starts at 0 seconds but the output window begins at 5 seconds, the filter would show the frame at 5 seconds into the animation rather than frame 0. This creates the "middle of the animation" artifact described in Hard Rule 4 of SKILL.md.

How the PTS Shift Mechanism Works

The mechanism operates through a mathematical transformation in FFmpeg's setpts filter.

The FFmpeg setpts Filter Syntax

The core expression is:

setpts=PTS-STARTPTS+<T>/TB

Where:

• PTS-STARTPTS resets the stream's timestamps so the first frame becomes 0.
• <T> represents the start time of the overlay window in the output timeline (ov["start_in_output"]).
• TB is the time-base of the stream (e.g., 1/24 for 24 fps).

Dividing <T> by TB converts the offset into the correct time-base units. Adding this offset shifts the entire animation forward, placing frame 0 at the exact moment the overlay window should appear.

Implementation in render.py

In helpers/render.py, the build_final_composite function constructs this filter for each overlay. The code iterates through overlay metadata and applies the timestamp shift before layering (lines 21-24).

filter_parts = []
for idx, ov in enumerate(overlays, start=1):
    t = float(ov["start_in_output"])
    filter_parts.append(f"[{idx}:v]setpts=PTS-STARTPTS+{t}/TB[a{idx}]")

This generates labeled outputs like [a1], [a2], each representing a time-shifted overlay stream ready for compositing.

Complete Code Implementation

The full pipeline collects overlay metadata including start_in_output, duration, and file paths. It builds a filter graph that first shifts timestamps, then chains overlays onto the base video using the enable='between(t,<T>,<end>)' expression (lines 5-66).

# Build PTS-shift for each overlay

filter_parts: list[str] = []
for idx, ov in enumerate(overlays, start=1):
    t = float(ov["start_in_output"])
    filter_parts.append(f"[{idx}:v]setpts=PTS-STARTPTS+{t}/TB[a{idx}]")

# Chain overlays on base video

current = "[0:v]"
for idx, ov in enumerate(overlays, start=1):
    t = float(ov["start_in_output"])
    dur = float(ov["duration"])
    end = t + dur
    next_label = f"[v{idx}]"
    filter_parts.append(
        f"{current}[a{idx}]overlay=enable='between(t,{t:.3f},{end:.3f})'{next_label}"
    )
    current = next_label

The shifted streams are layered using the overlay filter with time-based enabling, restricting visibility to the specified window. According to the video-use hard rules documented in SKILL.md, this sequence prevents displaying mid-animation frames at window start.

A minimal FFmpeg command demonstrating the same principle:

ffmpeg -i base.mp4 -i overlay.mp4 \
-filter_complex \
"[1:v]setpts=PTS-STARTPTS+5/TB[a1]; \
 [0:v][a1]overlay=enable='between(t,5,8)'" \
-c:v libx264 -crf 18 -c:a copy final.mp4

Here, overlay.mp4 starts at 0 seconds internally, but the setpts filter adds 5 seconds, causing frame 0 to appear at 5 seconds in the final output.

Summary

• PTS-STARTPTS resets overlay timestamps to zero, establishing a clean baseline for frame synchronization.
• Time-base conversion (<T>/TB) mathematically shifts the animation forward by the exact window offset.
• Pre-filter shifting occurs before the overlay filter in helpers/render.py, ensuring frame 0 aligns with window start rather than displaying mid-animation content.
• Window restriction uses enable='between(t,<T>,<end>)' to limit overlay visibility to the intended duration.
• The implementation follows the hard rule documented in SKILL.md to prevent timing artifacts in the final composite.

Frequently Asked Questions

What does PTS-STARTPTS do in FFmpeg?

PTS-STARTPTS subtracts the first presentation timestamp from all subsequent timestamps in the stream, effectively resetting the timeline so that the first frame equals zero. This normalization is essential for overlay animations that must align with specific start times in the output video.

Why not simply use the overlay filter without shifting timestamps?

Without shifting, the overlay filter would place the animation at its original timestamps. If the animation starts at 0 seconds but the output window begins at 5 seconds, the filter would display the frame at 5 seconds into the animation rather than the first frame. This creates a misalignment where the middle of the animation appears at the window start.

How does video-use calculate the time-base (TB) value?

The time-base (TB) represents the stream's frame rate denominator (e.g., 1/24 for 24 fps). FFmpeg automatically derives this from the input stream's codec parameters. The expression <T>/TB converts the time offset into the stream's native time-base units, ensuring mathematical precision when shifting timestamps.

Can this mechanism handle multiple overlapping overlay windows?

Yes. The build_final_composite function in helpers/render.py processes each overlay independently, generating unique labels like [a1], [a2], and chaining them sequentially onto the base video. Each overlay receives its own PTS shift calculation based on its specific start_in_output value, allowing multiple animations to align correctly with their respective windows regardless of overlap.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}