How PTS Shifting Aligns Overlays with Correct Timestamps in video-use

video-use shifts overlay Presentation Time Stamps (PTS) using FFmpeg's setpts=PTS-STARTPTS+T/TB filter to ensure animation frame 0 renders exactly at the start_in_output timestamp defined in the EDL, preventing desynchronized animations that would otherwise show mid-sequence frames.

The browser-use/video-use repository processes video through three distinct stages: per-segment extraction, lossless concatenation, and final compositing. During the compositing phase in helpers/render.py, the pipeline must align externally rendered animations—whose internal timelines always begin at zero—with specific moments in the output video. This article explains how PTS shifting solves this alignment challenge without trimming or re-encoding source material.

The Overlay Timing Problem

When compositing multiple video layers, overlay source files contain animations that start at timestamp 0 internally. If placed directly onto the output timeline without adjustment, frame 0 of the overlay would appear at frame 0 of the final video, regardless of when the overlay should actually appear. This causes animations to display their middle frames instead of starting from the beginning when the overlay window opens.

PTS Shifting Implementation in helpers/render.py

The compositing step implements a precise timestamp adjustment using FFmpeg's setpts filter. For each overlay entry, the pipeline calculates a PTS shift using the formula:

[<idx>:v]setpts=PTS-STARTPTS+<t>/TB[a<idx>]

Where:

• PTS-STARTPTS resets the overlay's timeline to start at zero
• t represents the start_in_output value from the EDL entry
• TB is the time base (inverse of frame rate)

This operation occurs before the overlay filter receives the stream. The Python implementation in helpers/render.py constructs this filter chain dynamically:

# PTS‑shift every overlay so its frame 0 lands at start_in_output

for idx, ov in enumerate(overlays, start=1):
    t = float(ov["start_in_output"])
    filter_parts.append(f"[{idx}:v]setpts=PTS-STARTPTS+{t}/TB[a{idx}]")

# Chain overlays on top of the base clip

current = "[0:v]"
for idx, ov in enumerate(overlays, start=1):
    t = float(ov["start_in_output"])
    dur = float(ov["duration"])
    end = t + dur
    next_label = f"[v{idx}]"
    filter_parts.append(
        f"{current}[a{idx}]overlay=enable='between(t,{t:.3f},{end:.3f})'{next_label}"
    )
    current = next_label

EDL Structure and start_in_output

The Edit Decision List (EDL) provides the timing data driving the PTS shift calculations. Each overlay entry includes a start_in_output field that specifies exactly when the animation should begin in the final video:

{
  "file": "edit/animations/slot_1/render.mp4",
  "start_in_output": 12.5,
  "duration": 4.0
}

According to the SKILL.md documentation, this field is critical for Rule 4: overlays must use setpts=PTS-STARTPTS+T/TB to avoid showing the middle of an animation during its intended window. The duration parameter works with the calculated end time (t + dur) to constrain the overlay filter's visibility window.

Complete Filter Graph Execution

The final FFmpeg command combines the base video with PTS-shifted overlays. The overlay filter uses the enable='between(t,start,end)' parameter to activate only during the specified time window, while the shifted PTS ensures frame 0 appears at exactly start_in_output:

ffmpeg -i base.mp4 -i overlay.mp4 -filter_complex \
"[1:v]setpts=PTS-STARTPTS+12.5/TB[a1];
 [0:v][a1]overlay=enable='between(t,12.500,16.500)'[outv]" \
 -map "[outv]" -c:v libx264 -preset fast -crf 18 final.mp4

In this example, the overlay's internal frame 0 maps to 12.5 seconds in the output, while the overlay filter limits visibility to the 4-second window between 12.5 and 16.5 seconds.

Summary

• PTS shifting uses setpts=PTS-STARTPTS+t/TB to remap overlay timestamps to the output timeline in helpers/render.py
• The start_in_output field in EDL entries defines the exact offset applied to align animation start points
• Frame 0 of each overlay aligns precisely with its designated start time, preventing mid-animation entry
• The overlay filter's enable='between(t,start,end)' parameter constrains visibility to the calculated duration window
• This implementation follows Rule 4 documented in SKILL.md, ensuring synchronization with audio narration

Frequently Asked Questions

What does PTS stand for in video processing?

PTS stands for Presentation Time Stamp, a metadata field in video streams that indicates exactly when a specific frame should be displayed to the viewer relative to the stream's start. In the context of video-use, manipulating PTS values allows the pipeline to reschedule when overlay frames appear without altering the actual frame data or requiring re-rendering of the source animation.

Why does video-use shift PTS instead of trimming the overlay video?

Trimming would permanently remove frames from the beginning of the animation, potentially cutting off intended content, whereas PTS shifting preserves the complete animation sequence while delaying its presentation. The setpts filter mathematically remaps the timeline so that frame 0 of the overlay corresponds to the start_in_output timestamp in the final composite, ensuring the animation plays from its beginning exactly when intended.

How are multiple overlays with different start times handled?

Each overlay receives an independent PTS calculation based on its specific start_in_output value. The helpers/render.py script iterates through the overlay list, assigning sequential input indices and generating unique filter labels (e.g., [a1], [a2]), then chains them sequentially onto the base video. Each overlay maintains its own timing offset and duration constraints within the single filter graph.

What happens if the PTS shift is omitted from the filter chain?

Without PTS shifting, the overlay's frame 0 would map to time 0 of the output video, causing the animation to display whatever frame happens to correspond to the start_in_output moment in its internal timeline. According to SKILL.md Rule 4, this results in showing the middle of the animation rather than the beginning, breaking synchronization with spoken narration or other visual cues that expect the animation to start fresh at its designated window.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}