How the PTS Shift Mechanism in render.py Achieves Precise Overlay Syncing

The PTS shift mechanism in helpers/render.py synchronizes overlays to the output timeline by resetting each overlay's timestamps to zero with PTS-STARTPTS, then adding the desired start offset in time-base units, ensuring frames align exactly with start_in_output regardless of source frame rates.

The browser-use/video-use repository provides a Python-based video rendering pipeline that composites multiple overlays onto a base video using FFmpeg. The critical synchronization logic resides in build_final_composite within helpers/render.py, where the PTS shift mechanism ensures that every overlay appears at the exact timestamp specified in the Edit Decision List (EDL).

Understanding the PTS Shift Mechanism

The PTS shift mechanism operates through a precise multi-stage pipeline that transforms overlay timestamps to match the output timeline.

Overlay Definition and EDL Structure

Each overlay entry in the EDL contains three essential fields:

• file: Path to the overlay video or image
• start_in_output: Target timestamp in seconds on the final timeline
• duration: Visibility window in seconds

These values pass unchanged from the EDL to build_final_composite (lines 42-44), serving as the foundation for timestamp calculations.

Preparing Input Streams

For every overlay, the script appends an additional -i argument to the FFmpeg command (lines 15-19). This creates separate video streams labeled [1:v], [2:v], etc., making each overlay available as an independent input stream for the filter complex.

The PTS Shift Calculation

Before compositing, each overlay stream undergoes timestamp normalization. The filter expression applied in the filter_parts loop (lines 21-25) is:

[idx:v]setpts=PTS-STARTPTS+{t}/TB[a{idx}]

This expression performs two critical operations:

1. PTS-STARTPTS: Resets the stream's presentation timestamps to zero, eliminating any temporal offset from the source file
2. +{t}/TB: Adds the target offset (t equals start_in_output) converted to time-base units (TB)

The result stores in temporary labels (a1, a2, etc.) for subsequent overlay operations.

Overlay Composition with Timeline Syncing

The shifted overlays blend onto the base video using the overlay filter with temporal gating (lines 26-36):

[current][a{idx}]overlay=enable='between(t,{t:.3f},{end:.3f})'[v{idx}]

The enable expression uses between(t,start,end) to activate the overlay only during its designated window. The chain of [v1] → [v2] → ... preserves overlay ordering, with each stage feeding into the next to build the final composite stream.

Subtitle Handling Order

After all overlays merge, subtitles process last (lines 38-44). This ordering guarantees that text renders above all visual overlays, maintaining the design hierarchy specified in the file header.

Code Examples and Implementation Details

Minimal Overlay Configuration

Consider a single overlay starting at 2.5 seconds:

{
  "overlays": [
    {
      "file": "assets/intro.mp4",
      "start_in_output": 2.5,
      "duration": 4.0
    }
  ]
}

The generated FFmpeg filter string executes three operations:

1. Adds -i assets/intro.mp4 as input stream [1:v]
2. Applies setpts=PTS-STARTPTS+2.5/TB to shift the first frame to 2.5 seconds
3. Enables the overlay via between(t,2.500,6.500) for the 4-second duration

Multiple Overlapping Overlays

For concurrent overlays with different start times:

{
  "overlays": [
    { "file": "assets/logo.mp4",   "start_in_output": 0.0,  "duration": 10.0 },
    { "file": "assets/badge.mp4", "start_in_output": 5.0,  "duration": 5.0 }
  ]
}

The resulting filter chain creates a sequential composition:

[1:v]setpts=PTS-STARTPTS+0/TB[a1];
[2:v]setpts=PTS-STARTPTS+5/TB[a2];
[0:v][a1]overlay=enable='between(t,0.000,10.000)'[v1];
[v1][a2]overlay=enable='between(t,5.000,10.000)'[v2];

The logo displays for the full 10 seconds, while the badge appears from 5 to 10 seconds, layered on top.

Command Line Execution

Process the EDL through the rendering pipeline:

python helpers/render.py project.edl.json -o final.mp4

The script automatically extracts source clips, concatenates them into base.mp4, then invokes build_final_composite to apply the PTS shift and overlay composition described above.

Summary

• The PTS shift mechanism in helpers/render.py uses setpts=PTS-STARTPTS+{t}/TB to normalize and offset overlay timestamps
• Each overlay receives independent input stream handling through FFmpeg's -i arguments (lines 15-19)
• Temporal alignment relies on time-base unit conversion (/TB) to ensure frame-accurate positioning
• The between(t,start,end) enable expression restricts visibility to the specified start_in_output and duration window
• Subtitles render last (lines 38-44) to guarantee top-layer visibility above all overlays

Frequently Asked Questions

What does PTS stand for in video processing?

PTS stands for Presentation Timestamp, a metadata field indicating when a specific video frame should display relative to the stream's timeline. The PTS shift mechanism manipulates these timestamps to align overlay frames with the output video's global timeline.

Why is PTS-STARTPTS necessary before adding the offset?

PTS-STARTPTS resets each overlay's internal clock to zero, removing any pre-existing temporal offsets from the source file. This normalization ensures that adding {t}/TB creates an absolute position on the output timeline rather than a relative shift from the source's original timing.

How does the overlay filter know when to enable and disable?

The enable='between(t,{t:.3f},{end:.3f})' expression evaluates the global output time t against the calculated start and end timestamps. When t falls within this range, FFmpeg activates the overlay; outside this window, the filter passes the input through unchanged, effectively hiding the overlay.

Can overlays have different frame rates than the base video?

Yes. The PTS shift mechanism operates independently of frame rate because it converts seconds to time-base units (TB). The setpts filter handles temporal scaling automatically, allowing overlays with 24fps, 30fps, or 60fps sources to synchronize correctly with the base video timeline regardless of rate differences.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}