How Overlay PTS Shifting Ensures Precise Animation Timing in video-use

Overlay PTS shifting works by resetting each overlay's presentation timestamps to zero and then adding the desired start offset, ensuring the first frame aligns exactly with the specified output timestamp in the final composite.

The browser-use/video-use repository constructs final videos using a single FFmpeg filter graph that concatenates graded clips and composites animations on top. To guarantee that GIFs, motion graphics, or other overlays appear at their designated moments, the pipeline manipulates Presentation Timestamps (PTS) before stacking layers. This timestamp normalization is essential because overlay files often have internal timelines that do not match the master video's clock.

The Anatomy of an Overlay in the EDL

Each overlay is defined in the Edit Decision List (EDL) with two critical timing fields:

• start_in_output: The exact second in the final video where the overlay should appear
• duration: How long the overlay remains visible

A typical overlay entry looks like this:

{
  "overlays": [
    {
      "file": "assets/bounce.gif",
      "start_in_output": 12.0,
      "duration": 2.5
    }
  ]
}

These values drive the filter generation logic in helpers/render.py, where the code translates human-readable timestamps into FFmpeg filter expressions.

PTS Shifting Mechanism in render.py

The core timing correction happens in helpers/render.py (lines 22-25), where each overlay stream undergoes a setpts transformation before compositing. The code constructs a filter that resets the overlay's timeline and then shifts it forward by the desired amount:

# PTS-shift every overlay so its frame 0 lands at start_in_output

for idx, ov in enumerate(overlays, start=1):
    t = float(ov["start_in_output"])
    filter_parts.append(f"[{idx}:v]setpts=PTS-STARTPTS+{t}/TB[a{idx}]")

This expression performs two operations:

1. PTS-STARTPTS: Resets the overlay's internal timestamps to zero, stripping away any original timing metadata from the source file
2. +{t}/TB: Adds the target offset in seconds, where TB represents the timebase (the reciprocal of the frame rate), converting seconds into FFmpeg's internal time units

The result is that frame 0 of the overlay is now timestamped at exactly start_in_output seconds on the master timeline, regardless of when the source file actually begins.

Temporal Windowing with the Overlay Filter

After shifting timestamps, the pipeline uses FFmpeg's overlay filter with an enable expression to restrict visibility to the active window. In helpers/render.py (lines 27-35), the code calculates the end time and constructs the stacking filter:

for idx, ov in enumerate(overlays, start=1):
    t = float(ov["start_in_output"])
    dur = float(ov["duration"])
    end = t + dur
    next_label = f"[v{idx}]"
    filter_parts.append(
        f"{current}[a{idx}]overlay=enable='between(t,{t:.3f},{end:.3f})'{next_label}"
    )
    current = next_label

Because the overlay's PTS has already been shifted to align with the master clock, the between(t,…) condition references the same timebase as the base video. The overlay is drawn only when the master timeline reaches the specified interval, ensuring the animation appears exactly at start_in_output and disappears after duration seconds.

Complete Filter Graph Assembly

The individual filter components are joined into a complex filter graph that FFmpeg processes in a single pass. For an overlay starting at 12 seconds with a 2.5-second duration, the assembled filter looks like this:

filter_complex = ";".join([
    "[1:v]setpts=PTS-STARTPTS+12.0/TB[a1]",
    "[0:v][a1]overlay=enable='between(t,12.000,14.500)'[v1]"
])

The resulting FFmpeg command maps the base video as stream 0 and the overlay as stream 1, applying the timestamp shift and compositing operations before encoding:

ffmpeg -i base.mp4 -i assets/bounce.gif \
       -filter_complex "[1:v]setpts=PTS-STARTPTS+12.0/TB[a1];[0:v][a1]overlay=enable='between(t,12.000,14.500)'[v1]" \
       -map "[v1]" -map "0:a" -c:v libx264 -crf 18 -c:a copy final.mp4

This guarantees that the bounce animation starts precisely at 12 seconds, runs for exactly 2.5 seconds, and then disappears, maintaining perfect synchronization with the underlying video content.

Summary

• PTS normalization via setpts=PTS-STARTPTS+{t}/TB resets overlay timelines and applies the target offset before compositing
• Temporal gating using overlay=enable='between(t,start,end)' restricts visibility to the exact window specified in the EDL
• Single-pass rendering combines concatenation, overlay stacking, and optional subtitle burning in one FFmpeg filter graph
• Frame-rate independence is achieved by using the timebase (TB) conversion, allowing overlays with different frame rates to align correctly

Frequently Asked Questions

What does PTS stand for in video processing?

PTS stands for Presentation TimeStamp, a metadata field in video streams that indicates when a specific frame should be displayed relative to the start of the file. In FFmpeg workflows, manipulating PTS allows you to re-time streams without re-encoding individual frames, effectively sliding content forward or backward on the timeline.

Why is PTS-STARTPTS necessary before adding the offset?

PTS-STARTPTS subtracts the first timestamp of the stream from every frame's timestamp, effectively re-basing the overlay's timeline to start at zero. This is necessary because source files like GIFs or motion graphics may have non-zero starting timestamps or arbitrary internal clocks. Normalizing to zero ensures that when you add the start_in_output offset, the overlay's first frame lands exactly at the intended moment in the final composition.

How does video-use handle overlays with different frame rates?

The /TB component in the setpts expression divides the time offset by the timebase, which converts seconds into the stream's internal time units. Since TB (timebase) is the reciprocal of the frame rate, this calculation automatically scales the offset to accommodate overlays with different frame rates, ensuring temporal alignment regardless of whether the overlay is 30fps, 60fps, or variable frame rate.

Can multiple overlays overlap on the same timeline?

Yes. The filter graph builds a sequential chain of overlay operations, where each subsequent overlay is composited onto the result of the previous one. Each overlay receives its own independent PTS shift based on its specific start_in_output value, and each has its own between(t,…) enable window. This allows multiple animations to appear simultaneously, sequentially, or with overlapping durations while maintaining individual timing controls.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}