Overlay PTS Shifting in video-use: Why Timestamps Must Be Rewritten for Proper Animation Timing

Overlay PTS shifting rewrites the presentation timestamps of animation overlays so their first frame aligns with their designated start time in the final output, preventing viewers from seeing an incorrect slice of the animation.

In the browser-use/video-use pipeline, rendering a finished video requires compositing a base clip with animation overlays—short videos that play during specific windows in the output timeline. The setpts filter performs critical timestamp manipulation to synchronize these overlays correctly. This article explains the technical necessity of PTS shifting, its implementation in helpers/render.py, and the hard rule that governs this behavior.

Why Overlay Timestamps Require Adjustment

Each overlay animation has its own internal timeline beginning at PTS = 0. When FFmpeg processes multiple inputs, it does not automatically align these timelines to the output video's clock. Without intervention, an overlay's first frame would appear at time 0 of the final render—regardless of when the overlay window actually opens.

This creates a temporal misalignment problem. Consider an overlay scheduled to play from 5.0 to 10.0 seconds in the output. If its timestamps remain unshifted, the viewer sees frame 0 at output time 0, then sees frame 5 seconds into the animation when the overlay window finally opens. The result is jarring: users see the middle of the animation instead of its beginning.

The PTS Shifting Formula

FFmpeg's setpts filter rewrites timestamps using expressions. For overlay alignment, video-use applies:

setpts=PTS-STARTPTS+<window_start>/TB

Breaking down this expression:

• PTS-STARTPTS — normalizes the stream so its first frame becomes PTS 0
• <window_start> — the overlay's start time in seconds (e.g., 5.000)
• TB — timebase (inverse of frame rate), converting seconds to timestamp units

This shifts the overlay's entire timeline forward by its designated window start time. Frame 0 now appears exactly when the overlay window opens.

Hard Rule 4: The Documentation Standard

The requirement for PTS shifting is codified as Hard Rule 4 in the project's skill definition. From SKILL.md:

"Overlays use setpts=PTS-STARTPTS+T/TB to shift the overlay's frame 0 to its window start. Otherwise you see the middle of the animation during the overlay window."

This rule appears at lines 24-26 of the skill documentation, establishing PTS shifting as a non-negotiable step in the rendering pipeline.

Implementation in helpers/render.py

The core rendering logic lives in helpers/render.py. For each overlay, the code constructs a setpts filter that incorporates the overlay's start offset:

# Extracted from helpers/render.py, lines 21-25

for idx, ov in enumerate(overlays, start=1):
    t = float(ov["start_in_output"])
    # Shift overlay so its first frame aligns with its window start

    filter_parts.append(f"[{idx}:v]setpts=PTS-STARTPTS+{t}/TB[a{idx}]")

The loop processes overlays starting from index 1 (0 is reserved for the base video). Each shifted stream receives a labeled output ([a1], [a2], etc.) for subsequent overlay operations.

Complete Filter Graph Assembly

The PTS-shifted streams feed into FFmpeg's overlay filter at precise time windows. Here is a representative command structure:

ffmpeg -i base.mp4 -i overlay1.mp4 -i overlay2.mp4 \
-filter_complex "[1:v]setpts=PTS-STARTPTS+5.000/TB[a1];
                 [2:v]setpts=PTS-STARTPTS+12.500/TB[a2];
                 [0:v][a1]overlay=enable='between(t,5.000,10.000)'[v1];
                 [v1][a2]overlay=enable='between(t,12.500,17.500)'[outv]" \
-map "[outv]" -map "0:a" -c:v libx264 final.mp4

Processing order matters critically:

1. PTS shifting (setpts) occurs first, retiming each overlay stream
2. Compositing (overlay) occurs second, applying enabled windows
3. Final mapping selects the composite video and original audio

Rendering with EDL Files

Practical usage involves an EDL (Edit Decision List) JSON file specifying overlay windows:

python helpers/render.py edl.json -o final.mp4

The Python script parses timing metadata, computes t values for each overlay, and assembles the complete filter graph automatically.

Summary

• PTS shifting aligns independent timelines — overlay animations and base video use different clocks that must be synchronized
• setpts=PTS-STARTPTS+t/TB is the standard formula — used in helpers/render.py for every overlay stream
• Hard Rule 4 mandates this behavior — documented in SKILL.md as essential for correct visual output
• Without shifting, viewers see animation midpoints — the overlay window displays incorrect content from the animation's internal timeline

Frequently Asked Questions

What happens if I omit the PTS shift in video-use rendering?

The overlay plays from its internal time 0 at output time 0, regardless of when its window opens. When the overlay window finally activates, the viewer sees whatever frame corresponds to that moment in the animation's timeline—typically the middle or end rather than the beginning. This produces visually broken output where animations appear to start mid-action.

How does STARTPTS differ from PTS in the filter expression?

PTS is the current frame's presentation timestamp. STARTPTS is the first frame's PTS of the stream. Subtracting yields a normalized timeline where the first frame equals zero. This normalization allows the +t/TB offset to work consistently regardless of the overlay's original timestamp values.

Where is the overlay timing data defined in video-use?

Timing originates from transcription metadata generated by helpers/transcribe.py, which produces word-level timestamps used to schedule overlay windows. These values populate the start_in_output field in the overlay configuration, ultimately becoming the t parameter in the setpts expression within helpers/render.py.

Can multiple overlays overlap in the same time window?

Yes. The filter graph chains overlays sequentially: [0:v][a1]overlay produces [v1], then [v1][a2]overlay produces the next stage. Each overlay maintains independent PTS shifting via its designated aN label, allowing arbitrary overlap without timing interference.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}