Debugging Overlay Timing Problems in video-use: PTS-STARTPTS+T/TB Shifting Explained

video-use solves overlay timing drift by resetting each overlay's presentation timestamps (PTS) to zero with PTS-STARTPTS and then adding the absolute start offset using +T/TB, ensuring frame 0 aligns exactly with the intended insertion point in the final composite.

The video-use repository constructs deterministic FFmpeg filter graphs to composite multiple video overlays onto a base timeline. When concatenating clips without timestamp adjustment, cumulative offset errors cause overlays to appear earlier or later than intended. The fix implemented in helpers/render.py uses a precise PTS-shifting technique—PTS-STARTPTS+T/TB—to map each overlay into the global timeline.

How the PTS Shift Works in helpers/render.py

The rendering pipeline in helpers/render.py builds a single FFmpeg filter graph that processes each overlay entry from the EDL (Edit Decision List). According to the comment at the top of the file, the pipeline extracts raw clips, shifts timestamps so that frame 0 lands at the overlay window start, and then composites the result.

For every overlay, the code calculates the absolute start time t in seconds relative to the final output timeline. It then injects a setpts filter that rewrites the overlay stream's timestamps:

filter_parts.append(f"[{idx}:v]setpts=PTS-STARTPTS+{t}/TB[a{idx}]")

This expression performs two critical operations:

• PTS-STARTPTS resets the stream's timestamps so that the first frame becomes zero.
• +{t}/TB converts the desired start time t into timebase units and adds it to the stream, shifting the overlay to its absolute position in the final video.

After the timestamp shift, the overlay is composited over the base video using the overlay filter with a time-based enable expression:

f"{current}[a{idx}]overlay=enable='between(t,{t:.3f},{end:.3f})'{next_label}"

The between(t,start,end) expression ensures the overlay only renders while the current output time is inside the overlay's window. Because the overlay stream's PTS has already been shifted to align with the global timeline, this guard triggers at exactly the correct moment.

Why PTS-STARTPTS+T/TB Eliminates Timing Bugs

When overlays are concatenated without timestamp adjustment, FFmpeg treats each overlay as if it starts at its own zero point. After the first overlay, subsequent clips inherit the accumulated duration of previous ones, causing a cumulative offset error that pushes each overlay later than intended.

By resetting each overlay's PTS (PTS-STARTPTS) and then adding the absolute start time (+T/TB), the graph forces every overlay into the global timeline. The overlay=enable='between(t,…)' guard then acts as a reliable on/off switch that matches the intended timeline slices, eliminating drift or "early/late" rendering bugs.

Edge Case Handling in video-use

The implementation in helpers/render.py handles several timing edge cases without requiring additional logic:

• Overlay longer than base video: The end value is calculated as t + ov["duration"]. If this exceeds the base video length, the overlay filter automatically stops rendering once the base stream ends.
• Zero-length overlays: When t equals end, the between expression evaluates to false for all frames, so no overlay is drawn and no unnecessary filter nodes are processed.
• Non-monotonic EDL order: The code processes overlays in EDL sequence, but each PTS shift uses the absolute start time t. This allows temporal overlaps to resolve correctly through the overlay filter's enable expression, regardless of processing order.

Practical Implementation Example

Below is the minimal Python logic required to duplicate the timing shift in your own FFmpeg pipelines, based on the build-final-composite function in helpers/render.py:

# Assume `overlays` is a list of dicts with keys:

#   "path"      – path to the overlay video/animation

#   "start"     – start time (seconds) in the final video

#   "duration"  – how long the overlay should stay visible

for idx, ov in enumerate(overlays, start=1):
    t = ov["start"]                     # absolute start in seconds

    
    # 1️⃣ Reset and shift timestamps

    filter_parts.append(
        f"[{idx}:v]setpts=PTS-STARTPTS+{t}/TB[a{idx}]"
    )
    
    # 2️⃣ Composite over the base video, enabled only within the window

    end = t + ov["duration"]
    filter_parts.append(
        f"[base][a{idx}]overlay=enable='between(t,{t:.3f},{end:.3f})'[base]"
    )

The final composite stream [outv] contains a clean video where every graphic, animation, or caption appears at the precise moment defined in the EDL.

Summary

• PTS-STARTPTS+T/TB is the core formula used in video-use to prevent cumulative timing errors in multi-overlay videos.
• The setpts filter in helpers/render.py resets overlay timestamps to zero before adding the absolute offset, aligning frame 0 with the intended insertion point.
• The overlay filter's enable='between(t,start,end)' expression ensures precise visibility windows that match the global timeline.
• This technique handles edge cases like zero-length overlays, non-monotonic edit lists, and clips exceeding base duration without additional error-correction logic.

Frequently Asked Questions

What does PTS-STARTPTS actually do in FFmpeg?

PTS-STARTPTS subtracts the presentation timestamp of the first frame from every frame in the stream. This normalization resets the stream's timeline so that the first frame becomes zero, creating a clean starting point for timestamp arithmetic regardless of the source file's original timing.

Why is the timebase (TB) necessary in the offset calculation?

FFmpeg expressions require timestamps in the stream's internal timebase units, not seconds. Dividing the start time t (in seconds) by TB (the timebase denominator) converts the offset into the correct units for the setpts filter. Without this conversion, the shift would be scaled incorrectly, causing the overlay to appear at the wrong position.

How does video-use handle overlapping overlays?

When the EDL contains overlapping time ranges, the code processes each overlay in sequence and shifts its PTS by the absolute start time t. The overlay filter's enable='between(t,start,end)' expression determines visibility based on the global output clock, so overlapping regions naturally composite according to the filter graph's layer order (later overlays appear on top of earlier ones).

What happens if an overlay's duration extends beyond the base video?

The end timestamp is calculated as t + duration, but the overlay filter automatically stops rendering when the base video stream ends. If the overlay window exceeds the base duration, FFmpeg simply drops the remaining overlay frames without error, making the pipeline tolerant to length mismatches between assets and the final output.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}