How PTS Shift Aligns Overlay Frame 0 with the Correct Window Start in video-use

PTS shift ensures overlay frame 0 aligns with the correct window start by resetting the overlay's presentation timestamps to zero and adding a precise time-base offset equal to the desired start time in the output timeline.

The video-use repository composes final videos by overlaying animation clips onto a base video according to an Edit Decision List (EDL). The critical mechanism that guarantees each overlay appears at its exact scheduled moment is the PTS shift implemented in the rendering pipeline.

How the PTS Shift Works in helpers/render.py

Inside helpers/render.py, the rendering logic constructs FFmpeg filter chains that synchronize each overlay with its designated timestamp. The critical filter expression that performs the shift is:

# PTS-shift every overlay so its frame 0 lands at start_in_output

filter_parts.append(f"[{idx}:v]setpts=PTS-STARTPTS+{t}/TB[a{idx}]")

This operation, found within the overlay processing loop, performs three synchronized actions:

• Stream Selection: [{idx}:v] selects the video stream from the idx-th input file. Since input 0 is the base video, overlays begin at index 1.
• Timestamp Reset and Shift: setpts=PTS-STARTPTS+{t}/TB first resets the stream's presentation timestamps so frame 0 starts at 0, then adds the offset {t}/TB where t equals overlay["start_in_output"] and TB is the stream time-base.
• Labeling: [a{idx}] tags the shifted stream for subsequent overlay operations.

Why Frame 0 Aligns with the Window Start

The alignment relies on the mathematical relationship between PTS (Presentation Timestamp) values and the composite timeline:

• PTS Normalization: PTS-STARTPTS subtracts the original start time from every frame, stripping any pre-existing timestamps from the source file and forcing the overlay's first frame to timestamp 0.
• Time-Base Conversion: Dividing the target time t by TB converts seconds into the stream's internal time-base units, ensuring FFmpeg interprets the offset in its native integer timestamp format.
• Absolute Positioning: After the shift, the overlay's frame 0 carries a PTS value exactly equal to t seconds. When the composite timeline reaches t, FFmpeg presents this frame immediately.

Integration with the Overlay Filter

After shifting timestamps, the pipeline applies the overlay filter using time-based enablement:

overlay=enable='between(t,{t:.3f},{end:.3f})'

Because the PTS shift already positioned the overlay's internal timeline at the correct offset, the between(t,...) expression activates precisely when the output clock matches start_in_output. The overlay remains visible until end (calculated as t + duration), maintaining perfect synchronization without requiring additional offset calculations within the enable expression.

Practical Implementation Example

The following Python snippet demonstrates how video-use processes an overlay list to generate synchronized filter chains:

overlays = [
    {
        "file": "assets/intro.mp4",
        "start_in_output": 5.0,  # Display at 5 seconds

        "duration": 2.0,
    }
]

filter_parts = []
for idx, ov in enumerate(overlays, start=1):
    t = ov["start_in_output"]
    # Shift PTS so frame 0 starts at t seconds

    filter_parts.append(f"[{idx}:v]setpts=PTS-STARTPTS+{t}/TB[a{idx}]")
    # Overlay only during the specified window

    end = t + ov["duration"]
    filter_parts.append(
        f"[0:v][a{idx}]overlay=enable='between(t,{t:.3f},{end:.3f})'[v{idx}]"
    )

This generates an FFmpeg command equivalent to:

ffmpeg -i base.mp4 -i assets/intro.mp4 \
   -filter_complex "[1:v]setpts=PTS-STARTPTS+5/TB[a1];
                    [0:v][a1]overlay=enable='between(t,5.000,7.000)'[outv]" \
   -map "[outv]" -map 0:a -c:v libx264 -crf 18 output.mp4

The resulting video displays intro.mp4 exactly at 5.0 seconds and hides it at 7.0 seconds, regardless of the original timestamps within the overlay source file.

Summary

• PTS shift occurs in helpers/render.py via the setpts=PTS-STARTPTS+{t}/TB FFmpeg filter.
• The shift resets overlay timestamps to zero and adds the start_in_output offset, aligning frame 0 with the target window start.
• Time-base division (/TB) ensures precise second-based positioning within FFmpeg's internal timeline.
• The overlay filter's enable='between(t,...)' expression relies on these shifted timestamps to toggle visibility at the correct moments.
• This approach guarantees frame-accurate placement even when overlay source files contain arbitrary or conflicting timestamp metadata.

Frequently Asked Questions

What does PTS-STARTPTS accomplish in the FFmpeg filter?

PTS-STARTPTS subtracts the first presentation timestamp of the stream from every frame's timestamp, effectively resetting the stream so its initial frame starts at time zero. This normalization removes any pre-existing temporal offsets from the source file, creating a clean baseline for the subsequent time shift.

Why is the time-base (TB) necessary when calculating the PTS shift?

FFmpeg stores timestamps as integer counts of time-base units rather than floating-point seconds. Dividing the target time t by TB converts the second-based start_in_output value into the correct integer units that FFmpeg's setpts filter expects, preventing rounding errors and ensuring frame-accurate positioning.

How does video-use handle multiple overlays with different start times?

The rendering function iterates through the overlay list, applying an independent PTS shift to each stream using its specific overlay["start_in_output"] value. Each shifted stream receives a unique label ([a1], [a2], etc.), and the filter chain overlays them sequentially onto the base video, allowing each to appear at its designated absolute timestamp without interfering with others.

What happens if an overlay source file contains its own timestamps?

The PTS-STARTPTS operation discards all original timestamps from the source file, treating the overlay as a raw frame sequence starting at time zero. This ensures that video-use maintains complete control over timing through the EDL's start_in_output field, preventing source file metadata from causing synchronization errors in the final composite.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}