How Animation Payoff Timing Synchronizes Overlays with Narration in Video‑Use

Animation payoff timing synchronizes overlays by starting the animation reveal_duration seconds before the spoken payoff word lands, ensuring the final frame coincides exactly with the narration.

The browser-use/video-use repository automates video production by aligning animated overlays with spoken narration through a deterministic timing system. This animation payoff timing mechanism ensures that visual reveals hit precisely when the narrator speaks key words, creating a polished, professional viewing experience. The synchronization relies on timestamp extraction from transcripts and precise FFmpeg filter-graph manipulation.

The Payoff Timing Rule

According to the canonical production rules in SKILL.md (lines 224–226), Video‑Use implements a three-step synchronization process that anchors overlay animations to specific narration timestamps.

Identify the Payoff Word

During transcript processing, the system scans takes_packed.md (generated by helpers/pack_transcripts.py) to locate the exact timestamp of the word carrying the visual cue. This becomes the anchor point for the entire animation sequence.

Compute the Reveal Start

The overlay does not start at the payoff word itself. Instead, the system calculates the reveal start by subtracting the reveal duration (the time required for the animation to complete its visual buildup) from the payoff timestamp. This computed value determines when the overlay clip begins in the final timeline.

Insert the Overlay

The calculated reveal start time is stored in the Edit Decision List (edl.json) within the overlays array as the start_in_output value. This entry tells the rendering pipeline exactly when to introduce the overlay video into the composite.

FFmpeg PTS-Shift Implementation

The actual synchronization happens in helpers/render.py through FFmpeg's setpts filter. As documented in SKILL.md (lines 25–27), the overlay video uses the expression setpts=PTS-STARTPTS+T/TB to shift its frame 0 to the start of its window. This PTS manipulation ensures that regardless of the source video's time-base, the overlay's final frame lands exactly at the payoff timestamp.

Practical Implementation

Here is how to implement animation payoff timing in your Video‑Use workflow.

Extracting the Payoff Timestamp

First, parse the packed transcript to identify the target word's timestamp:

# helpers/pack_transcripts.py creates takes_packed.md with lines like:

# [012.34-015.67] S0 This is the *key* concept we want to illustrate.

payoff_word = "concept"
payoff_ts = 14.5   # seconds – extracted from transcript parser

Configuring the EDL Entry

Calculate the start_in_output value by subtracting the reveal duration from the payoff timestamp:

{
  "overlays": [
    {
      "file": "edit/animations/slot_5/render.mp4",
      "start_in_output": 13.0,          // 14.5 (payoff_ts) - 1.5 (reveal_duration)
      "duration": 2.0                   // total overlay length
    }
  ]
}

The overlay begins 1.5 seconds before the word "concept" is spoken, ensuring the payoff frame appears exactly at 14.5 seconds.

Rendering with FFmpeg

Execute the render pipeline:

python helpers/render.py edl.json -o final.mp4 --build-subtitles

The underlying FFmpeg command constructed by render.py applies the PTS shift:

ffmpeg -i src.mp4 -i edit/animations/slot_5/render.mp4 \
  -filter_complex "[1:v]setpts=PTS-STARTPTS+13/TB[v1]; \
                   [0:v][v1]overlay=shortest=1[vid]" \
  -c:v libx264 -crf 20 -c:a copy final.mp4

Python Helper Function

Use this utility to compute overlay start times:

def overlay_start(payoff_ts: float, reveal_dur: float) -> float:
    """Return the time the overlay should start so its final frame syncs with payoff."""
    return max(0.0, payoff_ts - reveal_dur)

# Example usage:

start = overlay_start(14.5, 1.5)   # → 13.0 seconds

Key Files for Animation Payoff Timing

• SKILL.md: Contains the explicit payoff-timing rule (lines 224–226) and PTS-shift documentation (lines 25–27).
• helpers/render.py: Implements the FFmpeg filter-graph construction.
• helpers/pack_transcripts.py: Generates timestamped transcripts.
• edl.json: Stores overlay entries with start_in_output values.

Summary

• Animation payoff timing synchronizes overlays by calculating start_in_output = payoff_timestamp - reveal_duration.
• The system extracts payoff word timestamps from transcripts processed by helpers/pack_transcripts.py.
• FFmpeg's setpts=PTS-STARTPTS+T/TB expression aligns the overlay's frame 0 to its window start.
• helpers/render.py orchestrates the final composite using the EDL's overlays array.
• Without this synchronization, animations appear disconnected from narration; with it, the final frame lands exactly on the spoken word.

Frequently Asked Questions

What is the reveal duration in animation payoff timing?

The reveal duration is the time required for an animation to complete its visual buildup before reaching the final "payoff" frame. Video‑Use subtracts this duration from the payoff word's timestamp to determine when the overlay should begin, ensuring the animation finishes exactly when the word is spoken.

How does FFmpeg synchronize the overlay without re-encoding the source?

Video‑Use uses FFmpeg's setpts filter with the expression PTS-STARTPTS+T/TB to temporally shift the overlay video stream. This PTS manipulation aligns the overlay's first frame to the calculated start_in_output time without altering the source video's time-base, allowing frame-accurate compositing through the overlay filter.

Where is the animation payoff timing rule documented?

The canonical rule resides in SKILL.md at lines 224–226 of the browser-use/video-use repository. This documentation specifies that the overlay must start reveal_duration seconds before the payoff word to achieve perfect synchronization with the narration.

Can I use animation payoff timing with pre-rendered video clips?

Yes. Video‑Use treats animations as independent video clips (such as Manim, HyperFrames, or Remotion renders) stored in edit/animations/slot_*/render.mp4. The payoff timing calculation works with any pre-rendered overlay, as the system only manipulates the temporal placement via the EDL's start_in_output field and FFmpeg PTS shifting.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}