How Video-Use Verifies Rendered Output Against Source Footage Using `timeline_view`

Video-use compares rendered videos against source footage by running helpers/timeline_view.py twice—once per clip—to generate matching PNG visualizations that expose frame drift, audio mismatches, and transcript misalignment.

The browser-use/video-use repository provides a deterministic verification process that lets developers audit rendered media against original sources. By leveraging the timeline_view helper script, you create side-by-side visual fingerprints of both clips, making discrepancies immediately visible without scrubbing through video timelines.

How timeline_view Generates Verification Visuals

The verification process centers on helpers/timeline_view.py, which renders a composite PNG containing three diagnostic layers. When you invoke the script with identical start and end timestamps for both source and rendered clips, it produces directly comparable images.

Extracting the Film-Strip Frame Sequence

The script calls extract_frames (lines 37–62) to pull evenly spaced frames using FFmpeg. By specifying --n-frames, you control the granularity of the film-strip—typically 8–12 thumbnails per clip. These frames serve as the visual anchor for verifying that no frames were dropped, duplicated, or reordered during rendering.

Computing the Audio RMS Envelope

Audio verification relies on compute_envelope (lines 68–110). The script first dumps a temporary 16 kHz mono WAV, then calculates a normalized RMS waveform using a sliding window. This envelope visualization reveals volume discrepancies, phase shifts, or unexpected silence that pure waveform comparison might miss.

Overlaying Transcript Words and Silence Shading

The final layer uses words_in_range (lines 118–132) to filter JSON transcript entries falling within your specified time window. The render_timeline function (lines 84–122) draws these words onto the image at their precise temporal positions. Meanwhile, find_silences (lines 135–148) identifies gaps longer than 0.4 seconds and applies shading to highlight unexpected pauses or inserted audio.

Running the Video-Use Verification Workflow

To verify that your rendered output matches the source, execute timeline_view.py twice with identical parameters, then compare the resulting PNGs.

Generate the source verification image:

python helpers/timeline_view.py \
    path/to/source.mp4 0.0 10.0 \
    --n-frames 12 \
    --transcript path/to/source_transcript.json \
    -o verify/source_0-10.png

Generate the rendered verification image using the same time window and frame count:

python helpers/timeline_view.py \
    path/to/rendered.mp4 0.0 10.0 \
    --n-frames 12 \
    --transcript path/to/rendered_transcript.json \
    -o verify/rendered_0-10.png

For automated quantitative comparison, pipe the outputs into ImageMagick:

compare -metric PSNR verify/source_0-10.png verify/rendered_0-10.png diff.png

Interpreting the Comparison Results

Opening both PNGs reveals specific failure modes that indicate rendering errors or processing drift.

Detecting Frame-Level Drift

Misaligned thumbnails between the source and rendered film-strips indicate frame drops, duplication, or timestamp drift. If the rendered clip contains extra frames, the thumbnails shift position relative to the audio envelope and transcript markers below them.

Identifying Audio Mismatches

The RMS waveform layer exposes loudness normalization errors, codec artifacts, or phase cancellation. A higher or lower envelope curve in the rendered PNG signals that the audio stream was altered during processing, even if the video frames appear identical.

Spotting Transcript Alignment Issues

Word labels that appear horizontally offset between the two images indicate A/V sync drift or speech timing changes. Unexpected silence shading (gaps over 0.4 s) in one PNG but not the other highlights where the rendered clip contains edited pauses or missing dialogue.

Summary

• helpers/timeline_view.py generates a three-layer verification PNG combining frame thumbnails, RMS audio envelopes, and transcript overlays.
• Run the script twice with identical --n-frames, timestamps, and transcript inputs to create comparable source and rendered visualizations.
• Frame drift appears as thumbnail misalignment, while audio issues manifest as differing waveform amplitudes.
• Silence gaps exceeding 0.4 seconds are shaded automatically to surface unexpected pauses.
• Optional ImageMagick integration provides pixel-level quantitative diffing via the compare command.

Frequently Asked Questions

What is the timeline_view script in video-use?

The timeline_view script is a diagnostic tool located at helpers/timeline_view.py in the browser-use/video-use repository. It generates a static PNG visualization that combines a film-strip of video frames, an RMS audio waveform, and word-level transcript overlays to create a visual fingerprint of a video clip.

How do I ensure accurate comparison between source and rendered videos?

You must invoke timeline_view.py with identical parameters for both clips: the same start/end timestamps, the same --n-frames value, and compatible transcript JSON files. Any deviation in these inputs will produce PNGs with different layouts, making visual comparison impossible.

What does the silence shading indicate in the verification PNG?

Silence shading highlights audio gaps longer than 0.4 seconds detected by the find_silences function (lines 135–148). These shaded regions help you identify unexpected pauses in the rendered output or sections where dialogue was accidentally removed during processing.

Can I automate the comparison between two timeline_view outputs?

Yes. Because the output is a standard PNG, you can use image processing tools like ImageMagick’s compare command to calculate metrics such as PSNR (Peak Signal-to-Noise Ratio) or AE (Absolute Error). This allows CI/CD pipelines to fail builds when the rendered output deviates quantifiably from the source.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}