How `timeline_view.py` Composites the Filmstrip, Waveform, and Word Labels

The timeline_view.py script composites a multi-layered visualization by vertically stacking an FFmpeg-extracted filmstrip, a filled RMS waveform, and transcript-based word labels and silence shading onto a single PIL canvas.

The browser-use/video-use repository provides a sophisticated visualization utility that transforms video segments into annotated timeline images. In helpers/timeline_view.py, the render_timeline function orchestrates a six-stage pipeline that combines visual frames, audio envelopes, and transcript data into a unified PNG composite. This compositing process generates clear, information-dense summaries for video analysis by precisely layering distinct visual elements.

The Visual Layer Architecture

The compositing engine constructs the final image by stacking five distinct visual elements in a specific vertical arrangement, starting with a horizontal offset of 50 pixels and maintaining strategic 20-pixel buffers between major components.

Filmstrip Frame Extraction

The top visual layer consists of evenly spaced frames extracted via FFmpeg using the extract_frames function. Each frame is resized to a common height while preserving aspect ratio, then pasted horizontally onto the canvas starting at coordinates (50, 50). The code calculates whether the accumulated width exceeds the canvas boundary, applying uniform scaling when necessary to fit the available space. The resulting strip dimensions are stored in filmstrip_h with the vertical position fixed at filmstrip_y = 50 [source-lines 37-63].

Waveform Visualization

Directly beneath the filmstrip, the audio layer renders as a filled "mountain" shape representing the RMS envelope. The vertical position is calculated as wave_y = filmstrip_y + filmstrip_h + 20, creating a 20-pixel gap between visual elements. The compute_envelope function generates a mono 16 kHz PCM segment via FFmpeg, normalizes the amplitude values to the range [0, 1], and maps these to canvas coordinates. The rendering uses draw.line and draw.polygon with the WAVE color constant to create the filled waveform shape [source-lines 68-110].

Silence Shading and Word Labels

When a transcript is provided, the system enhances the waveform layer with two additional elements. First, words_in_range identifies transcript words overlapping the time range, while find_silences detects audio gaps exceeding 0.4 seconds. Semi-transparent SILENCE rectangles are drawn over these gaps atop the waveform background [source-lines 135-148].

For word labels, each word longer than 50 milliseconds is processed through time_to_x to convert temporal data to horizontal pixel coordinates. The code draws a tick mark on the waveform and renders the word text using small_font positioned just above the waveform. A collision detection system ensures labels maintain at least 28 pixels of horizontal separation to prevent overlap [source-lines 92-111].

Time Ruler

Beneath the waveform, a simple time ruler provides temporal context. The code draws six major ticks with absolute timestamp labels using label_font and draw.text, aligned to the horizontal scale established by the filmstrip and waveform layers [source-lines 114-122].

The Rendering Pipeline

The render_timeline function executes a sequential orchestration to build the final composite:

1. extract_frames – Pulls N evenly spaced JPGs from the video segment using FFmpeg
2. Frame Layout – Resizes frames, computes total width, and determines whether to apply uniform scaling
3. compute_envelope – Generates normalized RMS envelope data from 16 kHz PCM audio
4. Transcript Analysis – Uses words_in_range and find_silences to identify word positions and silence gaps
5. Drawing Operations – Composites filmstrip, waveform background, silence shading, envelope fill, word ticks, labels, and time ruler onto the PIL canvas
6. PNG Export – Writes the final image via canvas.save(out_path, "PNG", optimize=True) [source-lines 128-132]

Key Implementation Details

Coordinate System and Spacing

The canvas employs a fixed horizontal margin of 50 pixels for the filmstrip start. Vertical positioning follows a strict hierarchical offset: the filmstrip anchors at y = 50, the waveform sits 20 pixels below the filmstrip bottom, and the time ruler appears beneath the waveform. This consistent spacing ensures visual separation between the filmstrip, waveform, and annotation layers [source-lines 37-63][source-lines 68-110].

Audio Processing Pipeline

The waveform generation relies on FFmpeg to extract mono 16 kHz PCM audio for the specified time range. The compute_envelope function calculates RMS values, normalizes them to a 0-1 range, and maps these to the available vertical space for the polygon fill. The WAVE constant from helpers/grade.py defines the specific color used for the filled mountain shape [source-lines 74-89].

Transcript Integration

Word labels require a JSON transcript typically generated by helpers/transcribe.py. The time_to_x utility function maps temporal data to horizontal pixel positions based on the canvas width and time duration. The rendering engine filters out words shorter than 50 milliseconds and enforces minimum spacing constraints to maintain readability [source-lines 92-111].

Usage Examples

Generate timeline visualizations using the command-line interface:

# Basic timeline for seconds 12-18

python helpers/timeline_view.py demo.mp4 12 18 -o demo_12-18.png

# Increase filmstrip density to 20 frames

python helpers/timeline_view.py demo.mp4 12 18 --n-frames 20 -o demo_12-18.png

# Add transcript data for word labels and silence detection

python helpers/timeline_view.py demo.mp4 12 18 --transcript path/to/transcript.json

Summary

• timeline_view.py composites five distinct visual layers onto a single PIL Image canvas initialized with the BG color
• FFmpeg powers both frame extraction (extract_frames) and audio PCM generation (compute_envelope) for the waveform
• Vertical stacking follows strict coordinates: filmstrip at y=50, waveform 20px below, ruler at bottom
• Silence detection overlays semi-transparent SILENCE rectangles on audio gaps ≥ 0.4s when transcripts are provided
• Word labels include collision detection (28px minimum spacing) and 50ms duration filtering
• Output is optimized PNG via canvas.save() with the optimize=True parameter

Frequently Asked Questions

How does timeline_view.py handle video frames of different sizes?

The code resizes all extracted frames to a common height while preserving aspect ratios. It calculates the total width required for the filmstrip and applies uniform scaling if the width exceeds the canvas boundary, ensuring the entire strip fits within the available horizontal space [source-lines 37-63].

What audio format does the waveform visualization use?

The tool extracts mono 16 kHz PCM audio using FFmpeg, then computes the RMS envelope to create the filled mountain shape. This normalized envelope provides the y-coordinate data for the draw.polygon operations that render the waveform visualization [source-lines 74-89].

Can I customize the colors used in the timeline visualization?

Yes, the script references color constants including WAVE, SILENCE, and BG imported from helpers/grade.py. Modifying these constants changes the waveform fill color, silence overlay shading, and canvas background color respectively.

How does the script prevent word labels from overlapping?

The rendering engine enforces a minimum 28-pixel spacing between adjacent word labels during the drawing phase. Additionally, it filters out words shorter than 50 milliseconds, ensuring only significant utterances receive annotations and maintaining visual clarity [source-lines 92-111].

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}