How timeline_view.py Generates Filmstrip and Waveform Composites for Decision Making

The render_timeline function in helpers/timeline_view.py composites a single synchronized PNG from a video segment by layering a horizontal filmstrip, an RMS audio waveform ribbon, transcript word labels, and silence markers so editors can make precise visual cut decisions.

The browser-use/video-use toolkit automates video editing workflows, and its helpers/timeline_view.py module is the core compositor that translates raw media into a decision-ready visual summary. By locking every graphic element to a single time-to-pixel mapping, the script gives editors a pixel-perfect correlation between frames, audio intensity, spoken words, and pauses. Understanding how timeline_view.py generates these filmstrip and waveform composites reveals why it is effective for on-demand drill-down tasks such as verifying retake boundaries or spotting ambiguous silences.

The render_timeline Entry Point

render_timeline (lines 84‑130) orchestrates the entire pipeline. It accepts a video path, start/end timestamps, and an optional transcript JSON, then delegates to focused helpers for frame extraction, audio analysis, and text layout before assembling a 1920-pixel-wide canvas.

Frame Extraction via extract_frames

extract_frames (lines 37‑62) runs a single ffmpeg command per target frame to pull evenly spaced thumbnails between start and end. The resulting JPEGs are resized to a uniform height of frame_h = 180 pixels and stored for later horizontal assembly.

Audio Envelope Generation with compute_envelope

compute_envelope (lines 68‑111) extracts the segment to a temporary 16 kHz PCM WAV and reads the raw samples with Python’s built-in wave module. It computes a windowed RMS array of length samples (default 2000), normalizes the values to the [0, 1] range, and returns the envelope data that drives the waveform ribbon.

Transcript Parsing and Silence Detection

words_in_range (lines 118‑132) loads the JSON transcript ({ "words": [...] }) and returns only entries whose timestamps intersect the [start, end] interval. find_silences (lines 135‑148) then scans those words for gaps ≥ threshold seconds (default 0.4 s), recording the intervals that are later rendered as semi-transparent blue bands.

Canvas Layout and Coordinate Mapping

Inside render_timeline, a blank 1920-pixel canvas is created with Image.new. Layout constants define vertical positions for the filmstrip (filmstrip_y), waveform (wave_y), and label ruler (label_y). If the combined width of the thumbnails exceeds available space, the frames are scaled uniformly to fit.

The helper time_to_x (lines 159‑162) maps any timestamp in [start, end] to an absolute X-pixel coordinate using (t - start) / (end - start) * strip_span. All subsequent graphics—silence bands, waveform points, word ticks, and the time ruler—reuse this conversion to maintain perfect temporal alignment.

Drawing the Filmstrip, Waveform, and Labels

Pasting the Filmstrip

The resized frames are pasted sequentially onto the canvas beginning at x = 50. If scaling was applied, each frame is adjusted and vertically centered. The final span (strip_span) is stored for the time_to_x conversion.

Silence Shading

For each silence gap returned by find_silences, the code draws a rectangle in the pre-defined SILENCE colour—a semi-transparent blue—directly beneath the waveform ribbon.

Waveform Ribbon Rendering

The normalized envelope array is plotted as two polylines, points_top and points_bot, tracing the upper and lower halves of the wave. A translucent polygon fills the area between them, producing the familiar ribbon appearance.

Word-Label Overlays

While iterating over the selected words, the script drops tokens shorter than 50 ms and skips labels that would crowd within 28 pixels of the previous mark. For each accepted word, it draws a short tick on the waveform and renders the text just above it in a 12 pt small font loaded via load_font (lines 154‑171).

Time Ruler and Legend

A ruler with six major ticks is drawn below the waveform, each annotated with an absolute time label formatted as {t:.2f}s. If any silences were detected, a legend line is added before the final PNG is written with canvas.save.

CLI and Python Usage Examples

The most common entry point is the command-line interface in helpers/timeline_view.py.

# Show a 10-second segment (12 frames) with transcript word labels

python helpers/timeline_view.py \
    path/to/video.mp4  30.0  40.0  \
    --n-frames 12 \
    --transcript path/to/video.json \
    -o /tmp/segment.png

For programmatic use, import render_timeline and supply the required paths and timestamps.

from pathlib import Path
from helpers.timeline_view import render_timeline

video = Path("movie.mp4")
output = Path("movie_75.5-85.0.png")

render_timeline(
    video=video,
    start=75.5,
    end=85.0,
    out_path=output,
    n_frames=15,
    transcript=Path("movie.json"),
)
print(f"Composite saved to {output}")

You can also call compute_envelope directly to retrieve the normalized audio envelope as a NumPy array.

import numpy as np
from helpers.timeline_view import compute_envelope
from pathlib import Path

env = compute_envelope(Path("lecture.mp4"), 120.0, 150.0, samples=1000)
print("Peak amplitude:", env.max())

Summary

• render_timeline in helpers/timeline_view.py orchestrates a three-layer PNG composite: filmstrip, waveform, and transcript labels.
• extract_frames (lines 37‑62) pulls uniformly spaced thumbnails via ffmpeg and resizes them to 180 px height.
• compute_envelope (lines 68‑111) derives a 2000-sample RMS envelope from a temporary 16 kHz WAV for ribbon plotting.
• words_in_range (lines 118‑132) and find_silences (lines 135‑148) supply the transcript words and silence bands that overlay the waveform.
• A shared time_to_x mapping guarantees that frames, audio, words, and pauses align to the same pixel coordinates.

Frequently Asked Questions

What dependencies does timeline_view.py require?

The script relies on ffmpeg for frame extraction, Pillow for image compositing, NumPy for envelope math, and Python’s built-in wave module for PCM audio reading. A transcript JSON is optional but must follow the { "words": [...] } schema when provided.

Can I change the number of thumbnails or waveform resolution?

Yes. The n_frames parameter controls how many thumbnails extract_frames requests from ffmpeg, and the samples parameter passed to compute_envelope sets the length of the RMS array that shapes the waveform ribbon.

Why are some transcript words missing from the composite?

Words shorter than 50 ms are filtered out, and labels are suppressed if they would appear within 28 pixels of the previous label to prevent crowding. This keeps the overlay readable in dense dialogue segments.

How does the silence detection threshold work?

find_silences uses a default gap threshold of 0.4 seconds. Any pause between consecutive transcript words that meets or exceeds this value triggers a semi-transparent blue rectangle under the waveform, making long pauses visually obvious.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}