How to Manage Video Player State with video-use: A Transcript-Centric Approach

video-use treats the transcript as the single source of truth for video state, generating on-demand visual composites via timeline_view helpers instead of maintaining a traditional embedded video player.

Managing video player state with video-use requires abandoning conventional frame-based players in favor of a transcript-centric architecture. In the browser-use/video-use repository, all editing decisions—cuts, fades, subtitles, and color grades—are derived from word-level timestamps supplied by the ElevenLabs Scribe service. This design keeps LLM token budgets minimal (approximately 12 KB of text plus a few PNGs) while providing precise, frame-accurate visual feedback.

Understanding the Transcript-Centric State Model

Traditional video players maintain state through embedded UI components and frame buffers. In contrast, video-use treats the transcript as the authoritative state representation. When the LLM needs to reason about a specific moment, the system builds a visual composite on-demand using the timeline_view pipeline. This composite contains a film-strip of selected frames, the audio waveform, and word-label overlays, generated only for the requested time range.

Because state is transcript-driven, any player-style operation—seek, pause, or rewind—is expressed as a time range passed to the timeline_view helpers. These helpers compute the necessary data (frames, envelope, word list) and return a PNG, allowing the LLM or a downstream UI to display the video's appearance at that exact moment.

The Six-Step State Management Pipeline

The state management flow follows a strict pipeline defined in the source code:

1. Transcribe – One ElevenLabs Scribe call per source produces word-level timestamps, speaker diarization, and audio events. Core module: helpers/transcribe.py.

2. Pack – Collates all takes into a compact markdown file (takes_packed.md) that the LLM reads. Core module: helpers/pack_transcripts.py.

3. LLM Reasoning – The LLM proposes a cut-list (EDL) based on the transcript data. Core module: SKILL.md (production rules).

4. Render – Applies cuts, color grades, fades, subtitles, and other effects. Core module: helpers/render.py.

5. Self-Eval – For every cut boundary, the LLM requests timeline_view to render a PNG of the output and validates it for visual pops or jumps. Core module: helpers/timeline_view.py.

6. Persist – Saves the final edit/final.mp4 and session markdown for subsequent sessions. Core module: project.md (session memory).

Core State Data Structures

While the pipeline runs, video player state resides in lightweight Python structures:

• Current time – A float value representing seconds supplied to helper functions.
• Segments – A list of (start, end) tuples derived from the EDL.
• Audio envelope – A NumPy array computed by helpers/timeline_view.compute_envelope that enables silence detection for safe cuts.
• Word-level data – Dictionaries returned by helpers/timeline_view.words_in_range containing timestamps and text.

These structures live in memory during pipeline execution and are never persisted beyond the generated PNGs and the final edited video.

Querying Player State with timeline_view

The helpers/timeline_view.py module provides the primary interface for managing player state. Use these functions to inspect specific time ranges without loading the full video into memory.

Locating Words by Time Range

To retrieve transcript entries for a specific playback segment, use words_in_range. This function treats the transcript as the state source and returns all words falling within the specified bounds.

from pathlib import Path
from helpers.timeline_view import words_in_range

transcript_path = Path("edit/takes_packed.md")
start_sec = 12.3
end_sec = 15.0

words = words_in_range(transcript_path, start_sec, end_sec)
print(words)  # [{'start': 12.30, 'end': 12.45, 'word': '...'}, ...]

Computing Audio Envelopes for Silence Detection

To detect silence gaps suitable for cutting, compute the audio envelope using compute_envelope. This returns a NumPy array of RMS amplitudes where low values indicate silence.

from helpers.timeline_view import compute_envelope
import numpy as np

envelope = compute_envelope(
    video=Path("raw/video.mp4"),
    start=start_sec,
    end=end_sec,
    samples=2000,
)

# envelope is a np.ndarray of RMS amplitudes

Rendering Visual Composites

To generate the visual feedback equivalent to a paused video frame, use render_timeline. This creates a PNG containing the film-strip, waveform, and word labels.

from helpers.timeline_view import render_timeline

render_timeline(
    video=Path("raw/video.mp4"),
    transcript=transcript_path,
    start=start_sec,
    end=end_sec,
    output_path=Path("debug/state_12.3-15.0.png"),
    n_frames=8,  # number of frames in the filmstrip

)

Implementing Seek and Playback Operations

Because video-use lacks a traditional player component, you implement seek operations by iterating time ranges and generating timeline views. This approach simulates playback while maintaining minimal memory footprint.

import time
from pathlib import Path
from helpers.timeline_view import render_timeline

video_path = Path("raw/video.mp4")
transcript_path = Path("edit/takes_packed.md")
duration = 60.0  # one-minute clip

cursor = 0.0

while cursor < duration:
    render_timeline(
        video=video_path,
        transcript=transcript_path,
        start=cursor,
        end=cursor + 5.0,
        output_path=Path(f"debug/frame_{cursor:.0f}.png"),
        n_frames=5,
    )
    cursor += 5.0
    time.sleep(0.2)  # simulate playback tick

These lightweight Python calls allow you to seek, inspect silence gaps, and visualize cuts without ever instantiating a heavy video player object.

Summary

• video-use manages video player state through the transcript (takes_packed.md) rather than embedded UI components.
• The helpers/timeline_view.py module generates on-demand visual composites (film-strips, waveforms, word labels) as PNGs for specific time ranges.
• State queries use words_in_range for text data, compute_envelope for audio analysis, and render_timeline for visual feedback.
• The six-step pipeline (Transcribe → Pack → LLM Reasoning → Render → Self-Eval → Persist) operates entirely on word-level timestamps from ElevenLabs Scribe.
• This architecture keeps LLM token usage at approximately 12 KB while providing frame-accurate editing feedback.

Frequently Asked Questions

Why doesn't video-use use a traditional video player component?

video-use is designed for LLM-driven editing workflows where loading full video frames into context would exceed token limits. By treating the transcript as the single source of truth and generating lightweight PNG composites via timeline_view, the system maintains a minimal memory footprint while still providing precise visual validation for every editing decision.

How does video-use keep token usage low when processing video?

Instead of embedding video bytes or heavy player state, video-use passes only the packed transcript (takes_packed.md) and small PNG snapshots to the LLM. A typical interaction contains approximately 12 KB of text plus a few generated images, allowing the LLM to reason about hour-long videos without hitting context window limits.

What file defines the production rules for LLM reasoning?

The SKILL.md file in the repository root contains the 12 hard production rules that govern how the LLM interprets transcript data and proposes edit decisions (EDL). It defines the interface between the LLM and the video state representation.

How can I programmatically seek to a specific timestamp?

Seeking is implemented by calling render_timeline from helpers/timeline_view.py with your target start and end times as float values. The function returns a PNG of that specific range, effectively serving as a "pause" frame without requiring a persistent player instance.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}