Best Practices for Using video-use in Production: A Complete Guide

To run video-use in production, enforce the hard rules from SKILL.md (subtitles last, 30ms fades, word-boundary cuts), cache transcripts with transcribe_batch.py, and validate outputs with timeline_view.py before delivery.

The browser-use/video-use repository provides a production-grade video editing pipeline that enables language models to edit raw footage by reasoning over word-level transcripts and visual snapshots. Following the architectural hard rules and helper script conventions ensures broadcast-ready output without silent failures like missing subtitles or audio pops. This guide covers the complete workflow from transcription to final render.

Architectural Overview

The video-use pipeline consists of three distinct layers that separate concerns between transcription, visual decision-making, and final composition.

Layer 1: Audio Transcript Processing

A single ElevenLabs Scribe call per source creates a word-level JSON stored in transcripts/*.json. The helpers/transcribe_batch.py script processes these files in parallel with caching, while helpers/pack_transcripts.py compresses them into a lightweight takes_packed.md file that the LLM consumes without token explosion.

Layer 2: Visual Composite Generation

Instead of processing every frame, helpers/timeline_view.py generates on-demand film-strip and waveform PNGs only at decision points. This selective visualization reduces token usage while providing the LLM with enough context to identify cut boundaries and visual jumps.

Layer 3: Render Pipeline

The helpers/render.py script orchestrates per-segment extraction, color grading, audio fades, overlays, and subtitle burn-in. It enforces filter graph ordering and handles HDR auto-detection with tone-mapping chains to prevent oversaturation on SDR platforms.

Production Workflow

Follow this eight-step sequence to maintain reproducibility and avoid the silent failures that occur when hard rules are violated.

1. One-time setup – Clone the repository, run uv sync (or pip install -e .), install ffmpeg and yt-dlp, and configure your ElevenLabs API key in .env per install.md.

2. Verify environment – Confirm that ffprobe, ffmpeg, and node (for HyperFrames/Remotion) are available on your system PATH before processing any footage.

3. Transcribe once – Execute python helpers/transcribe_batch.py /path/to/videos --workers 8 to generate cached JSON transcripts. Re-transcription only triggers when source files change, preserving API quota.

4. Pack transcripts – Run python helpers/pack_transcripts.py --edit-dir /path/to/videos/edit to create the compact takes_packed.md consumed by the LLM.

5. Strategy planning – The LLM reads the packed transcript, asks clarifying questions, and proposes a cut/animation/grade plan. You must explicitly confirm this plan before editing begins, as required by Hard Rule 11 in SKILL.md.

6. Execute the plan – Generate an edl.json via the editor sub-agent, then invoke python helpers/render.py /path/to/edl.json -o final.mp4. The renderer extracts each segment, applies grades from helpers/grade.py, inserts 30ms audio fades, and constructs the final filter graph.

7. Self-evaluation – Run python helpers/timeline_view.py on the rendered output at every cut boundary to detect visual jumps or subtitle-overlay conflicts. The system allows up to three auto-fix passes before escalating persistent issues.

8. Persist – The session records to project.md while the final video (final.mp4) must reside under <videos_dir>/edit/ per Hard Rule 12, keeping all outputs separate from the repository source code.

Hard Rules for Rendering

The SKILL.md file enumerates non-negotiable constraints that helpers/render.py enforces automatically. Breaking any rule causes silent failures such as double-encoding or missing captions.

• Rule 1: Apply subtitles last in the filter graph to prevent overlays from hiding text.
• Rule 2: Use per-segment extract followed by lossless -c copy concatenation to avoid generational loss.
• Rule 3: Insert 30ms audio fades at every cut edge to eliminate pops.
• Rule 4: Shift overlays with setpts=PTS-STARTPTS+T/TB to maintain timeline synchronization.
• Rule 5: Compute Master SRT timestamps from output-timeline offsets, not source timings.
• Rule 6: Snap all cuts to word boundaries to preserve linguistic coherence.
• Rule 7: Pad cut edges by 30-200ms to absorb Scribe timestamp drift and prevent choppy audio.
• Rules 8-12: Cache transcripts, run parallel animation agents, and keep all session outputs under <videos_dir>/edit/.

Performance Optimization

Optimize your production workflow with these targeted strategies validated against the source code.

Cache transcripts aggressively. The transcribe_batch.py helper checks file modification times to skip unchanged sources, reducing API costs and iteration time.

Spawn animation sub-agents in parallel. Use the Agent tool for HyperFrames, Remotion, or Manim simultaneously. Wall-time equals the slowest animation rather than the sum of all durations (Hard Rule 10).

Run visual inspection selectively. Execute timeline_view.py only at decision points rather than continuously to minimize token usage while preserving quality control.

Validate HDR sources early. The render.py auto-detection logic inserts tone-mapping chains automatically, but verifying HDR metadata upfront prevents pipeline restarts.

Use preset grades. The helpers/grade.py file provides warm_cinematic and neutral_punch presets via ffmpeg filter strings, offering a solid baseline before custom grading.

Production Setup Guide

Execute these commands to initialize a production environment and process your first batch.

# Install once (run only on a fresh machine)

git clone https://github.com/browser-use/video-use ~/Developer/video-use
cd ~/Developer/video-use
uv sync                     # or: pip install -e .

brew install ffmpeg yt-dlp # macOS; use apt/yum on Linux

# Set up ElevenLabs API key

cp .env.example .env
$EDITOR .env               # add ELEVENLABS_API_KEY=your_key_here

# Transcribe a folder of raw takes (parallel workers)

python helpers/transcribe_batch.py /path/to/videos --workers 8

# Pack transcripts into a single markdown file

python helpers/pack_transcripts.py --edit-dir /path/to/videos/edit

# Inspect a slice (optional) – visual aid for the LLM

python helpers/timeline_view.py /path/to/videos/video.mp4 12.0 16.0 --n-frames 5 -o slice.png

# Let the LLM propose a plan, confirm it, then generate an EDL

# (The LLM runs the editor sub‑agent; here we just invoke the helper)

python helpers/render.py /path/to/edl.json -o final.mp4

# Quick preview (lower quality, fast)

python helpers/render.py /path/to/edl.json -o preview.mp4 --preview

# Build subtitles (optional)

python helpers/render.py /path/to/edl.json -o final.mp4 --build-subtitles

For parallel animation generation, spawn separate processes and wait for completion:

# Inside <videos_dir>/edit/animations/slot_01/

npx --yes hyperframes render --input scene.py --output anim.mp4 &
npx create-video@latest render --input animation.ts --output anim2.mp4 &
wait   # both agents run in parallel

Summary

• video-use separates concerns into three layers: audio transcription (transcribe_batch.py), visual snapshots (timeline_view.py), and rendering (render.py).
• Hard rules in SKILL.md enforce subtitles-last ordering, 30ms fades, word-boundary cuts, and 30-200ms padding to prevent broadcast failures.
• Always cache transcripts, confirm editing plans before execution, and validate rendered output with timeline_view.py before delivery.
• Keep all outputs under <videos_dir>/edit/ to avoid repository contamination.
• Run animation agents in parallel and use preset grades from helpers/grade.py to optimize wall-time and visual consistency.

Frequently Asked Questions

What dependencies are required to run video-use in production?

You need ffmpeg and ffprobe for video processing, yt-dlp for downloads, and node for HyperFrames or Remotion animations. Python dependencies install via uv sync or pip install -e .. You must also configure an ElevenLabs API key in .env for the Scribe transcription service used by helpers/transcribe_batch.py.

Why must subtitles be applied last in the filter graph?

Rule 1 in SKILL.md mandates subtitles as the final filter step because ffmpeg processes filters sequentially. If overlays or grades come after subtitles, they obscure the text. The helpers/render.py script enforces this ordering automatically to guarantee caption visibility.

How does video-use handle HDR video sources?

The helpers/render.py pipeline auto-detects HDR metadata and inserts a tone-mapping chain before color grading. This prevents oversaturated output on SDR platforms without manual intervention. For optimal results, validate HDR flags with ffprobe before transcoding to ensure the auto-detection triggers correctly.

What is the purpose of the 30-200ms padding rule for cuts?

Hard Rule 7 requires padding cut edges by 30-200ms to absorb timestamp drift from ElevenLabs Scribe and avoid choppy audio at word boundaries. This working window ensures smooth transitions even when the transcript timestamps vary slightly from the actual audio waveform, which is critical for professional broadcast quality.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}