How the video-use Pipeline Processes Video from Transcription to Final Rendering

The video-use pipeline automates video production through three sequential stages: FFmpeg-based audio extraction and ElevenLabs Scribe transcription, phrase-level markdown packing for editorial review, and FFmpeg-powered rendering with HDR tone-mapping, loudness normalization, and final compositing.

The browser-use/video-use repository provides a fully automated video processing pipeline that transforms raw footage into publication-ready content. This open-source workflow handles everything from speech-to-text transcription to final MP4 rendering with professional audio leveling and color grading. The system operates through three distinct phases managed by helper scripts in the helpers/ directory.

Stage 1: Transcription with helpers/transcribe.py

The pipeline begins by extracting audio and generating detailed transcripts using ElevenLabs Scribe. This stage produces a JSON file containing word-level timestamps, speaker diarization, and audio event detection.

Audio Extraction with FFmpeg

First, the pipeline converts the input video to a mono 16 kHz WAV file optimized for speech recognition. In helpers/transcribe.py, the extract_audio function executes FFmpeg with precise parameters:

subprocess.run(["ffmpeg", "-y", "-i", str(video_path),
                "-vn", "-ac", "1", "-ar", "16000",
                "-c:a", "pcm_s16le", str(dest)], check=True)

This command strips video streams (-vn), forces mono audio (-ac 1), sets the sample rate to 16000 Hz (-ar 16000), and outputs 16-bit PCM (pcm_s16le).

ElevenLabs Scribe Integration

The extracted WAV file uploads to ElevenLabs Scribe via a POST request to https://api.elevenlabs.io/v1/speech-to-text. The code reads the API key from environment variables or .env files:

resp = requests.post(SCRIBE_URL,
                     headers={"xi-api-key": api_key},
                     files={"file": (audio_path.name, f, "audio/wav")},
                     data=data, timeout=1800)

The response contains granular transcription data including individual words, speaker IDs, and precise timestamps.

Caching Strategy

The transcription stage implements intelligent caching to avoid redundant API calls. Before uploading, the script checks if <edit_dir>/transcripts/<video>.json already exists using if out_path.exists(). If found, the stage skips processing and uses the cached JSON.

Stage 2: Packing and Editing with helpers/pack_transcripts.py

Raw Scribe JSON is verbose and unsuitable for human editors. The packing stage collapses this data into a compact, phrase-level markdown file that serves as the primary artifact for editorial decision-making.

Grouping Words into Phrases

The group_into_phrases function processes the JSON to create logical speaking segments. It breaks phrases on speaker changes or silences exceeding 0.5 seconds:

if t == "spacing" and (gap := end - start) >= silence_threshold:
    flush()

This threshold-based grouping creates natural editorial boundaries while preserving temporal accuracy.

Markdown Output Format

Each phrase receives a timecode tag and optional speaker identifier. The script formats entries as:

lines.append(f"  [{format_time(p['start'])}-{format_time(p['end'])}]{spk_tag} {p['text']}")

The resulting takes_packed.md file contains header metadata and entries like [00:12.340-00:15.200]S0 Hello world, providing editors with a token-efficient timeline for creating Edit Decision Lists (EDLs).

Stage 3: Rendering with helpers/render.py

The final stage consumes an EDL JSON produced by the editor sub-agent and executes the heavy lifting of video production. This includes segment extraction, color grading, concatenation, subtitle generation, and audio normalization.

Per-Segment Extraction

The extract_segment function handles individual clip processing using FFmpeg's seek and cut capabilities. For each segment in the EDL, it applies:

• HDR to SDR tone-mapping: If is_hdr_source() detects HLG or PQ color spaces, the TONEMAP_CHAIN filter prepends the processing pipeline
• Resolution scaling: Portrait videos scale by height, landscape by width
• Audio fading: 30 millisecond fade-in/out to eliminate pops

# Audio fade parameters (lines 87-90)

fade_in = "afade=t=in:ss=0:d=0.03"
fade_out = "afade=t=out:st={end}:d=0.03"

HDR Handling and Color Grading

The rendering pipeline automatically detects high dynamic range content. When present, it applies tone-mapping before color grading. The resolve_grade_filter function supports preset names, raw filter strings, or "auto" mode which triggers auto_grade_for_clip for segment-specific adjustments.

Concatenation and Subtitles

After extracting all segments, the concat_segments function generates a temporary _concat.txt file listing all segments and executes ffmpeg -f concat -c copy for lossless assembly.

For subtitle generation, build_master_srt merges per-source transcripts with EDL offsets, producing uppercase two-word cues synchronized to the final timeline.

Loudness Normalization

The pipeline implements two-pass loudness normalization targeting social media standards of -14 LUFS integrated loudness, -1 dBTP true peak, and LRA 11 loudness range:

• First pass: measure_loudness analyzes the audio stream
• Second pass: apply_loudnorm_two_pass applies the measured correction parameters

Final Compositing

The build_final_composite function overlays graphics and animations with PTS shifting to align the first frame with each segment's start_in_output. The subtitle filter applies last to ensure text renders above all video layers. Final encoding uses libx264 with CRF 18 and the fast preset for optimal quality-to-speed ratio.

Complete Workflow Example

Execute the full pipeline from command line:

# Stage 1: Transcribe

python helpers/transcribe.py ./raw/example.mp4 --edit-dir ./edit

# Stage 2: Pack for editing

python helpers/pack_transcripts.py --edit-dir ./edit

# Stage 3: Render (requires edl.json from editor)

python helpers/render.py edl.json -o final.mp4 \
    --build-subtitles \
    --preview  # For 1080p preview, or use --draft for 720p

Stage 1 outputs ./edit/transcripts/example.json. Stage 2 generates ./edit/takes_packed.md for editorial review. Stage 3 produces the final MP4 with optional subtitles and normalized audio.

Summary

• Transcription: helpers/transcribe.py extracts 16 kHz mono audio, uploads to ElevenLabs Scribe, and caches detailed JSON transcripts with word-level timestamps
• Packing: helpers/pack_transcripts.py collapses raw JSON into phrase-level markdown using 0.5-second silence thresholds, creating human-readable timelines for EDL creation
• Rendering: helpers/render.py executes segment extraction with HDR tone-mapping, lossless concatenation, subtitle generation, two-pass loudness normalization (-14 LUFS), and final compositing with libx264 encoding
• Integration: The pipeline accepts raw video and produces publication-ready MP4 through automated FFmpeg operations and intelligent caching

Frequently Asked Questions

What is the video-use pipeline and what problem does it solve?

The video-use pipeline is an open-source automation framework maintained by browser-use that transforms raw video footage into edited, publication-ready content. It solves the problem of manual video editing by automating transcription, rough-cut generation, color grading, and audio normalization through a three-stage Python and FFmpeg-based workflow.

How does the transcription caching mechanism work in video-use?

The transcription stage checks for existing files at <edit_dir>/transcripts/<video>.json before calling the ElevenLabs API. If the JSON file exists, the script skips the upload and uses the cached transcript, preventing redundant API calls and reducing processing time for previously analyzed footage.

What is the purpose of the Edit Decision List (EDL) in the rendering stage?

The EDL is a JSON file created by the editor sub-agent that describes which time ranges to keep, which color grades to apply, and where to place overlays. The helpers/render.py script consumes this EDL to execute extract_segment operations, concatenation, and final compositing, serving as the bridge between editorial decisions and automated video production.

How does the video-use pipeline handle HDR video sources?

The pipeline detects HDR content (HLG or PQ color spaces) using is_hdr_source() and automatically prepends the TONEMAP_CHAIN filter during segment extraction. This converts high dynamic range footage to standard dynamic range before applying color grades, ensuring consistent output across different source formats.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}