# How the video-use Pipeline Processes Video from Transcription to Final Rendering

> Discover the browser-use video pipeline. Learn how it automates video production from transcription to final HDR rendering, including audio extraction, markdown packing, and compositing.

- Repository: [Browser Use/video-use](https://github.com/browser-use/video-use)
- Tags: internals
- Published: 2026-07-08

---

**The video-use pipeline automates video production through three sequential stages: FFmpeg-based audio extraction and ElevenLabs Scribe transcription, phrase-level markdown packing for editorial review, and FFmpeg-powered rendering with HDR tone-mapping, loudness normalization, and final compositing.**

The browser-use/video-use repository provides a fully automated video processing pipeline that transforms raw footage into publication-ready content. This open-source workflow handles everything from speech-to-text transcription to final MP4 rendering with professional audio leveling and color grading. The system operates through three distinct phases managed by helper scripts in the `helpers/` directory.

## Stage 1: Transcription with [`helpers/transcribe.py`](https://github.com/browser-use/video-use/blob/main/helpers/transcribe.py)

The pipeline begins by extracting audio and generating detailed transcripts using ElevenLabs Scribe. This stage produces a JSON file containing word-level timestamps, speaker diarization, and audio event detection.

### Audio Extraction with FFmpeg

First, the pipeline converts the input video to a mono 16 kHz WAV file optimized for speech recognition. In [`helpers/transcribe.py`](https://github.com/browser-use/video-use/blob/main/helpers/transcribe.py), the `extract_audio` function executes FFmpeg with precise parameters:

```python
subprocess.run(["ffmpeg", "-y", "-i", str(video_path),
                "-vn", "-ac", "1", "-ar", "16000",
                "-c:a", "pcm_s16le", str(dest)], check=True)

```

This command strips video streams (`-vn`), forces mono audio (`-ac 1`), sets the sample rate to 16000 Hz (`-ar 16000`), and outputs 16-bit PCM (`pcm_s16le`).

### ElevenLabs Scribe Integration

The extracted WAV file uploads to ElevenLabs Scribe via a POST request to `https://api.elevenlabs.io/v1/speech-to-text`. The code reads the API key from environment variables or `.env` files:

```python
resp = requests.post(SCRIBE_URL,
                     headers={"xi-api-key": api_key},
                     files={"file": (audio_path.name, f, "audio/wav")},
                     data=data, timeout=1800)

```

The response contains granular transcription data including individual words, speaker IDs, and precise timestamps.

### Caching Strategy

The transcription stage implements intelligent caching to avoid redundant API calls. Before uploading, the script checks if `<edit_dir>/transcripts/<video>.json` already exists using `if out_path.exists()`. If found, the stage skips processing and uses the cached JSON.

## Stage 2: Packing and Editing with [`helpers/pack_transcripts.py`](https://github.com/browser-use/video-use/blob/main/helpers/pack_transcripts.py)

Raw Scribe JSON is verbose and unsuitable for human editors. The packing stage collapses this data into a compact, phrase-level markdown file that serves as the primary artifact for editorial decision-making.

### Grouping Words into Phrases

The `group_into_phrases` function processes the JSON to create logical speaking segments. It breaks phrases on speaker changes or silences exceeding 0.5 seconds:

```python
if t == "spacing" and (gap := end - start) >= silence_threshold:
    flush()

```

This threshold-based grouping creates natural editorial boundaries while preserving temporal accuracy.

### Markdown Output Format

Each phrase receives a timecode tag and optional speaker identifier. The script formats entries as:

```python
lines.append(f"  [{format_time(p['start'])}-{format_time(p['end'])}]{spk_tag} {p['text']}")

```

The resulting [`takes_packed.md`](https://github.com/browser-use/video-use/blob/main/takes_packed.md) file contains header metadata and entries like `[00:12.340-00:15.200]S0 Hello world`, providing editors with a token-efficient timeline for creating Edit Decision Lists (EDLs).

## Stage 3: Rendering with [`helpers/render.py`](https://github.com/browser-use/video-use/blob/main/helpers/render.py)

The final stage consumes an EDL JSON produced by the editor sub-agent and executes the heavy lifting of video production. This includes segment extraction, color grading, concatenation, subtitle generation, and audio normalization.

### Per-Segment Extraction

The `extract_segment` function handles individual clip processing using FFmpeg's seek and cut capabilities. For each segment in the EDL, it applies:

- **HDR to SDR tone-mapping**: If `is_hdr_source()` detects HLG or PQ color spaces, the `TONEMAP_CHAIN` filter prepends the processing pipeline
- **Resolution scaling**: Portrait videos scale by height, landscape by width
- **Audio fading**: 30 millisecond fade-in/out to eliminate pops

```python

# Audio fade parameters (lines 87-90)

fade_in = "afade=t=in:ss=0:d=0.03"
fade_out = "afade=t=out:st={end}:d=0.03"

```

### HDR Handling and Color Grading

The rendering pipeline automatically detects high dynamic range content. When present, it applies tone-mapping before color grading. The `resolve_grade_filter` function supports preset names, raw filter strings, or `"auto"` mode which triggers `auto_grade_for_clip` for segment-specific adjustments.

### Concatenation and Subtitles

After extracting all segments, the `concat_segments` function generates a temporary [`_concat.txt`](https://github.com/browser-use/video-use/blob/main/_concat.txt) file listing all segments and executes `ffmpeg -f concat -c copy` for lossless assembly.

For subtitle generation, `build_master_srt` merges per-source transcripts with EDL offsets, producing uppercase two-word cues synchronized to the final timeline.

### Loudness Normalization

The pipeline implements two-pass loudness normalization targeting social media standards of **-14 LUFS** integrated loudness, **-1 dBTP** true peak, and **LRA 11** loudness range:

- **First pass**: `measure_loudness` analyzes the audio stream
- **Second pass**: `apply_loudnorm_two_pass` applies the measured correction parameters

### Final Compositing

The `build_final_composite` function overlays graphics and animations with PTS shifting to align the first frame with each segment's `start_in_output`. The subtitle filter applies last to ensure text renders above all video layers. Final encoding uses `libx264` with CRF 18 and the fast preset for optimal quality-to-speed ratio.

## Complete Workflow Example

Execute the full pipeline from command line:

```bash

# Stage 1: Transcribe

python helpers/transcribe.py ./raw/example.mp4 --edit-dir ./edit

# Stage 2: Pack for editing

python helpers/pack_transcripts.py --edit-dir ./edit

# Stage 3: Render (requires edl.json from editor)

python helpers/render.py edl.json -o final.mp4 \
    --build-subtitles \
    --preview  # For 1080p preview, or use --draft for 720p

```

Stage 1 outputs [`./edit/transcripts/example.json`](https://github.com/browser-use/video-use/blob/main/./edit/transcripts/example.json). Stage 2 generates [`./edit/takes_packed.md`](https://github.com/browser-use/video-use/blob/main/./edit/takes_packed.md) for editorial review. Stage 3 produces the final MP4 with optional subtitles and normalized audio.

## Summary

- **Transcription**: [`helpers/transcribe.py`](https://github.com/browser-use/video-use/blob/main/helpers/transcribe.py) extracts 16 kHz mono audio, uploads to ElevenLabs Scribe, and caches detailed JSON transcripts with word-level timestamps
- **Packing**: [`helpers/pack_transcripts.py`](https://github.com/browser-use/video-use/blob/main/helpers/pack_transcripts.py) collapses raw JSON into phrase-level markdown using 0.5-second silence thresholds, creating human-readable timelines for EDL creation
- **Rendering**: [`helpers/render.py`](https://github.com/browser-use/video-use/blob/main/helpers/render.py) executes segment extraction with HDR tone-mapping, lossless concatenation, subtitle generation, two-pass loudness normalization (-14 LUFS), and final compositing with `libx264` encoding
- **Integration**: The pipeline accepts raw video and produces publication-ready MP4 through automated FFmpeg operations and intelligent caching

## Frequently Asked Questions

### What is the video-use pipeline and what problem does it solve?

The video-use pipeline is an open-source automation framework maintained by browser-use that transforms raw video footage into edited, publication-ready content. It solves the problem of manual video editing by automating transcription, rough-cut generation, color grading, and audio normalization through a three-stage Python and FFmpeg-based workflow.

### How does the transcription caching mechanism work in video-use?

The transcription stage checks for existing files at `<edit_dir>/transcripts/<video>.json` before calling the ElevenLabs API. If the JSON file exists, the script skips the upload and uses the cached transcript, preventing redundant API calls and reducing processing time for previously analyzed footage.

### What is the purpose of the Edit Decision List (EDL) in the rendering stage?

The EDL is a JSON file created by the editor sub-agent that describes which time ranges to keep, which color grades to apply, and where to place overlays. The [`helpers/render.py`](https://github.com/browser-use/video-use/blob/main/helpers/render.py) script consumes this EDL to execute `extract_segment` operations, concatenation, and final compositing, serving as the bridge between editorial decisions and automated video production.

### How does the video-use pipeline handle HDR video sources?

The pipeline detects HDR content (HLG or PQ color spaces) using `is_hdr_source()` and automatically prepends the `TONEMAP_CHAIN` filter during segment extraction. This converts high dynamic range footage to standard dynamic range before applying color grades, ensuring consistent output across different source formats.