# How the Video-Use Pipeline Manages Multiple Takes and Selects Beats Across Clips

> Discover how the video-use pipeline synchronizes multiple video takes by transcribing, analyzing beats with Librosa, and merging high-scoring beats for a cohesive montage.

- Repository: [Browser Use/video-use](https://github.com/browser-use/video-use)
- Tags: internals
- Published: 2026-07-09

---

**The video-use pipeline ingests independent video takes, transcribes them via OpenAI Whisper, aggregates the results into a unified metadata file, detects rhythmic beats using Librosa, and merges high-scoring beats across all clips while enforcing a configurable minimum interval to generate a synchronized montage.**

The `browser-use/video-use` repository automates the editing workflow for multi-take video projects. By combining speech-to-text metadata with audio onset detection, the pipeline can manage an arbitrary number of source clips and intelligently select the best beats across them to create a cohesive final video.

## The Three-Stage Pipeline Architecture

### Stage 1: Per-Take Transcription with Scribe JSON

Each raw video file is processed independently by [`helpers/transcribe.py`](https://github.com/browser-use/video-use/blob/main/helpers/transcribe.py). This script invokes the **OpenAI Whisper** model to generate a **Scribe JSON** file (e.g., [`take_1.json`](https://github.com/browser-use/video-use/blob/main/take_1.json)). These files contain word-level timestamps, confidence scores, and duration metadata, ensuring that every take is annotated before entering the selection pool.

### Stage 2: Packing and Normalizing with pack_transcripts.py

The [`helpers/pack_transcripts.py`](https://github.com/browser-use/video-use/blob/main/helpers/pack_transcripts.py) script reads all per-take JSON files from the edit directory, trims silent sections, normalizes timestamps relative to a global zero point, and writes a single [`takes_packed.md`](https://github.com/browser-use/video-use/blob/main/takes_packed.md) file. This markdown file serves as the single source of truth, enumerating each take with its file path, **duration**, and **confidence score**.

### Stage 3: Beat Detection and Global Timeline Construction

[`helpers/timeline_view.py`](https://github.com/browser-use/video-use/blob/main/helpers/timeline_view.py) orchestrates the beat selection. It loads the packed metadata, extracts audio using `pydub.AudioSegment`, and runs `librosa.onset.onset_detect` with configurable `hop_length` and `backtrack` parameters to identify rhythmic peaks. Each beat is scored based on **RMS energy** (audio amplitude) and **speech density** (count of transcript tokens within ±0.5 seconds). The algorithm then sorts beats chronologically and filters them using a `min_interval` constraint (default 0.5 seconds), keeping only the highest-scoring beat when conflicts occur. The final beat list is serialized to [`timeline_beats.json`](https://github.com/browser-use/video-use/blob/main/timeline_beats.json).

## How Multiple Takes Are Managed

The pipeline treats each take as an isolated asset during ingestion. The transcription step produces `take_<N>.json` files that never interfere with one another. During packing, [`pack_transcripts.py`](https://github.com/browser-use/video-use/blob/main/pack_transcripts.py) extracts the *take number*, *duration*, and *confidence score* from the Whisper output and appends them to the consolidated markdown. This allows the timeline builder to reference dozens of takes through a single unified view without needing to access individual source files again.

## Beat Selection Logic Across Clips

Cross-clip beat selection occurs in three steps. First, [`timeline_view.py`](https://github.com/browser-use/video-use/blob/main/timeline_view.py) detects local beats for every take using **Librosa**'s onset detection. Second, it assigns a composite score to each beat: **energy** (RMS amplitude of a short window) plus **speech density** (count of transcript tokens nearby). Third, the global merge iterates through chronologically sorted beats and applies a spacing constraint; if two beats fall within `min_interval` seconds, the lower-scoring beat is discarded. This produces a rhythmically consistent timeline that may span multiple disparate clips.

## Practical Implementation: Code Examples

### Packing Takes into a Unified Markdown File

```python

# helpers/pack_transcripts.py – command-line usage

import pathlib
import argparse

parser = argparse.ArgumentParser(
    description="Pack Scribe transcripts into takes_packed.md"
)
parser.add_argument(
    "edit_dir", type=pathlib.Path,
    help="Directory containing take_<N>.json files"
)
parser.add_argument(
    "-o", "--output", type=pathlib.Path,
    help="Output path (default: <edit-dir>/takes_packed.md)"
)
args = parser.parse_args()

out_path = args.output or (args.edit_dir / "takes_packed.md")

# Internal logic reads every take_*.json, normalizes timestamps,

# extracts duration and confidence, and writes the markdown file.

print(f"✅ Packed takes saved to {out_path}")

```

*Source:* [[`helpers/pack_transcripts.py`](https://github.com/browser-use/video-use/blob/main/helpers/pack_transcripts.py)](https://github.com/browser-use/video-use/blob/main/helpers/pack_transcripts.py)

### Detecting and Scoring Beats Globally

```python

# helpers/timeline_view.py – core workflow

from pathlib import Path
import librosa
import json
import pydub

def load_packed(packed_md: Path) -> list[dict]:
    """Parse takes_packed.md into a list of take metadata."""
    ...

def detect_beats(audio_path: Path, sr: int = 22050) -> list[float]:
    """Return beat timestamps in seconds using Librosa onset detection."""
    y, _ = librosa.load(audio_path, sr=sr)
    onset_frames = librosa.onset.onset_detect(
        y=y, sr=sr, hop_length=512, backtrack=True
    )
    return librosa.frames_to_time(onset_frames, sr=sr)

def score_beat(timestamp: float, take: dict) -> float:
    """Calculate composite score from RMS energy and speech density."""
    energy = calculate_rms(take["audio_path"], timestamp)
    density = count_tokens_near(take["transcript"], timestamp, window=0.5)
    return energy * 0.6 + density * 0.4

def build_global_timeline(packed: list[dict], min_interval: float = 0.5):
    all_beats = []
    for take in packed:
        local_beats = detect_beats(take["audio_path"])
        for ts in local_beats:
            score = score_beat(ts, take)
            all_beats.append((ts, score, take["take_id"]))

    # Sort chronologically and enforce minimum spacing

    all_beats.sort(key=lambda x: x[0])
    timeline = []
    last_time = -float("inf")
    for ts, sc, tid in all_beats:
        if ts - last_time >= min_interval:
            timeline.append({"time": ts, "score": sc, "take": tid})
            last_time = ts
    return timeline

if __name__ == "__main__":
    packed = load_packed(Path("takes_packed.md"))
    beats = build_global_timeline(packed, min_interval=0.5)
    Path("timeline_beats.json").write_text(json.dumps(beats, indent=2))
    print("✅ Timeline with beats written to timeline_beats.json")

```

*Source:* [[`helpers/timeline_view.py`](https://github.com/browser-use/video-use/blob/main/helpers/timeline_view.py)](https://github.com/browser-use/video-use/blob/main/helpers/timeline_view.py)

### Rendering the Final Montage

```python

# helpers/render.py – typical usage

from pathlib import Path
from moviepy.editor import VideoFileClip, concatenate_videoclips
import json

def load_beats(json_path: Path) -> list[dict]:
    with open(json_path) as f:
        return json.load(f)

beats = load_beats("timeline_beats.json")
clips = []
for beat in beats:
    # Extract a 2-second segment around each selected beat

    start = beat["time"]
    end = start + 2.0
    clip = VideoFileClip("final_take.mp4").subclip(start, end)
    clips.append(clip)

final = concatenate_videoclips(clips)
final.write_videofile("montage.mp4", codec="libx264", audio_codec="aac")

```

*Source:* [[`helpers/render.py`](https://github.com/browser-use/video-use/blob/main/helpers/render.py)](https://github.com/browser-use/video-use/blob/main/helpers/render.py)

## Summary

- **Independent Ingestion:** Each take is transcribed separately into Scribe JSON files by [`helpers/transcribe.py`](https://github.com/browser-use/video-use/blob/main/helpers/transcribe.py), preventing cross-contamination during the initial speech-to-text phase.
- **Unified Metadata:** [`helpers/pack_transcripts.py`](https://github.com/browser-use/video-use/blob/main/helpers/pack_transcripts.py) consolidates all takes into [`takes_packed.md`](https://github.com/browser-use/video-use/blob/main/takes_packed.md), normalizing timestamps and exposing duration and confidence metadata.
- **Intelligent Beat Scoring:** [`helpers/timeline_view.py`](https://github.com/browser-use/video-use/blob/main/helpers/timeline_view.py) uses Librosa onset detection and a composite scoring function (energy + speech density) to rank beats within each clip.
- **Global Merge with Constraints:** The pipeline enforces a configurable `min_interval` (default 0.5s) when merging beats across clips, ensuring rhythmic coherence in the final [`timeline_beats.json`](https://github.com/browser-use/video-use/blob/main/timeline_beats.json).
- **Flexible Rendering:** The resulting beat file is consumed by [`helpers/render.py`](https://github.com/browser-use/video-use/blob/main/helpers/render.py) (or any MoviePy-based script) to splice segments into the final montage.

## Frequently Asked Questions

### How does the pipeline handle takes with no detectable speech?

Whisper still generates a low-confidence transcript even for silent or musical segments. When speech density is near zero, the beat-selection algorithm in [`helpers/timeline_view.py`](https://github.com/browser-use/video-use/blob/main/helpers/timeline_view.py) relies primarily on **RMS energy** to score beats, ensuring that purely musical or ambient takes remain eligible for the final cut.

### Can the minimum interval between beats be adjusted?

Yes. The `min_interval` parameter in [`helpers/timeline_view.py`](https://github.com/browser-use/video-use/blob/main/helpers/timeline_view.py) is fully configurable (default 0.5 seconds). Increasing this value forces sparser, more deliberate cuts, while decreasing it allows for rapid-fire montage editing suitable for high-energy content.

### What audio formats are supported for beat detection?

[`helpers/timeline_view.py`](https://github.com/browser-use/video-use/blob/main/helpers/timeline_view.py) utilizes `pydub.AudioSegment`, which supports any format that FFmpeg handles (MP3, WAV, AAC, FLAC, etc.). The pipeline automatically resamples audio to the rate required by Librosa (default 22,050 Hz) during processing, so no manual conversion is required.

### How are transcript timestamps aligned after packing?

[`helpers/pack_transcripts.py`](https://github.com/browser-use/video-use/blob/main/helpers/pack_transcripts.py) normalizes all timestamps relative to a global zero point by accumulating the durations of preceding takes. This ensures that [`takes_packed.md`](https://github.com/browser-use/video-use/blob/main/takes_packed.md) contains monotonically increasing timestamps that [`helpers/timeline_view.py`](https://github.com/browser-use/video-use/blob/main/helpers/timeline_view.py) can treat as a continuous timeline, even when beats originate from disparate source files.