How the Video-Use Pipeline Manages Multiple Takes and Selects Beats Across Clips

The video-use pipeline ingests independent video takes, transcribes them via OpenAI Whisper, aggregates the results into a unified metadata file, detects rhythmic beats using Librosa, and merges high-scoring beats across all clips while enforcing a configurable minimum interval to generate a synchronized montage.

The browser-use/video-use repository automates the editing workflow for multi-take video projects. By combining speech-to-text metadata with audio onset detection, the pipeline can manage an arbitrary number of source clips and intelligently select the best beats across them to create a cohesive final video.

The Three-Stage Pipeline Architecture

Stage 1: Per-Take Transcription with Scribe JSON

Each raw video file is processed independently by helpers/transcribe.py. This script invokes the OpenAI Whisper model to generate a Scribe JSON file (e.g., take_1.json). These files contain word-level timestamps, confidence scores, and duration metadata, ensuring that every take is annotated before entering the selection pool.

Stage 2: Packing and Normalizing with pack_transcripts.py

The helpers/pack_transcripts.py script reads all per-take JSON files from the edit directory, trims silent sections, normalizes timestamps relative to a global zero point, and writes a single takes_packed.md file. This markdown file serves as the single source of truth, enumerating each take with its file path, duration, and confidence score.

Stage 3: Beat Detection and Global Timeline Construction

helpers/timeline_view.py orchestrates the beat selection. It loads the packed metadata, extracts audio using pydub.AudioSegment, and runs librosa.onset.onset_detect with configurable hop_length and backtrack parameters to identify rhythmic peaks. Each beat is scored based on RMS energy (audio amplitude) and speech density (count of transcript tokens within ±0.5 seconds). The algorithm then sorts beats chronologically and filters them using a min_interval constraint (default 0.5 seconds), keeping only the highest-scoring beat when conflicts occur. The final beat list is serialized to timeline_beats.json.

How Multiple Takes Are Managed

The pipeline treats each take as an isolated asset during ingestion. The transcription step produces take_<N>.json files that never interfere with one another. During packing, pack_transcripts.py extracts the take number, duration, and confidence score from the Whisper output and appends them to the consolidated markdown. This allows the timeline builder to reference dozens of takes through a single unified view without needing to access individual source files again.

Beat Selection Logic Across Clips

Cross-clip beat selection occurs in three steps. First, timeline_view.py detects local beats for every take using Librosa's onset detection. Second, it assigns a composite score to each beat: energy (RMS amplitude of a short window) plus speech density (count of transcript tokens nearby). Third, the global merge iterates through chronologically sorted beats and applies a spacing constraint; if two beats fall within min_interval seconds, the lower-scoring beat is discarded. This produces a rhythmically consistent timeline that may span multiple disparate clips.

Practical Implementation: Code Examples

Packing Takes into a Unified Markdown File

# helpers/pack_transcripts.py – command-line usage

import pathlib
import argparse

parser = argparse.ArgumentParser(
    description="Pack Scribe transcripts into takes_packed.md"
)
parser.add_argument(
    "edit_dir", type=pathlib.Path,
    help="Directory containing take_<N>.json files"
)
parser.add_argument(
    "-o", "--output", type=pathlib.Path,
    help="Output path (default: <edit-dir>/takes_packed.md)"
)
args = parser.parse_args()

out_path = args.output or (args.edit_dir / "takes_packed.md")

# Internal logic reads every take_*.json, normalizes timestamps,

# extracts duration and confidence, and writes the markdown file.

print(f"✅ Packed takes saved to {out_path}")

Source: [helpers/pack_transcripts.py](https://github.com/browser-use/video-use/blob/main/helpers/pack_transcripts.py)

Detecting and Scoring Beats Globally

# helpers/timeline_view.py – core workflow

from pathlib import Path
import librosa
import json
import pydub

def load_packed(packed_md: Path) -> list[dict]:
    """Parse takes_packed.md into a list of take metadata."""
    ...

def detect_beats(audio_path: Path, sr: int = 22050) -> list[float]:
    """Return beat timestamps in seconds using Librosa onset detection."""
    y, _ = librosa.load(audio_path, sr=sr)
    onset_frames = librosa.onset.onset_detect(
        y=y, sr=sr, hop_length=512, backtrack=True
    )
    return librosa.frames_to_time(onset_frames, sr=sr)

def score_beat(timestamp: float, take: dict) -> float:
    """Calculate composite score from RMS energy and speech density."""
    energy = calculate_rms(take["audio_path"], timestamp)
    density = count_tokens_near(take["transcript"], timestamp, window=0.5)
    return energy * 0.6 + density * 0.4

def build_global_timeline(packed: list[dict], min_interval: float = 0.5):
    all_beats = []
    for take in packed:
        local_beats = detect_beats(take["audio_path"])
        for ts in local_beats:
            score = score_beat(ts, take)
            all_beats.append((ts, score, take["take_id"]))

    # Sort chronologically and enforce minimum spacing

    all_beats.sort(key=lambda x: x[0])
    timeline = []
    last_time = -float("inf")
    for ts, sc, tid in all_beats:
        if ts - last_time >= min_interval:
            timeline.append({"time": ts, "score": sc, "take": tid})
            last_time = ts
    return timeline

if __name__ == "__main__":
    packed = load_packed(Path("takes_packed.md"))
    beats = build_global_timeline(packed, min_interval=0.5)
    Path("timeline_beats.json").write_text(json.dumps(beats, indent=2))
    print("✅ Timeline with beats written to timeline_beats.json")

Source: [helpers/timeline_view.py](https://github.com/browser-use/video-use/blob/main/helpers/timeline_view.py)

Rendering the Final Montage

# helpers/render.py – typical usage

from pathlib import Path
from moviepy.editor import VideoFileClip, concatenate_videoclips
import json

def load_beats(json_path: Path) -> list[dict]:
    with open(json_path) as f:
        return json.load(f)

beats = load_beats("timeline_beats.json")
clips = []
for beat in beats:
    # Extract a 2-second segment around each selected beat

    start = beat["time"]
    end = start + 2.0
    clip = VideoFileClip("final_take.mp4").subclip(start, end)
    clips.append(clip)

final = concatenate_videoclips(clips)
final.write_videofile("montage.mp4", codec="libx264", audio_codec="aac")

Source: [helpers/render.py](https://github.com/browser-use/video-use/blob/main/helpers/render.py)

Summary

• Independent Ingestion: Each take is transcribed separately into Scribe JSON files by helpers/transcribe.py, preventing cross-contamination during the initial speech-to-text phase.
• Unified Metadata: helpers/pack_transcripts.py consolidates all takes into takes_packed.md, normalizing timestamps and exposing duration and confidence metadata.
• Intelligent Beat Scoring: helpers/timeline_view.py uses Librosa onset detection and a composite scoring function (energy + speech density) to rank beats within each clip.
• Global Merge with Constraints: The pipeline enforces a configurable min_interval (default 0.5s) when merging beats across clips, ensuring rhythmic coherence in the final timeline_beats.json.
• Flexible Rendering: The resulting beat file is consumed by helpers/render.py (or any MoviePy-based script) to splice segments into the final montage.

Frequently Asked Questions

How does the pipeline handle takes with no detectable speech?

Whisper still generates a low-confidence transcript even for silent or musical segments. When speech density is near zero, the beat-selection algorithm in helpers/timeline_view.py relies primarily on RMS energy to score beats, ensuring that purely musical or ambient takes remain eligible for the final cut.

Can the minimum interval between beats be adjusted?

Yes. The min_interval parameter in helpers/timeline_view.py is fully configurable (default 0.5 seconds). Increasing this value forces sparser, more deliberate cuts, while decreasing it allows for rapid-fire montage editing suitable for high-energy content.

What audio formats are supported for beat detection?

helpers/timeline_view.py utilizes pydub.AudioSegment, which supports any format that FFmpeg handles (MP3, WAV, AAC, FLAC, etc.). The pipeline automatically resamples audio to the rate required by Librosa (default 22,050 Hz) during processing, so no manual conversion is required.

How are transcript timestamps aligned after packing?

helpers/pack_transcripts.py normalizes all timestamps relative to a global zero point by accumulating the durations of preceding takes. This ensures that takes_packed.md contains monotonically increasing timestamps that helpers/timeline_view.py can treat as a continuous timeline, even when beats originate from disparate source files.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}