Why video-use Relies Exclusively on Word-Level Verbatim ASR for Rendering

video-use uses word-level verbatim ASR because phrase-level transcripts lose sub-second timing data required for precise audio cuts, deterministic EDL generation, and reliable LLM-driven editing decisions.

The browser-use/video-use project implements a two-layer video processing architecture where an LLM edits video solely through text transcripts. This design choice makes word-level verbatim ASR (Automatic Speech Recognition) non-negotiable—not merely a preference, but a hard architectural requirement encoded in the source code.

The Two-Layer Architecture

video-use separates video processing into distinct consumption and rendering layers:

1. LLM consumption layer — The model reads a compact text representation (takes_packed.md) to decide cuts, grades, and overlays
2. Rendering layer — Executes precise edits based on timestamps from the transcript

This split exists because the LLM never sees raw video frames. As noted in README.md (lines 79-80), the transcript serves as the LLM's complete and only view of the source material. Any timing imprecision in this layer propagates directly to visual glitches in the final output.

Four Technical Reasons for Word-Level Verbatim ASR

Precision ≥ Sub-Second

Video editing rules depend on exact speech boundaries and silent gaps. Phrase-level outputs—such as Whisper-generated SRT files—lose sub-second gap data that video-use requires foroperations like 30 ms audio fades and cut-point enforcement.

SKILL.md explicitly rejects phrase-level transcripts in lines 312-313, stating that only word-level verbatim output preserves the granularity needed for professional-grade editing.

Deterministic Edit Decision Lists

The pipeline builds an EDL (Edit Decision List) where each entry references precise start and end time ranges. Using word-level timestamps guarantees that an LLM instruction like "cut after the word 'but' at 02.54 s" resolves unambiguously.

The renderer honors these timestamps directly without interpolation or guessing, eliminating frame-level drift that accumulates with coarser granularity.

Consistent LLM View

SKILL.md (lines 12-13) mandates verbatim transcription—no filler normalization, no word merging. This preserves:

• Natural cadence that affects pacing decisions
• Filler words ("um", "uh") that influence storytelling rhythm
• Exact speaker transitions for diarization-aware cuts

Normalization would create a mismatched world: the LLM plans edits based on cleaned text, but the renderer executes against original audio timing.

Hosted Scribe API Integration

helpers/transcribe.py implements this requirement explicitly. Lines 68-69 show the API call to ElevenLabs Scribe:

# helpers/transcribe.py (simplified)

response = client.transcribe(
    file=audio_path,
    timestamps_granularity="word",  # Hard-coded requirement

    diarize=True
)

The returned JSON contains a words array where each element carries:

{
  "text": "Hello",
  "start": 0.12,
  "end": 0.34,
  "speaker": "S0"
}

This structure propagates directly through the pipeline without transformation.

Practical Implementation

Obtaining Word-Level Transcripts

python helpers/transcribe.py path/to/video.mp4

Output writes to edit/transcripts/video.json with the word-level structure shown above.

Building an EDL from Transcript Data

import json
import pathlib

transcript = json.load(open("edit/transcripts/video.json"))
edl = {"ranges": []}

for word in transcript["words"]:
    # Example: mark cut point after specific keyword

    if word["text"].lower() == "but":
        edl["ranges"].append({
            "source": "video",
            "start": word["start"],
            "end": word["end"],
            "beat": "CUT",
            "quote": word["text"]
        })

pathlib.Path("edit/edl.json").write_text(json.dumps(edl, indent=2))

Rendering with Precision

python helpers/render.py edit/edl.json -o final.mp4

Because each EDL range references word-level timestamps, the renderer splices at exact sample boundaries without audible pops or visual frame errors.

Source File Reference

File	Critical Function
helpers/transcribe.py	Enforces timestamps_granularity: "word" in ElevenLabs API calls
SKILL.md	Documents the hard rule rejecting phrase-level ASR outputs
README.md	Explains two-layer architecture and timestamp requirements
takes_packed.md	Runtime file containing LLM-facing transcript representation

Summary

• Sub-second precision — Word-level granularity preserves gap data for fade and cut rules
• Deterministic editing — EDL entries map unambiguously to audio timestamps
• Verbatim fidelity — Filler words and natural cadence remain available to the LLM
• API-enforced — helpers/transcribe.py hard-codes timestamps_granularity: "word" with no fallback

These constraints collectively enable pixel-perfect cuts without frame-by-frame visual analysis, making automatic video editing computationally feasible.

Frequently Asked Questions

What happens if I use phrase-level ASR with video-use?

The system will fail to enforce editing rules reliably. SKILL.md explicitly prohibits phrase-level transcripts because coarse timestamps make it impossible to calculate accurate 30 ms audio fades or locate precise cut points. The EDL generator would need to estimate word positions within phrases, introducing timing errors that compound through the render pipeline.

Does video-use support any ASR provider besides ElevenLabs Scribe?

According to the source in helpers/transcribe.py, the project currently implements only the ElevenLabs Scribe API. The timestamps_granularity: "word" parameter is ElevenLabs-specific. Alternative providers would need to expose equivalent word-level timestamp and speaker diarization features to satisfy the architectural requirements documented in SKILL.md.

Why does verbatim transcription matter for LLM editing?

Normalization removes filler words and disfluencies that humans use for natural pacing. When an LLM plans cuts based on cleaned text but the audio preserves original timing, the resulting edits destroy conversational rhythm. SKILL.md (lines 12-13) mandates verbatim output so the LLM's text view matches the acoustic reality it indirectly controls.

Can I use Whisper with video-use if I force word-level timestamps?

Standard Whisper outputs SRT or VTT files that lack the structured JSON format with speaker diarization that video-use consumes. While Whisper has word-level timestamp capabilities, the project explicitly excludes it per SKILL.md guidance. The ElevenLabs Scribe integration in helpers/transcribe.py handles both transcription and diarization in a single API call with guaranteed schema compatibility.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}