How Speaker Handoff Gaps Are Calculated for Different Pacing Styles in video-use

Speaker handoff gaps in video-use are calculated by measuring the silence duration between consecutive transcript entries and comparing it against a configurable silence_threshold (default 0.5s), where any gap exceeding this value triggers a new phrase boundary and cut point.

The browser-use/video-use repository automates video editing by analyzing ElevenLabs Scribe JSON transcripts to determine optimal speaker transition points. Understanding how speaker handoff gaps are calculated is essential for controlling the rhythm and flow of automated edits, from rapid-fire interviews to dramatic cinematic pacing.

Understanding Speaker Handoff Gap Detection

The detection logic resides in helpers/pack_transcripts.py, specifically within the group_into_phrases function. This utility processes the words array from ElevenLabs Scribe JSON, which contains three entry types: word, audio_event, and spacing.

Measuring Silence from Spacing Entries

When the parser encounters a spacing entry, it calculates the gap duration by subtracting the start time from the end time. This measurement represents the raw silence between adjacent transcript elements.

Threshold-Based Phrase Breaking

If the calculated gap meets or exceeds the silence_threshold, the current phrase is flushed and a new phrase begins. Additionally, any change in speaker_id automatically forces a phrase break regardless of gap duration, ensuring speaker boundaries are always respected.

if t == "spacing":
    gap = w.get("end") - w.get("start")
    if gap >= silence_threshold:        # ← handoff gap test

        flush()

Adjusting Pacing Styles via Silence Threshold

The silence_threshold parameter directly controls the editing pace. According to the SKILL.md documentation, typical handoff gaps range from 400ms to 600ms, with the recommendation: "Less for fast-paced, more for cinematic. Taste call."

Fast-Paced Style Configuration

For rapid dialogue or energetic content, reduce the threshold to capture shorter pauses as handoff points. This creates tighter cuts with minimal dead air between speakers.

python -m helpers.pack_transcripts \
    --edit-dir /path/to/edit \
    --silence-threshold 0.3   # fast-paced

Cinematic Style Configuration

For dramatic or documentary-style pacing, increase the threshold to allow longer breaths and pauses to remain within phrases, creating more deliberate transitions between speakers.

python -m helpers.pack_transcripts \
    --edit-dir /path/to/edit \
    --silence-threshold 0.6   # cinematic

Validating Handoff Gaps in Output

After processing, the generated takes_packed.md file reveals the calculated gaps in the timestamp boundaries. For example:

[002.52-005.36] S0 …   # speaker 0 phrase

[006.08-006.74] S1 …   # speaker 1 phrase (handoff gap ≈ 0.72 s)

The 720ms gap between 005.36 and 006.08 exceeds the default 500ms threshold, triggering a handoff cut. Had the threshold been set to 0.3s for fast-paced editing, this same gap would still qualify, but a smaller 250ms gap would be retained within a single phrase.

You can programmatically inspect these gaps using the pack_one_file function:

from pathlib import Path
from helpers.pack_transcripts import pack_one_file

edit_dir = Path("/path/to/edit")
json_path = edit_dir / "transcripts" / "take1.json"
_, _, phrases = pack_one_file(json_path, silence_threshold=0.5)

for p in phrases:
    print(f"[{p['start']:.2f}-{p['end']:.2f}] {p.get('speaker_id','?')} {p['text']}")

Summary

• Speaker handoff gaps are derived from spacing entries in ElevenLabs Scribe JSON transcripts by calculating end - start silence durations in helpers/pack_transcripts.py.
• The group_into_phrases function compares each gap against the silence_threshold (default 0.5s) to determine cut points.
• Pacing styles are controlled via the --silence-threshold CLI flag: lower values (0.3s) for fast-paced edits, higher values (0.6-0.8s) for cinematic pacing per SKILL.md guidelines.
• Speaker changes automatically trigger phrase breaks independent of gap duration, ensuring clean speaker transitions.
• Resulting phrase boundaries are stored in takes_packed.md and visualized in helpers/timeline_view.py before final rendering in helpers/render.py.

Frequently Asked Questions

What is the default silence threshold for speaker handoffs?

The default silence_threshold is 0.5 seconds (500ms). This value represents the minimum silence duration required between transcript elements to trigger a new phrase boundary and potential speaker handoff cut.

How does video-use handle speaker changes without silence?

When the speaker_id changes between consecutive words, the group_into_phrases function automatically flushes the current phrase and starts a new one, regardless of the measured gap duration. This ensures that speaker transitions are always respected as cut points even during rapid back-and-forth dialogue.

Can I use different thresholds for different speakers?

Currently, the silence_threshold is a global parameter applied uniformly across all speakers in a transcript. You cannot specify different thresholds per speaker through the standard CLI; however, you could process different transcript segments separately with varying thresholds and merge the results manually if needed.

Where are the calculated handoff gaps visible in the output?

The calculated gaps appear as time intervals between consecutive phrase entries in the generated takes_packed.md file. The helpers/timeline_view.py module also visualizes these gaps, and the final edit decision list (EDL) consumed by helpers/render.py uses these boundaries to execute the actual video cuts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}