How video-use Implements Speaker Diarization and Audio Event Tagging with ElevenLabs Scribe

The video-use repository delegates speaker diarization and audio event tagging to the ElevenLabs Scribe API, extracting mono 16kHz audio via ffmpeg and post-processing the JSON response to preserve audio events in the final transcript.

The browser-use/video-use project handles complex audio analysis without implementing heavy machine learning models locally. Instead, it leverages the ElevenLabs Scribe speech-to-text service to perform speaker diarization and audio event tagging, keeping the codebase lightweight while providing accurate multi-speaker transcripts and non-speech sound detection.

Audio Extraction and Standardization

Before sending data to the cloud API, video-use normalizes the source video into a format Scribe expects. The extract_audio() function in helpers/transcribe.py converts the input video to a mono-channel WAV file sampled at 16kHz.

# helpers/transcribe.py lines 49-55

def extract_audio(video_path: Path, output_path: Path):
    # ffmpeg command: -ac 1 (mono), -ar 16000 (16kHz)

    subprocess.run([
        "ffmpeg", "-i", str(video_path), 
        "-vn", "-ac", "1", "-ar", "16000", 
        str(output_path)
    ], check=True)

This standardization ensures compatibility with Scribe's ingestion pipeline and reduces payload size by removing unnecessary channels.

Enabling Diarization and Event Tagging in the API Call

The core implementation resides in helpers/transcribe.py, specifically within the call_scribe() function. Rather than processing audio locally, the code transmits the extracted WAV file to https://api.elevenlabs.io/v1/speech-to-text with explicit flags to enable advanced features.

The payload includes two critical parameters:

# helpers/transcribe.py lines 64-68

payload = {
    "model_id": "scribe_v1",
    "diarize": "true",           # Enable speaker diarization

    "tag_audio_events": "true",  # Enable audio event tagging

    "timestamps_granularity": "word",
}

Setting diarize to "true" instructs the service to partition the transcript by speaker. The tag_audio_events flag triggers detection of non-speech sounds such as laughter, applause, or background noise.

Post-Processing Scribe Responses

Once the API returns results, helpers/pack_transcripts.py processes the JSON to integrate audio events into the readable transcript. The Scribe response contains a words array where each entry carries a type field indicating whether it is a "word", "spacing", or "audio_event".

The post-processor specifically filters and retains audio_event entries alongside spoken words, as seen in lines 45-66 of helpers/pack_transcripts.py. This preservation allows downstream tools to display markers like [laugh] or [applause] within the transcript timeline.

Practical Usage Examples

Transcribe a Single Video with Full Features

Run the transcription helper to extract audio, send it to Scribe, and save the JSON output:

python helpers/transcribe.py path/to/video.mp4

This executes the default pipeline with both speaker diarization and audio event tagging enabled.

Specify Expected Speaker Count

When the number of speakers is known, pass the --num-speakers argument to improve diarization accuracy:

python helpers/transcribe.py path/to/video.mp4 --num-speakers 2

This maps to the num_speakers parameter in the Scribe API request.

Generate Final Transcripts with Audio Events

Convert the raw JSON into a markdown-compatible transcript that includes audio event markers:

python helpers/pack_transcripts.py edit_dir/transcripts/video.json

The script keeps entries where type == "audio_event", ensuring non-speech sounds appear in the final output alongside dialogue.

Summary

• video-use relies on the ElevenLabs Scribe API for computationally intensive tasks like speaker separation and sound classification.
• Audio extraction occurs in helpers/transcribe.py using ffmpeg to create mono 16kHz WAV files.
• The API payload explicitly sets diarize: "true" and tag_audio_events: "true" to enable these features.
• Post-processing in helpers/pack_transcripts.py preserves audio_event entries from the Scribe response, integrating them into the final transcript.
• The architecture keeps the repository lightweight by outsourcing heavy ML inference to specialized cloud services.

Frequently Asked Questions

Does video-use perform speaker diarization locally?

No. According to the source code in helpers/transcribe.py, video-use does not implement local diarization algorithms. Instead, it sends audio to the ElevenLabs Scribe API with the diarize parameter set to "true", allowing the external service to handle speaker separation.

What audio format does video-use prepare for the Scribe API?

The codebase converts all input videos to mono-channel WAV files with a 16kHz sample rate. The extract_audio() function in helpers/transcribe.py uses ffmpeg with the flags -ac 1 and -ar 16000 to ensure compatibility and optimize upload efficiency.

What types of audio events can be detected?

The Scribe API supports various non-speech sounds when tag_audio_events is enabled. Based on the implementation in helpers/pack_transcripts.py, the response parser handles entries marked as "audio_event" in the JSON, which typically includes sounds like laughter, applause, and background noises.

How are audio events included in the final transcript?

The pack_transcripts.py script processes the Scribe JSON response and specifically retains entries where the type field equals "audio_event". These events are preserved alongside "word" entries, allowing the final output to display markers such as [laugh] or [applause] within the text stream.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}