# How video-use Implements Speaker Diarization and Audio Event Tagging with ElevenLabs Scribe

> Implement speaker diarization and audio event tagging in video-use with ElevenLabs Scribe. Extract mono 16kHz audio with ffmpeg and process JSON for accurate transcripts.

- Repository: [Browser Use/video-use](https://github.com/browser-use/video-use)
- Tags: internals
- Published: 2026-07-03

---

**The video-use repository delegates speaker diarization and audio event tagging to the ElevenLabs Scribe API, extracting mono 16kHz audio via ffmpeg and post-processing the JSON response to preserve audio events in the final transcript.**

The browser-use/video-use project handles complex audio analysis without implementing heavy machine learning models locally. Instead, it leverages the ElevenLabs Scribe speech-to-text service to perform **speaker diarization** and **audio event tagging**, keeping the codebase lightweight while providing accurate multi-speaker transcripts and non-speech sound detection.

## Audio Extraction and Standardization

Before sending data to the cloud API, `video-use` normalizes the source video into a format Scribe expects. The `extract_audio()` function in [`helpers/transcribe.py`](https://github.com/browser-use/video-use/blob/main/helpers/transcribe.py) converts the input video to a mono-channel WAV file sampled at 16kHz.

```python

# helpers/transcribe.py lines 49-55

def extract_audio(video_path: Path, output_path: Path):
    # ffmpeg command: -ac 1 (mono), -ar 16000 (16kHz)

    subprocess.run([
        "ffmpeg", "-i", str(video_path), 
        "-vn", "-ac", "1", "-ar", "16000", 
        str(output_path)
    ], check=True)

```

This standardization ensures compatibility with Scribe's ingestion pipeline and reduces payload size by removing unnecessary channels.

## Enabling Diarization and Event Tagging in the API Call

The core implementation resides in [`helpers/transcribe.py`](https://github.com/browser-use/video-use/blob/main/helpers/transcribe.py), specifically within the `call_scribe()` function. Rather than processing audio locally, the code transmits the extracted WAV file to `https://api.elevenlabs.io/v1/speech-to-text` with explicit flags to enable advanced features.

The payload includes two critical parameters:

```python

# helpers/transcribe.py lines 64-68

payload = {
    "model_id": "scribe_v1",
    "diarize": "true",           # Enable speaker diarization

    "tag_audio_events": "true",  # Enable audio event tagging

    "timestamps_granularity": "word",
}

```

Setting `diarize` to `"true"` instructs the service to partition the transcript by speaker. The `tag_audio_events` flag triggers detection of non-speech sounds such as laughter, applause, or background noise.

## Post-Processing Scribe Responses

Once the API returns results, [`helpers/pack_transcripts.py`](https://github.com/browser-use/video-use/blob/main/helpers/pack_transcripts.py) processes the JSON to integrate audio events into the readable transcript. The Scribe response contains a `words` array where each entry carries a `type` field indicating whether it is a `"word"`, `"spacing"`, or `"audio_event"`.

The post-processor specifically filters and retains `audio_event` entries alongside spoken words, as seen in lines 45-66 of [`helpers/pack_transcripts.py`](https://github.com/browser-use/video-use/blob/main/helpers/pack_transcripts.py). This preservation allows downstream tools to display markers like `[laugh]` or `[applause]` within the transcript timeline.

## Practical Usage Examples

### Transcribe a Single Video with Full Features

Run the transcription helper to extract audio, send it to Scribe, and save the JSON output:

```bash
python helpers/transcribe.py path/to/video.mp4

```

This executes the default pipeline with both **speaker diarization** and **audio event tagging** enabled.

### Specify Expected Speaker Count

When the number of speakers is known, pass the `--num-speakers` argument to improve diarization accuracy:

```bash
python helpers/transcribe.py path/to/video.mp4 --num-speakers 2

```

This maps to the `num_speakers` parameter in the Scribe API request.

### Generate Final Transcripts with Audio Events

Convert the raw JSON into a markdown-compatible transcript that includes audio event markers:

```bash
python helpers/pack_transcripts.py edit_dir/transcripts/video.json

```

The script keeps entries where `type == "audio_event"`, ensuring non-speech sounds appear in the final output alongside dialogue.

## Summary

- **video-use** relies on the ElevenLabs Scribe API for computationally intensive tasks like speaker separation and sound classification.
- Audio extraction occurs in [`helpers/transcribe.py`](https://github.com/browser-use/video-use/blob/main/helpers/transcribe.py) using ffmpeg to create mono 16kHz WAV files.
- The API payload explicitly sets `diarize: "true"` and `tag_audio_events: "true"` to enable these features.
- Post-processing in [`helpers/pack_transcripts.py`](https://github.com/browser-use/video-use/blob/main/helpers/pack_transcripts.py) preserves `audio_event` entries from the Scribe response, integrating them into the final transcript.
- The architecture keeps the repository lightweight by outsourcing heavy ML inference to specialized cloud services.

## Frequently Asked Questions

### Does video-use perform speaker diarization locally?

No. According to the source code in [`helpers/transcribe.py`](https://github.com/browser-use/video-use/blob/main/helpers/transcribe.py), video-use does not implement local diarization algorithms. Instead, it sends audio to the ElevenLabs Scribe API with the `diarize` parameter set to `"true"`, allowing the external service to handle speaker separation.

### What audio format does video-use prepare for the Scribe API?

The codebase converts all input videos to mono-channel WAV files with a 16kHz sample rate. The `extract_audio()` function in [`helpers/transcribe.py`](https://github.com/browser-use/video-use/blob/main/helpers/transcribe.py) uses ffmpeg with the flags `-ac 1` and `-ar 16000` to ensure compatibility and optimize upload efficiency.

### What types of audio events can be detected?

The Scribe API supports various non-speech sounds when `tag_audio_events` is enabled. Based on the implementation in [`helpers/pack_transcripts.py`](https://github.com/browser-use/video-use/blob/main/helpers/pack_transcripts.py), the response parser handles entries marked as `"audio_event"` in the JSON, which typically includes sounds like laughter, applause, and background noises.

### How are audio events included in the final transcript?

The [`pack_transcripts.py`](https://github.com/browser-use/video-use/blob/main/pack_transcripts.py) script processes the Scribe JSON response and specifically retains entries where the `type` field equals `"audio_event"`. These events are preserved alongside `"word"` entries, allowing the final output to display markers such as `[laugh]` or `[applause]` within the text stream.