# How video-use Performs Speaker Diarization and Audio Event Tagging

> Learn how video-use performs speaker diarization and audio event tagging using the ElevenLabs Scribe API. Discover how it processes JSON for non-speech markers efficiently.

- Repository: [Browser Use/video-use](https://github.com/browser-use/video-use)
- Tags: how-to-guide
- Published: 2026-07-07

---

**video-use delegates speaker diarization and audio event tagging to the ElevenLabs Scribe API by sending specific configuration flags during the transcription request and processing the returned JSON to preserve non-speech markers.**

The **video-use** open-source repository provides a lightweight pipeline for extracting spoken content and acoustic events from video files. Rather than implementing complex machine learning models locally, the project leverages the cloud-based ElevenLabs Scribe speech-to-text service to handle speaker separation and sound classification. This architecture keeps the codebase minimal while providing production-grade diarization capabilities through straightforward API integration.

## How Speaker Diarization and Audio Event Tagging Work in video-use

The implementation follows a three-stage pipeline that converts video input into structured transcript data containing both speech segments and audio events.

### Step 1: Audio Extraction with ffmpeg

First, the system extracts a mono WAV file optimized for speech recognition. In [`helpers/transcribe.py`](https://github.com/browser-use/video-use/blob/main/helpers/transcribe.py), the `extract_audio()` function shells out to `ffmpeg` to convert the input video into a single-channel 16kHz audio stream:

```python

# From helpers/transcribe.py

command = [
    "ffmpeg",
    "-i", video_path,
    "-vn",                    # No video

    "-acodec", "pcm_s16le",   # PCM 16-bit little-endian

    "-ac", "1",               # Mono (1 channel)

    "-ar", "16000",           # 16kHz sample rate

    output_path
]

```

This standardization ensures compatibility with the Scribe API's expected input format while reducing bandwidth and processing time.

### Step 2: Configuring the ElevenLabs Scribe API Request

The core diarization and event tagging logic resides in the `call_scribe()` function within [`helpers/transcribe.py`](https://github.com/browser-use/video-use/blob/main/helpers/transcribe.py). Here, the code constructs a POST request to `https://api.elevenlabs.io/v1/speech-to-text` with two critical boolean flags enabled:

```python
payload = {
    "model_id": "scribe_v1",
    "diarize": "true",           # Enable speaker separation

    "tag_audio_events": "true",  # Enable laughter, applause, etc.

    "timestamps_granularity": "word",
    "num_speakers": num_speakers # Optional: hint for expected speaker count

}

```

Setting **diarize** to `"true"` instructs the API to cluster speech segments by speaker identity, while **tag_audio_events** triggers detection of non-speech sounds like laughter, applause, and background noises.

### Step 3: Processing the Response in pack_transcripts.py

After receiving the API response, [`helpers/pack_transcripts.py`](https://github.com/browser-use/video-use/blob/main/helpers/pack_transcripts.py) handles the JSON parsing. The Scribe service returns a `words` array where each entry contains a `type` field distinguishing between `"word"`, `"spacing"`, and `"audio_event"` entries.

The packing logic specifically filters and retains `audio_event` objects alongside spoken words:

```python

# From helpers/pack_transcripts.py

for word in words:
    if word["type"] == "word":
        # Process spoken word with speaker label

        speaker = word.get("speaker", "Unknown")
        text = word["text"]
    elif word["type"] == "audio_event":
        # Preserve non-speech markers like [laugh] or [applause]

        event_type = word.get("audio_event", "unknown")

```

This preservation ensures that final transcripts indicate both who is speaking and what environmental sounds occur throughout the video timeline.

## Practical Usage Examples

### Basic Transcription with Diarization Enabled

To process a video with default settings (speaker diarization and audio events both enabled):

```bash
python helpers/transcribe.py path/to/video.mp4

```

This command extracts audio, uploads it to Scribe with the diarization flags set, and writes the raw JSON response to `edit_dir/transcripts/<video_stem>.json`.

### Specifying Expected Speaker Count

When the number of speakers is known in advance, pass the `--num-speakers` flag to improve diarization accuracy:

```bash
python helpers/transcribe.py path/to/video.mp4 --num-speakers 2

```

This maps to the `num_speakers` parameter in the API payload, providing the diarization algorithm with a helpful constraint for speaker clustering.

### Generating the Final Transcript

Convert the raw JSON into a readable format that includes audio event markers:

```bash
python helpers/pack_transcripts.py edit_dir/transcripts/video.json

```

The output includes speaker labels (e.g., "Speaker 1:") and bracketed audio events (e.g., "[applause]") suitable for timeline rendering or subtitle generation.

## Key Implementation Files

The speaker diarization and audio event tagging pipeline relies on these specific components:

- **[`helpers/transcribe.py`](https://github.com/browser-use/video-use/blob/main/helpers/transcribe.py)** – Contains `extract_audio()` for ffmpeg audio extraction and `call_scribe()` for API communication with the ElevenLabs Scribe service
- **[`helpers/pack_transcripts.py`](https://github.com/browser-use/video-use/blob/main/helpers/pack_transcripts.py)** – Parses the Scribe JSON response, filtering `words` arrays to preserve both speech entries and `audio_event` markers
- **[`helpers/transcribe_batch.py`](https://github.com/browser-use/video-use/blob/main/helpers/transcribe_batch.py)** – Wrapper script for processing multiple videos sequentially with the same diarization configuration

## Summary

- **video-use** does not implement native speaker diarization; it delegates this task to the ElevenLabs Scribe API via HTTP requests.
- The `diarize` and `tag_audio_events` flags in [`helpers/transcribe.py`](https://github.com/browser-use/video-use/blob/main/helpers/transcribe.py) activate cloud-based speaker separation and acoustic event detection.
- Audio is standardized to mono 16kHz WAV format before transmission to ensure optimal recognition performance.
- [`helpers/pack_transcripts.py`](https://github.com/browser-use/video-use/blob/main/helpers/pack_transcripts.py) processes the API response to retain non-speech events alongside transcribed words, producing comprehensive transcripts that include both dialogue and sound effects.

## Frequently Asked Questions

### Does video-use perform speaker diarization natively?

No, the repository does not contain local machine learning models for speaker diarization. According to the source code in [`helpers/transcribe.py`](https://github.com/browser-use/video-use/blob/main/helpers/transcribe.py), the project sends audio to the ElevenLabs Scribe API with the `diarize: "true"` parameter, relying entirely on the external service to perform speaker clustering and identification.

### What audio format does video-use send to the Scribe API?

The pipeline converts all input videos to mono WAV files with a 16kHz sample rate using `ffmpeg`. This specific configuration in `extract_audio()` (found in [`helpers/transcribe.py`](https://github.com/browser-use/video-use/blob/main/helpers/transcribe.py)) ensures compatibility with the Scribe API's requirements while minimizing file size for faster uploads.

### How are audio events represented in the final transcript?

Audio events appear as entries with `type: "audio_event"` in the Scribe JSON response. When processed by [`helpers/pack_transcripts.py`](https://github.com/browser-use/video-use/blob/main/helpers/pack_transcripts.py), these entries are converted into bracketed markers such as `[laugh]` or `[applause]` within the final transcript text, maintaining chronological alignment with the spoken content.

### Can I limit the number of speakers detected in a video?

Yes, the transcription script accepts an optional `--num-speakers` argument that maps to the `num_speakers` field in the API payload. Providing this value helps the Scribe service's diarization algorithm constrain its speaker clustering logic, potentially improving accuracy when the expected speaker count is known beforehand.