How video-use Performs Speaker Diarization and Audio Event Tagging

video-use delegates speaker diarization and audio event tagging to the ElevenLabs Scribe API by sending specific configuration flags during the transcription request and processing the returned JSON to preserve non-speech markers.

The video-use open-source repository provides a lightweight pipeline for extracting spoken content and acoustic events from video files. Rather than implementing complex machine learning models locally, the project leverages the cloud-based ElevenLabs Scribe speech-to-text service to handle speaker separation and sound classification. This architecture keeps the codebase minimal while providing production-grade diarization capabilities through straightforward API integration.

How Speaker Diarization and Audio Event Tagging Work in video-use

The implementation follows a three-stage pipeline that converts video input into structured transcript data containing both speech segments and audio events.

Step 1: Audio Extraction with ffmpeg

First, the system extracts a mono WAV file optimized for speech recognition. In helpers/transcribe.py, the extract_audio() function shells out to ffmpeg to convert the input video into a single-channel 16kHz audio stream:

# From helpers/transcribe.py

command = [
    "ffmpeg",
    "-i", video_path,
    "-vn",                    # No video

    "-acodec", "pcm_s16le",   # PCM 16-bit little-endian

    "-ac", "1",               # Mono (1 channel)

    "-ar", "16000",           # 16kHz sample rate

    output_path
]

This standardization ensures compatibility with the Scribe API's expected input format while reducing bandwidth and processing time.

Step 2: Configuring the ElevenLabs Scribe API Request

The core diarization and event tagging logic resides in the call_scribe() function within helpers/transcribe.py. Here, the code constructs a POST request to https://api.elevenlabs.io/v1/speech-to-text with two critical boolean flags enabled:

payload = {
    "model_id": "scribe_v1",
    "diarize": "true",           # Enable speaker separation

    "tag_audio_events": "true",  # Enable laughter, applause, etc.

    "timestamps_granularity": "word",
    "num_speakers": num_speakers # Optional: hint for expected speaker count

}

Setting diarize to "true" instructs the API to cluster speech segments by speaker identity, while tag_audio_events triggers detection of non-speech sounds like laughter, applause, and background noises.

Step 3: Processing the Response in pack_transcripts.py

After receiving the API response, helpers/pack_transcripts.py handles the JSON parsing. The Scribe service returns a words array where each entry contains a type field distinguishing between "word", "spacing", and "audio_event" entries.

The packing logic specifically filters and retains audio_event objects alongside spoken words:

# From helpers/pack_transcripts.py

for word in words:
    if word["type"] == "word":
        # Process spoken word with speaker label

        speaker = word.get("speaker", "Unknown")
        text = word["text"]
    elif word["type"] == "audio_event":
        # Preserve non-speech markers like [laugh] or [applause]

        event_type = word.get("audio_event", "unknown")

This preservation ensures that final transcripts indicate both who is speaking and what environmental sounds occur throughout the video timeline.

Practical Usage Examples

Basic Transcription with Diarization Enabled

To process a video with default settings (speaker diarization and audio events both enabled):

python helpers/transcribe.py path/to/video.mp4

This command extracts audio, uploads it to Scribe with the diarization flags set, and writes the raw JSON response to edit_dir/transcripts/<video_stem>.json.

Specifying Expected Speaker Count

When the number of speakers is known in advance, pass the --num-speakers flag to improve diarization accuracy:

python helpers/transcribe.py path/to/video.mp4 --num-speakers 2

This maps to the num_speakers parameter in the API payload, providing the diarization algorithm with a helpful constraint for speaker clustering.

Generating the Final Transcript

Convert the raw JSON into a readable format that includes audio event markers:

python helpers/pack_transcripts.py edit_dir/transcripts/video.json

The output includes speaker labels (e.g., "Speaker 1:") and bracketed audio events (e.g., "[applause]") suitable for timeline rendering or subtitle generation.

Key Implementation Files

The speaker diarization and audio event tagging pipeline relies on these specific components:

• helpers/transcribe.py – Contains extract_audio() for ffmpeg audio extraction and call_scribe() for API communication with the ElevenLabs Scribe service
• helpers/pack_transcripts.py – Parses the Scribe JSON response, filtering words arrays to preserve both speech entries and audio_event markers
• helpers/transcribe_batch.py – Wrapper script for processing multiple videos sequentially with the same diarization configuration

Summary

• video-use does not implement native speaker diarization; it delegates this task to the ElevenLabs Scribe API via HTTP requests.
• The diarize and tag_audio_events flags in helpers/transcribe.py activate cloud-based speaker separation and acoustic event detection.
• Audio is standardized to mono 16kHz WAV format before transmission to ensure optimal recognition performance.
• helpers/pack_transcripts.py processes the API response to retain non-speech events alongside transcribed words, producing comprehensive transcripts that include both dialogue and sound effects.

Frequently Asked Questions

Does video-use perform speaker diarization natively?

No, the repository does not contain local machine learning models for speaker diarization. According to the source code in helpers/transcribe.py, the project sends audio to the ElevenLabs Scribe API with the diarize: "true" parameter, relying entirely on the external service to perform speaker clustering and identification.

What audio format does video-use send to the Scribe API?

The pipeline converts all input videos to mono WAV files with a 16kHz sample rate using ffmpeg. This specific configuration in extract_audio() (found in helpers/transcribe.py) ensures compatibility with the Scribe API's requirements while minimizing file size for faster uploads.

How are audio events represented in the final transcript?

Audio events appear as entries with type: "audio_event" in the Scribe JSON response. When processed by helpers/pack_transcripts.py, these entries are converted into bracketed markers such as [laugh] or [applause] within the final transcript text, maintaining chronological alignment with the spoken content.

Can I limit the number of speakers detected in a video?

Yes, the transcription script accepts an optional --num-speakers argument that maps to the num_speakers field in the API payload. Providing this value helps the Scribe service's diarization algorithm constrain its speaker clustering logic, potentially improving accuracy when the expected speaker count is known beforehand.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}