How Claude Video Handles Audio Extraction for Whisper Transcription

Claude Video uses ffmpeg to extract a lightweight mono 16kHz MP3 audio track from videos, automatically chunks files exceeding 24 MiB to respect Whisper API limits, and reconstructs accurate timelines by shifting segment offsets after transcription.

The bradautomates/claude-video repository provides a watch skill that transforms video content into searchable text by leveraging OpenAI's Whisper API. The system handles everything from format optimization to large-file chunking within the skills/watch/scripts/whisper.py module, ensuring reliable transcription regardless of video length.

FFmpeg-Based Audio Extraction

The transcription pipeline begins in skills/watch/scripts/whisper.py with the extract_audio() function. This utility invokes ffmpeg to strip the audio track from the source video and convert it to a mono 16kHz MP3 encoded at approximately 64 kbps.

This specific configuration produces a compact file—roughly 480 kB per minute of audio—while maintaining the fidelity required for accurate speech recognition. The 16kHz sample rate aligns with Whisper's native expectations, eliminating unnecessary resampling overhead during API processing.

# Conceptual usage within the transcription flow

from skills.watch.scripts.whisper import extract_audio
from pathlib import Path

video_path = Path("lecture.mp4")
audio_path = Path("lecture_audio.mp3")

# Extracts mono 16kHz MP3 optimized for Whisper

extract_audio(video_path, audio_path)

Chunking Strategy for Large Files

Once extracted, the transcribe_video() function validates the audio file size against the MAX_UPLOAD_BYTES constant (24 MiB). If the audio exceeds this threshold, the system invokes plan_chunks() to calculate an optimal segmentation strategy, followed by split_audio() to divide the MP3 into smaller fragments.

Each chunk is processed independently through the _post_whisper() helper, which constructs a multipart/form-data payload using _build_multipart() and transmits it to the selected backend endpoint—either GROQ_ENDPOINT or OPENAI_ENDPOINT. This chunking mechanism ensures compliance with the 25 MB upload ceiling while maintaining transcription continuity across segment boundaries.

# Simplified flow showing chunk handling

def transcribe_video(video_path: Path, audio_path: Path):
    extract_audio(video_path, audio_path)
    audio_bytes = audio_path.read_bytes()
    
    if len(audio_bytes) > MAX_UPLOAD_BYTES:  # 24 MiB limit

        chunks = plan_chunks(audio_path)  # Determine split points

        segments = []
        for chunk_file, offset in chunks:
            response = _post_whisper(chunk_file, backend="groq")
            chunk_segments = _segments_from_response(response)
            segments.extend(shift_segments(chunk_segments, offset))
        return segments
    else:
        return _post_whisper(audio_path, backend="openai")

Timestamp Reconstruction and Alignment

Because Whisper processes each chunk independently, returned timestamps begin at zero for every segment. To restore chronological alignment with the original video, the pipeline employs shift_segments() within whisper.py. This function offsets each transcription segment's start and end times by the chunk's original position in the audio timeline.

The result is a continuous, accurately timed transcript where the first word of the second chunk correctly aligns with the video timestamp where that chunk begins, rather than resetting to zero.

Integration in the Watch Workflow

The extraction and transcription logic integrates seamlessly into the broader video processing workflow via skills/watch/scripts/watch.py. This orchestration module imports transcribe_video() and executes it immediately after video download completion. The resulting segment list—containing precise text, start times, and end times—is then formatted for display or fed into downstream query-answering components.

# From skills/watch/scripts/watch.py

from skills.watch.scripts.whisper import transcribe_video

def process_video(video_url: str):
    # ... download logic ...

    audio_file = Path("temp_audio.mp3")
    segments, backend_used = transcribe_video(downloaded_video, audio_file)
    return format_transcript(segments)

Summary

• Audio extraction occurs via extract_audio() in whisper.py, producing a mono 16kHz MP3 at ~64 kbps to minimize file size while preserving speech clarity.
• Size validation against MAX_UPLOAD_BYTES (24 MiB) triggers automatic chunking via plan_chunks() and split_audio() when videos exceed API upload limits.
• API transmission uses _post_whisper() with _build_multipart() to send audio to either Groq or OpenAI Whisper endpoints.
• Timestamp alignment is restored through shift_segments(), which offsets chunk-relative times to match the original video timeline.
• Workflow integration happens in watch.py, which orchestrates the entire pipeline from download to searchable transcript.

Frequently Asked Questions

What audio format does Claude Video use for Whisper transcription?

Claude Video extracts audio as a mono MP3 file with a 16kHz sample rate encoded at approximately 64 kbps. This format balances compression efficiency with the fidelity required for accurate speech recognition, resulting in files roughly 480 kB per minute.

How does Claude Video handle videos longer than 25 MB?

When the extracted audio exceeds the MAX_UPLOAD_BYTES threshold of 24 MiB, the system automatically calculates a chunking plan using plan_chunks() and splits the audio into smaller segments with split_audio(). Each chunk is transcribed independently, and timestamps are reconstructed using shift_segments() to maintain chronological alignment.

Why does Claude Video shift segment timestamps after transcription?

Whisper returns timestamps relative to the start of each audio chunk (always beginning at zero). Since Claude Video splits large files into multiple chunks, shift_segments() adds the original offset of each chunk to its respective transcription segments, ensuring the final transcript aligns accurately with the original video timeline.

Which Whisper backends does Claude Video support?

The codebase supports both Groq and OpenAI Whisper endpoints. The _post_whisper() function in whisper.py constructs the appropriate multipart payload and routes it to the selected backend, allowing flexibility in API provider choice based on availability or pricing preferences.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}