How claude-video Performs Audio Extraction for Whisper API: Mono 16kHz MP3 Pipeline

claude-video extracts audio for the Whisper API by running a targeted FFmpeg command that converts video to mono 16kHz MP3 at 64kbps, then chunks large files using stream-copy to stay under API limits while preserving transcript continuity.

The claude-video repository provides a Python-based pipeline for transcribing video content using OpenAI's Whisper service. At the core of this system lies a specialized audio extraction process that optimizes file size and format compliance for API upload constraints. All processing lives in skills/watch/scripts/whisper.py and requires only Python ≥3.9 with the external binaries ffmpeg and ffprobe.

FFmpeg Audio Extraction Pipeline

The extract_audio function in skills/watch/scripts/whisper.py orchestrates the conversion using a single FFmpeg invocation. This command forces specific codec parameters to meet Whisper API requirements while minimizing bandwidth usage for cloud transmission.

Mono 16kHz Codec Configuration

The implementation passes -acodec libmp3lame to select the MP3 encoder, combined with -ar 16000 for the 16kHz sample rate and -ac 1 to enforce single-channel mono output. The bitrate is capped at 64kbps using -b:a 64k, producing the exact format required for efficient transcription.

ffmpeg -hide_banner -loglevel error -y -i <video> -vn \
       -acodec libmp3lame -ar 16000 -ac 1 -b:a 64k <out>.mp3

File Size Efficiency

These settings yield approximately 480 kB per minute of audio, ensuring uploads remain well below the 25MB limits imposed by both Groq and OpenAI endpoints. This compression ratio allows roughly 50 minutes of audio per upload while maintaining transcription accuracy.

Duration Detection and Chunk Planning

Before upload, the system measures audio length and segments files exceeding size thresholds to ensure API compliance.

Measuring Audio with ffprobe

The audio_duration function calls ffprobe with JSON output parsing to determine the exact length of the generated MP3. This measurement enables precise chunk planning when processing long-form video content.

Chunk Strategy for API Limits

When files exceed the MAX_UPLOAD_BYTES constant (24 MiB safety margin), the plan_chunks function calculates evenly-sized time slices that keep each segment under the byte ceiling. This planning occurs before any network requests, preventing upload failures.

Stream-Copy Chunking Without Re-encoding

The split_audio function uses FFmpeg's stream-copy mode (-c copy) to carve temporal slices from the MP3 without re-encoding. This preserves the original 16kHz mono quality and minimizes CPU overhead during processing, as the operation becomes a simple data transfer rather than a computational transcoding task.

API Upload and Transcript Assembly

The pipeline handles multipart construction and timestamp reconciliation across chunks to produce seamless results.

Multipart Request Construction

The _build_multipart function crafts the HTTP body for uploading MP3 segments to the chosen backend, preferring Groq when available with OpenAI as a fallback. The function manually constructs the multipart/form-data boundaries to ensure compatibility with both service endpoints.

Timestamp Alignment Across Chunks

Since the Whisper API returns timestamps starting at zero for every independent chunk, the shift_segments function offsets each segment by its original start time in the source video. This mathematical adjustment stitches multiple chunks into a single continuous transcript with accurate timing.

Implementation Examples

Run transcription from the command line using the default backend detection:

python3 -m skills.watch.scripts.whisper /path/to/video.mp4

Use the pipeline programmatically in Python:

from pathlib import Path
from skills.watch.scripts.whisper import transcribe_video

video_path = "example.mp4"
audio_out = Path("example_audio.mp3")
segments, backend = transcribe_video(video_path, audio_out)

print(f"Transcribed with {backend}:")
for seg in segments:
    print(f"[{seg['start']:.2f}–{seg['end']:.2f}] {seg['text']}")

Force a specific backend (OpenAI or Groq):

segments, backend = transcribe_video(
    video_path="example.mp4",
    audio_out=Path("tmp_audio.mp3"),
    backend="openai"
)

Summary

• extract_audio in whisper.py runs FFmpeg with -acodec libmp3lame -ar 16000 -ac 1 -b:a 64k to generate mono 16kHz MP3 files
• File size remains approximately 480 kB per minute, staying under the 25MB API limit
• audio_duration uses ffprobe JSON output to measure exact audio length for chunk planning
• split_audio employs stream-copy (-c copy) to segment large files without re-encoding
• shift_segments adjusts timestamps to create continuous transcripts from chunked uploads
• The system requires only Python ≥3.9, ffmpeg, and ffprobe with no additional Python dependencies

Frequently Asked Questions

What audio format does claude-video output for the Whisper API?

The pipeline produces mono MP3 files at 16kHz sample rate with 64kbps bitrate. This specific format balances transcription accuracy with file size efficiency, generating roughly 480 kB per minute of audio.

How does claude-video handle videos larger than 25MB?

The system checks file size against the MAX_UPLOAD_BYTES constant (24 MiB margin). Files exceeding this limit are split using plan_chunks to calculate time slices, then split_audio extracts segments using FFmpeg's stream-copy mode without re-encoding.

Why does claude-video use mono instead of stereo audio?

The -ac 1 flag forces single-channel mono output because the Whisper API processes speech content that does not require stereo separation. Mono encoding cuts file size in half compared to stereo while maintaining full transcription accuracy for spoken content.

Which Python file contains the audio extraction logic?

All audio processing functions—including extract_audio, audio_duration, plan_chunks, and split_audio—reside in skills/watch/scripts/whisper.py. This module handles the complete flow from video input to timestamped text segments.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}