How Claude-Video Extracts Audio for Whisper Transcription: MP3 Format and ffmpeg Pipeline Explained

Claude-video converts video soundtracks to mono 16 kHz MP3 at 64 kbps (~480 kB/min) using ffmpeg, then feeds this compact audio file to OpenAI or Groq Whisper APIs.

The claude-video repository handles video-to-text workflows by first extracting a Whisper-compatible audio track. Understanding this extraction process helps you debug failures, optimize for API limits, or adapt the pipeline for your own video transcription needs.

Audio Format: Mono 16 kHz MP3 at 64 kbps

The target format is deliberately constrained to satisfy Whisper API requirements:

• Container: MP3
• Codec: libmp3lame
• Sample rate: 16,000 Hz (16 kHz)
• Channels: 1 (mono)
• Bitrate: 64 kbps

This produces files of approximately 480 kB per minute, well under the 25 MiB upload limit imposed by both OpenAI and Groq Whisper endpoints. The mono channel and 16 kHz rate match Whisper's native training specifications, avoiding unnecessary bloat from stereo or high-frequency content.

The extract_audio Function in whisper.py

Audio extraction is performed by extract_audio in skills/watch/scripts/whisper.py. The function follows a three-phase pipeline:

1. Validate ffmpeg availability — aborts with actionable installation instructions if missing
2. Prepare output directory — creates parent directories as needed
3. Execute ffmpeg compression — applies hardcoded audio parameters

# from https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/whisper.py

def extract_audio(video_path: str, out_path: Path) -> Path:
    """Extract mono 16kHz 64kbps mp3 — ~480 kB/min, fits any Whisper limit."""
    if shutil.which("ffmpeg") is None:
        raise SystemExit("ffmpeg is not installed. Install with: brew install ffmpeg")

    out_path.parent.mkdir(parents=True, exist_ok=True)
    cmd = [
        "ffmpeg",
        "-hide_banner",
        "-loglevel", "error",
        "-y",
        "-i", str(Path(video_path).resolve()),
        "-vn",
        "-acodec", "libmp3lame",
        "-ar", "16000",
        "-ac", "1",
        "-b:a", "64k",
        str(out_path.resolve()),
    ]
    result = subprocess.run(cmd, capture_output=True, text=True)
    if result.returncode != 0:
        raise SystemExit(f"ffmpeg audio extraction failed: {result.stderr.strip()}")
    if not out_path.exists() or out_path.stat().st_size == 0:
        raise SystemExit("ffmpeg produced no audio — video may have no audio track")
    return out_path

ffmpeg Flags Explained

Flag	Purpose
-vn	Disable video stream (audio-only output)
-acodec libmp3lame	Select MP3 encoder
-ar 16000	Force 16 kHz sample rate
-ac 1	Force single (mono) channel
-b:a 64k	Cap bitrate at 64 kbps
-y	Overwrite existing files without prompt

The function validates ffmpeg's exit code and output file existence, raising descriptive errors for common failure modes like missing audio tracks or codec issues.

Integration with the Transcription Pipeline

The extracted MP3 flows into transcribe_video, also defined in whisper.py. This orchestration function:

1. Loads API credentials for OpenAI or Groq
2. Calls extract_audio to generate the MP3
3. Checks file size against API limits
4. Splits oversized audio into chunked MP3s if necessary (preserving the same format parameters)
5. Uploads to the selected backend and aggregates transcription segments

Chunking occurs transparently — the initial extraction format never changes, only the segmentation strategy adapts to API constraints.

Practical Usage Examples

Extract and Transcribe in One Call

from pathlib import Path
from skills.watch.scripts.whisper import transcribe_video

video_path = "example.mp4"
audio_out = Path("example_audio.mp3")

segments, backend = transcribe_video(
    video_path=video_path,
    audio_out=audio_out,
    backend="openai",  # or "groq"

)

print(f"Transcribed with {backend}:")
for seg in segments[:5]:
    print(f"[{seg['start']:.2f}s → {seg['end']:.2f}s] {seg['text']}")

Extract Audio Only

from pathlib import Path
from skills.watch.scripts.whisper import extract_audio

video = "sample.mov"
audio_file = Path("sample_audio.mp3")
extracted = extract_audio(video, audio_file)

print(f"Audio saved to: {extracted} ({extracted.stat().st_size / 1024:.0f} kB)")

Key Files in the Audio Pipeline

File	Role
skills/watch/scripts/whisper.py	Core workflow: audio extraction, chunking, API upload, segment aggregation
skills/watch/scripts/watch.py	Entry point coordinating download, frame extraction, and transcription
skills/watch/scripts/transcribe.py	WebVTT subtitle parsing (alternative to Whisper path)

Summary

• Claude-video extracts audio for Whisper transcription using extract_audio in whisper.py
• Output format is mono 16 kHz MP3 at 64 kbps, producing ~480 kB/min files
• ffmpeg with libmp3lame handles the conversion; ffmpeg must be installed separately
• The same format persists even when audio is split for API size limits
• Error handling covers missing ffmpeg, failed extraction, and silent video inputs

Frequently Asked Questions

What audio format does claude-video use for Whisper transcription?

Claude-video generates mono 16 kHz MP3 files encoded at 64 kbps. This format balances Whisper's technical requirements with API upload constraints, producing compact files that preserve speech intelligibility while staying well under provider size limits.

Why does claude-video convert audio to 16 kHz instead of keeping higher sample rates?

Whisper models were trained on 16 kHz audio. Higher sample rates increase file size without improving transcription accuracy, since the model downsamples internally. The claude-video pipeline eliminates this waste by forcing 16 kHz at the extraction stage.

What happens if my video's audio track is longer than the API upload limit?

The transcribe_video function in whisper.py detects oversized MP3s and splits them into smaller segments. Each chunk maintains the same mono 16 kHz 64 kbps MP3 format. Segments are transcribed individually and reassembled into a continuous transcript.

Do I need to install ffmpeg separately to use claude-video?

Yes. The extract_audio function checks for ffmpeg availability via shutil.which("ffmpeg") and exits with installation instructions if absent. On macOS, the error message suggests brew install ffmpeg; adapt for your platform as needed.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}