How the Whisper Fallback Mechanism Handles Unavailable Captions in Claude-Video

The Whisper fallback mechanism automatically transcribes video audio when native captions are missing by detecting empty caption results, extracting 16kHz mono MP3 audio, chunking files that exceed API limits, and uploading to Groq or OpenAI backends with automatic retry logic.

The claude-video repository by bradautomates provides intelligent video processing skills that prioritize native captions but seamlessly activate the Whisper fallback mechanism when subtitles are unavailable. Understanding this cascading pipeline is essential for processing videos without embedded or auto-generated captions.

The Two-Stage Transcript Acquisition Pipeline

The /watch skill implements a cascading strategy that attempts local caption retrieval before invoking external transcription APIs.

Stage 1: Native Caption Retrieval via yt-dlp

First, watch.py calls fetch_captions() from skills/watch/scripts/download.py, which executes yt-dlp with --write-subs --write-auto-subs flags to retrieve the best available VTT file (download.py line 65-70). If successful, parse_vtt() processes the subtitle file and sets transcript_source = "captions" (watch.py line 101-105).

Stage 2: Conditional Whisper Activation

When captions are unavailable, watch.py validates four specific conditions before triggering the fallback (watch.py line 39-44):

• No transcript segments exist (not transcript_segments)
• The user hasn't disabled Whisper (not args.no_whisper)
• The video contains audio tracks (meta.get("has_audio"))
• Valid API credentials are configured

Only when all conditions pass does the script invoke transcribe_video().

How the Whisper Fallback Mechanism Processes Audio

The transcribe_video() function in skills/watch/scripts/whisper.py executes a complete audio processing pipeline when the fallback activates.

Audio Extraction and Formatting

The system extracts a mono 16kHz MP3 stream using extract_audio(), optimizing for speech recognition accuracy while minimizing upload bandwidth (whisper.py line 15-39).

Chunked Upload for API Limits

If the audio file exceeds backend size constraints, plan_chunks() calculates segmentation points and split_audio() divides the stream into manageable chunks (whisper.py line 40-58).

Backend Selection and Multipart Requests

The implementation prefers Groq for speed with OpenAI as the fallback provider. Each chunk uploads via handcrafted multipart HTTP requests (whisper.py line 37-46). After transcription, the system shifts timestamps to match the original video timeline and returns combined segments (whisper.py line 71-81).

Error Handling and Edge Cases

The fallback includes robust error management for credential and network failures.

Missing API Key Detection

When load_api_key() fails to locate credentials, watch.py prints a diagnostic message directing users to run setup.py for configuration (watch.py line 55-64).

Retry Logic and Network Failures

If uploads encounter HTTP 429 or network errors, whisper.py implements automatic retries before raising SystemExit. The calling code in watch.py catches this exception and logs "whisper fallback failed" (watch.py line 52-53).

Graceful Degradation Without Transcripts

When Whisper is unavailable or fails, the final report notes that only frames will be returned and advises enabling transcription for future runs (watch.py line 77-84).

Practical Usage Examples

Enabling Automatic Fallback on Uncaptioned Videos

Run the skill on videos without native captions to trigger automatic Whisper processing:

watch https://example.com/video-without-captions.mp4

Typical output shows the transition:

[watch] checking metadata/captions via yt-dlp…
[watch] extracting audio for Whisper (groq)…
[watch] transcribed 312 segments via groq
- **Transcript:** 312 segments (via whisper (groq))

Disabling the Fallback Mechanism

Force frames-only mode when transcription isn't needed using the --no-whisper flag:

watch https://example.com/video.mp4 --no-whisper

This produces:

[watch] no transcript available — proceed with frames only.
- **Transcript:** none available

Specifying a Specific Backend

Override the default Groq preference with the --whisper argument:

watch https://example.com/video.mp4 --whisper openai

If OPENAI_API_KEY is configured, the transcript source reports as whisper (openai).

Summary

• The Whisper fallback mechanism activates only when yt-dlp returns no caption files and all prerequisites (audio present, API keys valid, user hasn't opted out) are satisfied.
• Audio processing converts video to 16kHz mono MP3, automatically chunking files that exceed API size constraints.
• Backend flexibility supports Groq as the primary provider with OpenAI as fallback, using handcrafted multipart uploads for compatibility.
• Error resilience includes credential validation hints, automatic retries for rate limits, and graceful degradation to frames-only output when transcription fails.

Frequently Asked Questions

What triggers the Whisper fallback mechanism?

The fallback triggers when fetch_captions() returns no VTT files from yt-dlp, resulting in zero transcript segments after parsing. According to the source code in watch.py, the system checks not transcript_segments combined with not args.no_whisper, valid API credentials loaded via load_api_key(), and audio presence (meta.get("has_audio")) before invoking transcribe_video().

Which Whisper backends does claude-video support?

The implementation supports Groq as the preferred backend for faster inference and OpenAI as the fallback option. Users select backends via the --whisper argument (accepting values groq or openai), with credentials loaded from environment variables or configuration files by load_api_key().

How does the system handle large video files?

Large audio files automatically split into chunks using plan_chunks() to calculate segmentation boundaries and split_audio() to perform the physical file splitting (whisper.py line 40-58). Each chunk uploads separately to the API, and the system shifts timestamps when stitching segments back together to maintain synchronization with the original video timeline.

What happens if the Whisper API is unavailable?

When API calls fail after exhaustion of retries (network errors, HTTP 429 rate limits, or invalid keys), whisper.py raises SystemExit, which watch.py catches to log "whisper fallback failed". The skill continues execution, returning only video frames with a diagnostic note explaining that transcripts are unavailable and suggesting the use of --no-whisper for future runs to skip the transcription attempt.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}