Whisper API Fallback for Unavailable Captions in Claude-Video: A Complete Implementation Guide

The Claude-Video watch skill automatically falls back to Whisper API transcription when native captions are missing, using a pure-Python implementation that supports both Groq and OpenAI backends with intelligent chunking and retry logic.

The bradautomates/claude-video repository provides a sophisticated video processing pipeline that ensures Claude always receives transcript data, even when source videos lack embedded subtitles. According to the source code, the watch skill (skills/watch/scripts/watch.py) implements a cascading fallback system that attempts native caption extraction first, then seamlessly transitions to cloud-based speech recognition via the Whisper API.

How the Caption Fallback Pipeline Works

The transcription workflow follows a strict priority order to maximize efficiency and minimize API costs.

Native Caption Detection

When processing a video, the pipeline first invokes scripts/transcribe.py to locate WebVTT files produced by yt-dlp. If a non-empty VTT file exists, the skill uses these native captions directly and skips audio extraction entirely. This avoids unnecessary API calls and preserves the original timing metadata.

Audio Extraction and Processing

If native captions are unavailable or the source is a local file without subtitles, watch.py calls transcribe_video() from skills/watch/scripts/whisper.py. This function handles the complete audio pipeline:

1. extract_audio() uses ffmpeg to generate a mono 16 kHz MP3 file (audio_out.mp3), producing approximately 480 kB per minute of video.
2. If the resulting audio exceeds the 24 MiB safety threshold, plan_chunks() calculates an even time-split that respects both size limits and API constraints.
3. split_audio() creates individual chunk files using ffmpeg with -c copy to avoid re-encoding, then processes each chunk individually.

API Key Discovery and Provider Selection

The load_api_key() function in whisper.py implements a hierarchical search strategy:

• Environment variables (GROQ_API_KEY or OPENAI_API_KEY)
• Configuration file at ~/.config/watch/.env (preferred for persistent storage)

Groq is the default provider due to superior speed and cost efficiency. OpenAI serves as an automatic fallback or can be forced via command-line flags.

Chunking and Upload Strategy for Large Files

Videos longer than approximately 50 minutes trigger the chunking system to comply with Whisper API file size limits (25 MB).

The _post_whisper() function manages multipart form construction manually through _build_multipart(), maintaining zero external dependencies beyond the Python standard library. This implementation includes a custom User-Agent header specifically to avoid Cloudflare blocks on the Groq endpoint.

After transcription, shift_segments() adjusts the timestamps from each chunk by adding the appropriate time offset, ensuring the final transcript aligns with the original video timeline rather than restarting at zero for each segment.

Retry Logic and Error Handling

The Whisper fallback implements robust error recovery with distinct strategies for different failure modes:

• Rate limiting (HTTP 429): Retries up to two times with exponential backoff, respecting the Retry-After header when present.
• Transient network errors: Up to three additional attempts with exponential backoff before failing gracefully.
• Missing API keys: The script prints a clear configuration hint and proceeds with frame-only output if --no-whisper is specified, setting SETUP_COMPLETE=true to suppress repeated warnings.

Configuring the Whisper API Fallback

Setting Up API Keys

Create a secure configuration file to persist your API credentials:

mkdir -p ~/.config/watch
cat > ~/.config/watch/.env <<'EOF'

# Groq (preferred - fastest and cheapest)

GROQ_API_KEY=your-groq-key-here

# OpenAI (fallback option)

# OPENAI_API_KEY=your-openai-key-here

EOF
chmod 600 ~/.config/watch/.env

Provider Selection and Usage

Run the skill with automatic fallback to Whisper when captions are unavailable:

python3 "$SKILL_DIR/skills/watch/scripts/watch.py" "https://example.com/video-without-captions.mp4"

Force a specific backend using the --whisper flag:

# Use OpenAI instead of default Groq

python3 skills/watch/scripts/watch.py "$URL" --whisper openai

Disable Whisper entirely for frames-only processing:

python3 skills/watch/scripts/watch.py "$URL" --no-whisper

Process specific time ranges with Whisper fallback:

python3 skills/watch/scripts/watch.py "$URL" --start 2:45 --end 3:10

Security and Privacy Considerations

The implementation minimizes data exposure by design. Only the extracted audio file (never the video or intermediate frames) transmits to external APIs. The ~/.config/watch/.env file requires mode 0600 permissions, and the codebase never logs API keys. According to the source code in skills/watch/scripts/whisper.py, all video processing and frame extraction remain local to the machine.

Summary

• The watch skill prioritizes native WebVTT captions and only invokes Whisper API when subtitles are missing or empty.
• transcribe_video() in skills/watch/scripts/whisper.py manages the complete fallback pipeline using pure Python standard library calls.
• Groq provides the default backend with OpenAI as an automatic or explicit fallback, discovered via environment variables or ~/.config/watch/.env.
• Intelligent chunking handles files up to 25 MB per segment, with shift_segments() reconstructing accurate timeline alignment.
• Retry logic handles rate limits and network errors with exponential backoff, while missing keys trigger helpful setup instructions rather than crashes.

Frequently Asked Questions

What happens if both Groq and OpenAI API keys are configured?

The system defaults to Groq for all transcriptions unless you explicitly specify --whisper openai on the command line. Groq is preferred due to faster processing speeds and lower costs, though both services provide equivalent accuracy for the 16 kHz mono MP3 input generated by the pipeline.

How does the skill handle videos that exceed the Whisper API file size limits?

When plan_chunks() detects an audio file larger than 24 MiB, it calculates an optimal split strategy and split_audio() divides the MP3 into smaller segments using ffmpeg's stream copy feature. Each chunk uploads separately, and shift_segments() adjusts the returned timestamps by adding the chunk's start time offset to maintain synchronization with the original video timeline.

Can I use the Whisper fallback with local video files?

Yes. The pipeline treats local files identically to downloaded content. If the local video lacks embedded subtitle tracks, watch.py automatically invokes transcribe_video() from whisper.py to extract audio and generate a transcript, provided you have configured a valid API key in the environment or ~/.config/watch/.env.

Is my API key secure when using this fallback mechanism?

The implementation stores keys exclusively in ~/.config/watch/.env with 0600 permissions and reads them via load_api_key() without logging or echoing them to stdout. The request construction in _build_multipart() and _post_whisper() transmits only the audio file content and necessary authentication headers; video data and extracted frames never leave your local machine.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}