How Whisper Fallback Works When Captions Are Unavailable in Claude Video

The /watch skill automatically falls back to OpenAI or Groq Whisper transcription when yt-dlp cannot retrieve native captions, extracting audio, chunking large files, and stitching segments only when the user hasn't disabled the feature and an API key is configured.

When processing videos without embedded subtitles, the bradautomates/claude-video repository provides a robust transcription pipeline. The /watch skill first attempts to fetch native captions using yt-dlp, then conditionally invokes Whisper transcription as a fallback mechanism. This ensures users receive transcript data even for videos lacking official caption tracks.

The Caption-First Retrieval Flow

The transcription process always prioritizes native captions over automated speech recognition. In skills/watch/scripts/watch.py, the system first calls fetch_captions() from download.py, which executes yt-dlp with --write-subs --write-auto-subs to retrieve the best available VTT file (download.py lines 65-70).

If a subtitle file is successfully returned, watch.py parses it using parse_vtt() and sets transcript_source = "captions" (watch.py lines 101-105). This bypasses the Whisper pipeline entirely, preserving API quota and ensuring accuracy when official captions exist.

Triggering the Whisper Fallback Mechanism

When native captions are missing, the fallback logic activates only after verifying several conditions. Following the caption parsing step, watch.py checks whether any transcript segments were produced. If the segments list is empty and the user hasn't disabled Whisper via --no-whisper, the system proceeds with audio transcription (watch.py lines 39-44):

if not transcript_segments and not args.no_whisper and video_path and meta.get("has_audio"):
    backend, api_key = load_api_key(args.whisper)
    if backend and api_key:
        all_segments, used_backend = transcribe_video(
            video_path, work / "audio.mp3", backend=backend, api_key=api_key
        )

This conditional ensures the fallback runs only when:

• No caption segments exist
• The video contains audio tracks (meta.get("has_audio"))
• A valid video file path is available
• The user has not passed the --no-whisper flag
• A valid API key is configured for Groq or OpenAI

The Whisper Transcription Pipeline

The transcribe_video() function in skills/watch/scripts/whisper.py implements the complete audio-to-text pipeline. This process handles large files through intelligent chunking and supports multiple API backends.

Audio Extraction and Formatting

The pipeline begins by extracting mono 16kHz MP3 audio using extract_audio() (whisper.py lines 15-39). This standardization ensures compatibility with Whisper's expected input format regardless of the source video codec.

Chunking for API Limits

If the extracted audio exceeds the API's file size limit, the system calculates optimal split points using plan_chunks() and divides the file via split_audio() (whisper.py lines 40-58). This allows transcription of long-form content without hitting provider restrictions.

Backend Selection and Upload

The system uploads each audio chunk to the selected Whisper backend using handcrafted multipart requests (whisper.py lines 37-46). Groq serves as the preferred provider for its speed and cost efficiency, with OpenAI acting as the fallback when specified via --whisper openai or when Groq credentials are unavailable.

After receiving API responses, the pipeline collects segments, adjusts timestamps to match the original video timeline, and returns the combined transcript (whisper.py lines 71-81).

Error Handling and Graceful Degradation

The fallback system includes multiple layers of error handling to ensure the skill remains functional even when transcription fails.

If no API key is found during initialization, watch.py displays a helpful hint directing users to run the setup script (setup.py) to configure credentials (watch.py lines 55-64).

For API failures such as network errors or HTTP 429 rate limiting, whisper.py implements retry logic. After exhausting retries, it raises a SystemExit that watch.py catches and logs as "whisper fallback failed" (watch.py lines 52-53).

When Whisper is unavailable or fails completely, the final report explicitly notes that only video frames will be returned and points users to enable Whisper for future runs (watch.py lines 77-84).

Practical Usage Examples

Running With Default Whisper Fallback

Process a video without native captions using the default Groq backend:

watch https://example.com/video-without-captions.mp4

Expected output:

[watch] checking metadata/captions via yt-dlp…
[watch] downloading video via yt-dlp…
[watch] extracting audio for Whisper (groq)…
[watch] transcribed 312 segments via groq

Disabling Whisper Fallback

Skip transcription entirely and receive frames-only output:

watch https://example.com/video-without-captions.mp4 --no-whisper

Expected output:

[watch] checking metadata/captions via yt-dlp…
[watch] downloading video via yt-dlp…
[watch] no transcript available — proceed with frames only.

Specifying the OpenAI Backend

Force the system to use OpenAI instead of Groq:

watch https://example.com/video.mp4 --whisper openai

The transcript source will be reported as whisper (openai) in the final output.

Summary

• Caption priority: The system always attempts to fetch native captions via yt-dlp before invoking Whisper.
• Conditional activation: Whisper fallback only triggers when captions are missing, audio is present, and the user hasn't disabled it with --no-whisper.
• Robust pipeline: Audio extraction converts files to mono 16kHz MP3, with automatic chunking for large files that exceed API limits.
• Dual backend support: Groq is the default provider for transcription speed, with OpenAI available as a fallback option.
• Graceful degradation: Missing API keys trigger setup instructions, while API failures result in frames-only output rather than script crashes.

Frequently Asked Questions

What happens if a video has no captions and I don't have an API key configured?

If native captions are unavailable and no Groq or OpenAI API key is present, watch.py prints a hint directing you to run setup.py to configure credentials (watch.py lines 55-64). The skill then proceeds with frames-only output, skipping transcription entirely.

Can I use Whisper fallback even if a video has native captions?

Currently, the logic in watch.py skips Whisper entirely when parse_vtt() successfully returns segments and sets transcript_source = "captions" (watch.py lines 101-105). There is no override flag to force Whisper transcription when official subtitles exist.

How does the system handle very long videos that exceed API file size limits?

The transcribe_video() function in whisper.py detects oversized audio files and automatically splits them into chunks using plan_chunks() and split_audio() (whisper.py lines 40-58). Each chunk is transcribed separately, and the segments are recombined with adjusted timestamps to maintain synchronization with the original video.

What is the difference between Groq and OpenAI Whisper backends?

Groq serves as the default backend due to faster inference speeds and competitive pricing. OpenAI functions as the fallback when specified via --whisper openai or when Groq credentials are unavailable. Both use the identical audio extraction and chunking pipeline in whisper.py, differing only in the API endpoint and authentication headers used for the multipart upload (whisper.py lines 37-46).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}