Transcript Fallback System in Claude Video: Automatic Whisper Transcription When Captions Are Missing

When yt-dlp cannot retrieve native captions, Claude Video automatically switches to OpenAI or Groq Whisper transcription, extracting audio from the video and converting speech to text in VTT-compatible segments.

The transcript fallback system in the bradautomates/claude-video repository ensures users always receive textual content from videos, even when subtitle tracks are unavailable. According to the source code in skills/watch/scripts/watch.py, the system first attempts to fetch native captions via yt-dlp, then seamlessly invokes a Whisper-based transcription pipeline when those captions are missing. This fallback requires either a Groq or OpenAI API key configured in ~/.config/watch/.env or a local .env file.

How the System Detects Missing Captions

The fallback mechanism begins immediately after the initial caption download attempt. In skills/watch/scripts/watch.py, the fetch_captions function attempts to retrieve subtitles via yt-dlp and stores the result in a download dictionary.

The code checks whether dl["subtitle_path"] exists in the returned data structure. If this key is absent, the transcript variables remain empty, signaling that native captions are unavailable and triggering the Whisper fallback pathway.

Configuring the Whisper Transcription Backend

The Whisper fallback is enabled by default, but users can control its behavior through command-line flags and configuration files.

Disabling the Fallback

To bypass audio transcription entirely and proceed with visual analysis only, use the --no-whisper flag implemented in watch.py at lines 53-56:

watch https://example.com/video.mp4 --no-whisper

API Key Resolution

The load_api_key function in skills/watch/scripts/whisper.py determines which backend to use by checking for valid credentials in the following order:

1. Groq — preferred provider if GROQ_API_KEY is present
2. OpenAI — fallback provider if OPENAI_API_KEY is present and Groq is unavailable

The function searches first in environment variables, then in ~/.config/watch/.env, and finally in a local .env file in the current directory.

Initial Setup

To scaffold the configuration file, run skills/watch/scripts/setup.py, which creates the .env file and prints configuration instructions:

python3 skills/watch/scripts/setup.py

Edit the created file at ~/.config/watch/.env to add your preferred key:

GROQ_API_KEY=your_groq_key

# OR

OPENAI_API_KEY=your_openai_key

The Audio-to-Text Transcription Pipeline

When captions are missing and a valid API key is detected, watch.py invokes the transcription pipeline at lines 252-263.

The transcribe_video function in whisper.py executes the following steps:

1. Audio Extraction — extracts the audio track from the video file
2. Chunk Planning — splits the audio into appropriately-sized segments to respect API limits
3. Parallel Upload — uploads each chunk to the chosen Whisper endpoint (Groq or OpenAI)
4. Segment Assembly — reconstructs the returned segments into VTT-compatible format using helpers from skills/watch/scripts/transcribe.py, including filter_range and format_transcript

The assembled transcript matches the same shape as native VTT subtitles, ensuring downstream processors handle caption-derived and Whisper-derived text identically.

Graceful Degradation to Frames-Only Mode

If the Whisper call fails due to network errors, invalid API keys, or missing configuration, the system degrades gracefully. In watch.py at lines 265-283, the error handling prints a helpful hint explaining the transcription failure, then continues execution with frames-only output.

This ensures the tool remains functional for visual analysis even when audio transcription is impossible. To force this behavior proactively, users can combine the --no-whisper flag with their command.

Summary

• The transcript fallback system activates when dl["subtitle_path"] is absent after the initial caption fetch attempt
• Whisper transcription is enabled by default but can be disabled via the --no-whisper flag (lines 53-56 in watch.py)
• The load_api_key function in whisper.py prefers Groq over OpenAI when resolving API credentials
• Audio processing occurs in transcribe_video (lines 252-263 in watch.py), which chunks audio and formats output using transcribe.py utilities
• When transcription fails or no API key exists, the system falls back to frames-only output (lines 265-283 in watch.py)

Frequently Asked Questions

What triggers the transcript fallback system in Claude Video?

The fallback triggers when the fetch_captions function fails to populate dl["subtitle_path"] in the download dictionary, indicating that yt-dlp could not retrieve native subtitles for the video. At this point, the script automatically attempts Whisper transcription if the user has not disabled it with --no-whisper.

How do I disable the Whisper fallback and use only visual frames?

Pass the --no-whisper flag when invoking the watch command. This bypasses the audio extraction and API calls entirely, forcing the system to proceed directly to frame extraction and visual analysis without attempting speech-to-text transcription.

Which Whisper API provider does Claude Video prefer?

According to the load_api_key implementation in whisper.py, the system prefers Groq over OpenAI. It checks for GROQ_API_KEY first in environment variables and configuration files, falling back to OPENAI_API_KEY only if Groq credentials are unavailable.

What happens if I don't have a Whisper API key configured?

If no valid API key is found in ~/.config/watch/.env, local .env, or environment variables, or if the API call fails, the script catches the error in watch.py (lines 265-283), prints a helpful message explaining the missing transcription, and continues with frames-only output. The video processing completes successfully using only visual information.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}