How Claude Video Handles Videos Without Existing Captions: A Technical Deep Dive

Claude Video employs a three-tier fallback strategy that first attempts to fetch embedded captions, falls back to Whisper speech-to-text transcription if audio is available, and gracefully degrades to frame-only analysis when neither option exists.

When processing videos that lack native transcripts or subtitles, the bradautomates/claude-video repository implements a robust pipeline designed to maximize content extraction while maintaining transparency with the user. The system prioritizes non-destructive caption retrieval before resorting to compute-intensive transcription, ensuring efficient handling of both captioned and uncaptioned source material.

The Layered Transcript Strategy

Claude Video's approach to handling videos without existing captions follows a deterministic, ordered workflow defined in skills/watch/scripts/watch.py. This architecture ensures that the tool attempts the fastest, cheapest methods first before escalating to API-dependent transcription services.

Step 1: Embedded Caption Retrieval

Before downloading video content, Claude Video attempts to extract existing subtitles without fetching the actual media file. The fetch_captions function in download.py queries the source URL (typically YouTube via yt-dlp) for available VTT subtitle tracks.

If subtitles are discovered, the system immediately downloads them and passes the content to parse_vtt in transcribe.py, which converts the VTT format into structured transcript segments. This occurs at lines 97-105 of watch.py, allowing the tool to avoid unnecessary video downloads and transcription costs when human-created captions already exist.

Step 2: Whisper Speech-to-Text Fallback

When no embedded captions are found and the user has not disabled the feature with --no-whisper, Claude Video proceeds to audio analysis. The system first validates the presence of an audio stream by checking meta.get("has_audio") from the video metadata.

If audio exists, the script invokes load_api_key to retrieve the appropriate Whisper backend credentials (Groq or OpenAI) and executes transcribe_video to generate a transcript on-the-fly. This fallback mechanism, implemented at lines 39-52 of watch.py, ensures that spoken content remains accessible even when publishers haven't provided captions.

Step 3: Graceful Degradation

When both caption retrieval and Whisper transcription are unavailable, Claude Video adapts its output rather than failing. The system distinguishes between two failure modes:

• Missing API Configuration: If Whisper is unavailable due to missing API keys or explicit --no-whisper flags, the script outputs a configuration hint directing users to run setup.py to configure their Whisper credentials (lines 55-63).
• Silent Videos: If the video lacks an audio track entirely, the system skips transcription and proceeds exclusively with frame extraction (lines 65-66).

Implementation Details and Reporting

The final markdown report reflects which tier of the strategy succeeded, ensuring users understand exactly what content was processed. According to the source code in watch.py, the transcript section displays one of three states:

1. Full transcript: When captions or Whisper succeeded
2. "No transcript available – proceed with frames only…": When both caption and Whisper methods failed but frame extraction continues (lines 77-84)
3. "No transcript available at transcript detail…": When the user explicitly requested --detail transcript mode but no transcription could be generated (lines 68-73)

CLI Usage Examples

The following commands demonstrate how to interact with Claude Video's caption handling logic:

# Default behavior: attempts captions first, then Whisper if needed

watch https://www.youtube.com/watch?v=example

# Disable Whisper fallback – forces frame-only mode when captions missing

watch https://www.youtube.com/watch?v=example --no-whisper

# Transcript-only mode – warns if neither captions nor Whisper available

watch https://www.youtube.com/watch?v=example --detail transcript

Summary

• Primary attempt: fetch_captions in download.py retrieves existing VTT subtitles without downloading video files
• Secondary fallback: transcribe_video via Whisper backend (Groq/OpenAI) processes audio when captions are absent and --no-whisper is not set
• Tertiary handling: Frame-only analysis proceeds when audio is missing or transcription is disabled, with clear user notifications
• Configuration: setup.py provides the interface for Whisper API key management, referenced automatically when transcription services are unavailable

Frequently Asked Questions

What happens if a video has no captions and I haven't configured Whisper?

Claude Video prints a helpful hint directing you to run the setup script to configure Whisper credentials, then continues with frame extraction only. The final report will display "No transcript available – proceed with frames only…" in the transcript section.

Can I force Claude Video to skip transcription entirely?

Yes. Pass the --no-whisper flag when invoking the watch command. This disables the Whisper fallback, forcing the tool to rely solely on embedded captions or proceed with frame extraction if none exist.

How does Claude Video handle silent videos without audio tracks?

If the video metadata indicates no audio stream (has_audio is false), the system skips transcription entirely at lines 65-66 of watch.py and proceeds directly to frame extraction. The report will indicate that no transcript is available due to the absence of audio content.

Does Claude Video download the entire video just to check for captions?

No. The fetch_captions function in download.py attempts to retrieve subtitle files via yt-dlp without downloading the video media itself, making the caption-checking phase lightweight and fast before any transcription decisions are made.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}