How Claude-Video Handles Videos Without Captions or Transcripts: A Deep Dive into the Fallback Pipeline

Claude-Video implements a three-tier fallback strategy that first attempts to retrieve embedded captions, then automatically generates transcripts using Whisper speech-to-text, and finally falls back to frame-only extraction with clear user notifications when neither option is available.

When processing videos that lack native captions, the bradautomates/claude-video repository employs a robust decision tree to ensure maximum content extraction. This open-source tool located in skills/watch/scripts/watch.py prioritizes transcript availability through multiple pathways before gracefully degrading to visual analysis alone.

Tier 1: Retrieving Embedded VTT Captions

The first attempt to handle videos without transcripts occurs before any video download completes. In watch.py (lines 97-105), the script calls fetch_captions from download.py to query for available VTT subtitle files without pulling the actual video content.

If the source platform hosts embedded subtitles, parse_vtt immediately converts these caption files into structured transcript segments. This lightweight approach avoids unnecessary processing when textual data already exists on the server.

Tier 2: On-Demand Whisper Speech-to-Text

When no embedded captions are available, Claude-Video automatically invokes Whisper speech-to-text provided the user has not disabled the feature with the --no-whisper flag. The logic in watch.py (lines 39-52) performs the following sequence:

1. Audio verification – Checks meta.get("has_audio") to confirm the video contains an audio stream
2. Backend initialization – Calls load_api_key to configure the appropriate Whisper backend (Groq or OpenAI)
3. Transcription execution – Runs transcribe_video to generate a complete transcript on-the-fly

This fallback ensures that spoken content remains accessible even when publishers do not provide captions.

Tier 3: Graceful Degradation to Frame Extraction

When transcript generation is impossible, Claude-Video continues processing rather than failing. The tool handles two specific edge cases in watch.py:

Missing Whisper Configuration – If the API key is unavailable or the user specified --no-whisper, the script prints a setup hint directing users to configure credentials via setup.py (lines 55-63), then proceeds to extract frames.

No Audio Stream – If meta.get("has_audio") returns false, the script skips transcription entirely and proceeds with frame extraction only (lines 65-66).

User Feedback and Reporting Logic

The generated markdown report clearly communicates the transcript status through three distinct messages defined in watch.py:

• Successful transcription – Displays the actual transcript text when either captions or Whisper succeed
• Frames-only mode – Shows "No transcript available – proceed with frames only…" when both caption retrieval and Whisper fail (lines 77-84)
• Transcript-only detail failure – Displays "No transcript available at transcript detail…" when the user explicitly requests --detail transcript but no textual content can be produced (lines 68-73)

Practical Usage Examples

Control the fallback behavior using these command-line patterns:

# Default behavior: attempts captions first, then Whisper if needed

watch https://www.youtube.com/watch?v=example

# Disable Whisper fallback – forces caption-only or frames-only mode

watch https://www.youtube.com/watch?v=example --no-whisper

# Force transcript-only mode (no frames). Warns if captions/Whisper unavailable

watch https://www.youtube.com/watch?v=example --detail transcript

Key Implementation Files

The transcript acquisition pipeline relies on modular components across the repository:

• skills/watch/scripts/watch.py – Orchestrates the entire workflow, deciding between caption retrieval, Whisper invocation, or frame extraction
• skills/watch/scripts/download.py – Handles fetch_captions to pull VTT subtitles without downloading video files
• skills/watch/scripts/whisper.py – Manages backend selection and runs transcribe_video for speech-to-text conversion
• skills/watch/scripts/transcribe.py – Parses VTT files into transcript segments
• skills/watch/scripts/setup.py – Configures Whisper API keys when users need to enable speech-to-text

Summary

• Claude-Video attempts embedded caption retrieval first via fetch_captions and parse_vtt before processing video bytes
• If captions are absent, it automatically falls back to Whisper speech-to-text using transcribe_video and load_api_key
• Users can disable Whisper with --no-whisper to force caption-only or frames-only processing
• Videos without audio streams skip transcription entirely and proceed with frame extraction
• The tool provides clear status messages in the final report indicating whether transcripts, frames, or both are available

Frequently Asked Questions

What happens if I don't have a Whisper API key configured?

If no API key is present and the video lacks captions, Claude-Video prints a hint directing you to run the setup script to configure Whisper credentials, then proceeds with frame extraction only. The final report will display "No transcript available – proceed with frames only…" as implemented in watch.py lines 77-84.

Can I force Claude-Video to skip Whisper and use only captions?

Yes. Pass the --no-whisper flag when running the watch command. This disables the speech-to-text fallback, allowing the tool to proceed directly to frame extraction if no embedded captions are found.

How does the tool handle videos without any audio track?

When meta.get("has_audio") returns false, Claude-Video skips the transcription workflow entirely and proceeds with frame extraction only. This check occurs in watch.py lines 65-66, ensuring the tool does not attempt to process silent videos.

What is the difference between --no-whisper and --detail transcript?

The --no-whisper flag disables the speech-to-text backend entirely, allowing frame extraction when captions are unavailable. The --detail transcript flag forces the tool to output only transcript content without frames; if neither captions nor Whisper can produce text, it displays "No transcript available at transcript detail…" and exits without generating frames.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}