# How the Whisper Fallback Mechanism Handles Unavailable Captions in Claude-Video

> Discover how the Whisper fallback mechanism transcribes missing video captions. Learn about automatic audio extraction, chunking, and API uploads for seamless processing.

- Repository: [bradautomates/claude-video](https://github.com/bradautomates/claude-video)
- Tags: internals
- Published: 2026-07-19

---

**The Whisper fallback mechanism automatically transcribes video audio when native captions are missing by detecting empty caption results, extracting 16kHz mono MP3 audio, chunking files that exceed API limits, and uploading to Groq or OpenAI backends with automatic retry logic.**

The `claude-video` repository by **bradautomates** provides intelligent video processing skills that prioritize native captions but seamlessly activate the **Whisper fallback mechanism** when subtitles are unavailable. Understanding this cascading pipeline is essential for processing videos without embedded or auto-generated captions.

## The Two-Stage Transcript Acquisition Pipeline

The `/watch` skill implements a cascading strategy that attempts local caption retrieval before invoking external transcription APIs.

### Stage 1: Native Caption Retrieval via yt-dlp

First, [`watch.py`](https://github.com/bradautomates/claude-video/blob/main/watch.py) calls `fetch_captions()` from [`skills/watch/scripts/download.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/download.py), which executes **yt-dlp** with `--write-subs --write-auto-subs` flags to retrieve the best available VTT file ([download.py line 65-70](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/download.py#L65-L70)). If successful, `parse_vtt()` processes the subtitle file and sets `transcript_source = "captions"` ([watch.py line 101-105](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/watch.py#L101-L105)).

### Stage 2: Conditional Whisper Activation

When captions are unavailable, [`watch.py`](https://github.com/bradautomates/claude-video/blob/main/watch.py) validates four specific conditions before triggering the fallback ([watch.py line 39-44](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/watch.py#L39-L44)):

- No transcript segments exist (`not transcript_segments`)
- The user hasn't disabled Whisper (`not args.no_whisper`)
- The video contains audio tracks (`meta.get("has_audio")`)
- Valid API credentials are configured

Only when all conditions pass does the script invoke `transcribe_video()`.

## How the Whisper Fallback Mechanism Processes Audio

The `transcribe_video()` function in [`skills/watch/scripts/whisper.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/whisper.py) executes a complete audio processing pipeline when the fallback activates.

### Audio Extraction and Formatting

The system extracts a **mono 16kHz MP3** stream using `extract_audio()`, optimizing for speech recognition accuracy while minimizing upload bandwidth ([whisper.py line 15-39](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/whisper.py#L15-L39)).

### Chunked Upload for API Limits

If the audio file exceeds backend size constraints, `plan_chunks()` calculates segmentation points and `split_audio()` divides the stream into manageable chunks ([whisper.py line 40-58](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/whisper.py#L40-L58)).

### Backend Selection and Multipart Requests

The implementation prefers **Groq** for speed with **OpenAI** as the fallback provider. Each chunk uploads via handcrafted multipart HTTP requests ([whisper.py line 37-46](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/whisper.py#L37-L46)). After transcription, the system shifts timestamps to match the original video timeline and returns combined segments ([whisper.py line 71-81](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/whisper.py#L71-L81)).

## Error Handling and Edge Cases

The fallback includes robust error management for credential and network failures.

### Missing API Key Detection

When `load_api_key()` fails to locate credentials, [`watch.py`](https://github.com/bradautomates/claude-video/blob/main/watch.py) prints a diagnostic message directing users to run [`setup.py`](https://github.com/bradautomates/claude-video/blob/main/setup.py) for configuration ([watch.py line 55-64](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/watch.py#L55-L64)).

### Retry Logic and Network Failures

If uploads encounter **HTTP 429** or network errors, [`whisper.py`](https://github.com/bradautomates/claude-video/blob/main/whisper.py) implements automatic retries before raising `SystemExit`. The calling code in [`watch.py`](https://github.com/bradautomates/claude-video/blob/main/watch.py) catches this exception and logs "whisper fallback failed" ([watch.py line 52-53](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/watch.py#L52-L53)).

### Graceful Degradation Without Transcripts

When Whisper is unavailable or fails, the final report notes that only frames will be returned and advises enabling transcription for future runs ([watch.py line 77-84](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/watch.py#L77-L84)).

## Practical Usage Examples

### Enabling Automatic Fallback on Uncaptioned Videos

Run the skill on videos without native captions to trigger automatic Whisper processing:

```bash
watch https://example.com/video-without-captions.mp4

```

Typical output shows the transition:

```

[watch] checking metadata/captions via yt-dlp…
[watch] extracting audio for Whisper (groq)…
[watch] transcribed 312 segments via groq
- **Transcript:** 312 segments (via whisper (groq))

```

### Disabling the Fallback Mechanism

Force frames-only mode when transcription isn't needed using the `--no-whisper` flag:

```bash
watch https://example.com/video.mp4 --no-whisper

```

This produces:

```

[watch] no transcript available — proceed with frames only.
- **Transcript:** none available

```

### Specifying a Specific Backend

Override the default Groq preference with the `--whisper` argument:

```bash
watch https://example.com/video.mp4 --whisper openai

```

If `OPENAI_API_KEY` is configured, the transcript source reports as `whisper (openai)`.

## Summary

- The **Whisper fallback mechanism** activates only when yt-dlp returns no caption files and all prerequisites (audio present, API keys valid, user hasn't opted out) are satisfied.
- **Audio processing** converts video to 16kHz mono MP3, automatically chunking files that exceed API size constraints.
- **Backend flexibility** supports Groq as the primary provider with OpenAI as fallback, using handcrafted multipart uploads for compatibility.
- **Error resilience** includes credential validation hints, automatic retries for rate limits, and graceful degradation to frames-only output when transcription fails.

## Frequently Asked Questions

### What triggers the Whisper fallback mechanism?

The fallback triggers when `fetch_captions()` returns no VTT files from yt-dlp, resulting in zero transcript segments after parsing. According to the source code in [`watch.py`](https://github.com/bradautomates/claude-video/blob/main/watch.py), the system checks `not transcript_segments` combined with `not args.no_whisper`, valid API credentials loaded via `load_api_key()`, and audio presence (`meta.get("has_audio")`) before invoking `transcribe_video()`.

### Which Whisper backends does claude-video support?

The implementation supports **Groq** as the preferred backend for faster inference and **OpenAI** as the fallback option. Users select backends via the `--whisper` argument (accepting values `groq` or `openai`), with credentials loaded from environment variables or configuration files by `load_api_key()`.

### How does the system handle large video files?

Large audio files automatically split into chunks using `plan_chunks()` to calculate segmentation boundaries and `split_audio()` to perform the physical file splitting ([whisper.py line 40-58](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/whisper.py#L40-L58)). Each chunk uploads separately to the API, and the system shifts timestamps when stitching segments back together to maintain synchronization with the original video timeline.

### What happens if the Whisper API is unavailable?

When API calls fail after exhaustion of retries (network errors, HTTP 429 rate limits, or invalid keys), [`whisper.py`](https://github.com/bradautomates/claude-video/blob/main/whisper.py) raises `SystemExit`, which [`watch.py`](https://github.com/bradautomates/claude-video/blob/main/watch.py) catches to log "whisper fallback failed". The skill continues execution, returning only video frames with a diagnostic note explaining that transcripts are unavailable and suggesting the use of `--no-whisper` for future runs to skip the transcription attempt.