# Whisper API Fallback for Unavailable Captions in Claude-Video: A Complete Implementation Guide

> Implement Whisper API fallback for unavailable captions in Claude-Video. This guide covers pure-Python integration with Groq & OpenAI for seamless transcription and retry logic without native captions.

- Repository: [bradautomates/claude-video](https://github.com/bradautomates/claude-video)
- Tags: how-to-guide
- Published: 2026-07-29

---

**The Claude-Video watch skill automatically falls back to Whisper API transcription when native captions are missing, using a pure-Python implementation that supports both Groq and OpenAI backends with intelligent chunking and retry logic.**

The `bradautomates/claude-video` repository provides a sophisticated video processing pipeline that ensures Claude always receives transcript data, even when source videos lack embedded subtitles. According to the source code, the **watch skill** ([`skills/watch/scripts/watch.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/watch.py)) implements a cascading fallback system that attempts native caption extraction first, then seamlessly transitions to cloud-based speech recognition via the Whisper API.

## How the Caption Fallback Pipeline Works

The transcription workflow follows a strict priority order to maximize efficiency and minimize API costs.

### Native Caption Detection

When processing a video, the pipeline first invokes [`scripts/transcribe.py`](https://github.com/bradautomates/claude-video/blob/main/scripts/transcribe.py) to locate WebVTT files produced by yt-dlp. If a non-empty VTT file exists, the skill uses these native captions directly and skips audio extraction entirely. This avoids unnecessary API calls and preserves the original timing metadata.

### Audio Extraction and Processing

If native captions are unavailable or the source is a local file without subtitles, [`watch.py`](https://github.com/bradautomates/claude-video/blob/main/watch.py) calls `transcribe_video()` from **[`skills/watch/scripts/whisper.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/whisper.py)**. This function handles the complete audio pipeline:

1. **`extract_audio()`** uses ffmpeg to generate a mono 16 kHz MP3 file (`audio_out.mp3`), producing approximately 480 kB per minute of video.
2. If the resulting audio exceeds the 24 MiB safety threshold, **`plan_chunks()`** calculates an even time-split that respects both size limits and API constraints.
3. **`split_audio()`** creates individual chunk files using ffmpeg with `-c copy` to avoid re-encoding, then processes each chunk individually.

### API Key Discovery and Provider Selection

The **`load_api_key()`** function in [`whisper.py`](https://github.com/bradautomates/claude-video/blob/main/whisper.py) implements a hierarchical search strategy:

- Environment variables (`GROQ_API_KEY` or `OPENAI_API_KEY`)
- Configuration file at `~/.config/watch/.env` (preferred for persistent storage)

**Groq is the default provider** due to superior speed and cost efficiency. OpenAI serves as an automatic fallback or can be forced via command-line flags.

## Chunking and Upload Strategy for Large Files

Videos longer than approximately 50 minutes trigger the chunking system to comply with Whisper API file size limits (25 MB).

The **`_post_whisper()`** function manages multipart form construction manually through **`_build_multipart()`**, maintaining zero external dependencies beyond the Python standard library. This implementation includes a custom `User-Agent` header specifically to avoid Cloudflare blocks on the Groq endpoint.

After transcription, **`shift_segments()`** adjusts the timestamps from each chunk by adding the appropriate time offset, ensuring the final transcript aligns with the original video timeline rather than restarting at zero for each segment.

## Retry Logic and Error Handling

The Whisper fallback implements robust error recovery with distinct strategies for different failure modes:

- **Rate limiting (HTTP 429):** Retries up to two times with exponential backoff, respecting the `Retry-After` header when present.
- **Transient network errors:** Up to three additional attempts with exponential backoff before failing gracefully.
- **Missing API keys:** The script prints a clear configuration hint and proceeds with frame-only output if `--no-whisper` is specified, setting `SETUP_COMPLETE=true` to suppress repeated warnings.

## Configuring the Whisper API Fallback

### Setting Up API Keys

Create a secure configuration file to persist your API credentials:

```bash
mkdir -p ~/.config/watch
cat > ~/.config/watch/.env <<'EOF'

# Groq (preferred - fastest and cheapest)

GROQ_API_KEY=your-groq-key-here

# OpenAI (fallback option)

# OPENAI_API_KEY=your-openai-key-here

EOF
chmod 600 ~/.config/watch/.env

```

### Provider Selection and Usage

Run the skill with automatic fallback to Whisper when captions are unavailable:

```bash
python3 "$SKILL_DIR/skills/watch/scripts/watch.py" "https://example.com/video-without-captions.mp4"

```

Force a specific backend using the `--whisper` flag:

```bash

# Use OpenAI instead of default Groq

python3 skills/watch/scripts/watch.py "$URL" --whisper openai

```

Disable Whisper entirely for frames-only processing:

```bash
python3 skills/watch/scripts/watch.py "$URL" --no-whisper

```

Process specific time ranges with Whisper fallback:

```bash
python3 skills/watch/scripts/watch.py "$URL" --start 2:45 --end 3:10

```

## Security and Privacy Considerations

The implementation minimizes data exposure by design. Only the extracted audio file (never the video or intermediate frames) transmits to external APIs. The `~/.config/watch/.env` file requires mode `0600` permissions, and the codebase never logs API keys. According to the source code in [`skills/watch/scripts/whisper.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/whisper.py), all video processing and frame extraction remain local to the machine.

## Summary

- The **watch skill** prioritizes native WebVTT captions and only invokes Whisper API when subtitles are missing or empty.
- **`transcribe_video()`** in [`skills/watch/scripts/whisper.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/whisper.py) manages the complete fallback pipeline using pure Python standard library calls.
- **Groq** provides the default backend with **OpenAI** as an automatic or explicit fallback, discovered via environment variables or `~/.config/watch/.env`.
- **Intelligent chunking** handles files up to 25 MB per segment, with `shift_segments()` reconstructing accurate timeline alignment.
- **Retry logic** handles rate limits and network errors with exponential backoff, while missing keys trigger helpful setup instructions rather than crashes.

## Frequently Asked Questions

### What happens if both Groq and OpenAI API keys are configured?

The system defaults to Groq for all transcriptions unless you explicitly specify `--whisper openai` on the command line. Groq is preferred due to faster processing speeds and lower costs, though both services provide equivalent accuracy for the 16 kHz mono MP3 input generated by the pipeline.

### How does the skill handle videos that exceed the Whisper API file size limits?

When `plan_chunks()` detects an audio file larger than 24 MiB, it calculates an optimal split strategy and `split_audio()` divides the MP3 into smaller segments using ffmpeg's stream copy feature. Each chunk uploads separately, and `shift_segments()` adjusts the returned timestamps by adding the chunk's start time offset to maintain synchronization with the original video timeline.

### Can I use the Whisper fallback with local video files?

Yes. The pipeline treats local files identically to downloaded content. If the local video lacks embedded subtitle tracks, [`watch.py`](https://github.com/bradautomates/claude-video/blob/main/watch.py) automatically invokes `transcribe_video()` from [`whisper.py`](https://github.com/bradautomates/claude-video/blob/main/whisper.py) to extract audio and generate a transcript, provided you have configured a valid API key in the environment or `~/.config/watch/.env`.

### Is my API key secure when using this fallback mechanism?

The implementation stores keys exclusively in `~/.config/watch/.env` with `0600` permissions and reads them via `load_api_key()` without logging or echoing them to stdout. The request construction in `_build_multipart()` and `_post_whisper()` transmits only the audio file content and necessary authentication headers; video data and extracted frames never leave your local machine.