# How Whisper Integration Works with Groq vs OpenAI Backends in Claude Video

> Learn how Claude Video integrates Whisper for audio transcription, comparing Groq vs OpenAI backends. Discover automatic backend selection for efficient video processing.

- Repository: [bradautomates/claude-video](https://github.com/bradautomates/claude-video)
- Tags: deep-dive
- Published: 2026-08-01

---

**Claude Video's "watch" skill uses a pure-stdlib Python script to transcribe video audio via either Groq's Whisper-large-v3 or OpenAI's Whisper-1 API, automatically selecting the backend based on available API keys with Groq taking precedence.**

The `bradautomates/claude-video` repository provides a "watch" skill that converts video content into searchable text transcripts. At the heart of this capability lies [`whisper.py`](https://github.com/bradautomates/claude-video/blob/main/whisper.py), a dependency-free Python module that interfaces with both Groq and OpenAI Whisper endpoints, giving users flexibility in choosing their transcription provider based on cost and availability.

## Backend Selection and API Key Precedence

In [`skills/watch/scripts/whisper.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/whisper.py), the `load_api_key()` function implements a clear precedence logic for backend selection. The function first searches for `GROQ_API_KEY` in the environment; if absent, it falls back to `OPENAI_API_KEY`. This default behavior reflects the repository's cost optimization strategy, as Groq's whisper-large-v3 runs at a fraction of OpenAI's pricing.

Users can override this automatic detection by passing the `--backend` argument via CLI or the `backend` parameter via Python API. When explicitly set to `"groq"` or `"openai"`, the function validates only that specific provider's key and returns a tuple `(backend, api_key)` for use throughout the transcription pipeline.

## Audio Extraction and Chunking Strategy

Before API transmission, the `extract_audio()` function runs `ffmpeg` to convert input videos into mono 16 kHz MP3 files at approximately 64 kbps. This standardization ensures compatibility with both Groq and OpenAI upload requirements while minimizing bandwidth usage.

The script handles large files through a chunking mechanism governed by `MAX_UPLOAD_BYTES` (24 MiB). The `plan_chunks()` function calculates time-contiguous split points for files exceeding this limit, and `split_audio()` executes the actual segmentation. Each chunk is processed independently, with timestamps later adjusted to reflect the original video timeline.

## Multipart Upload and Endpoint Routing

The `_build_multipart()` function constructs `multipart/form-data` payloads manually using only Python standard library modules, avoiding external dependencies like `requests` or vendor SDKs. This payload includes the audio file, model specification, and response format parameters.

The `_post_whisper()` function routes requests to backend-specific endpoints:

- **Groq**: Sends to `https://api.groq.com/openai/v1/audio/transcriptions` with model `"whisper-large-v3"` (defined as `GROQ_ENDPOINT` and `GROQ_MODEL`)
- **OpenAI**: Sends to `https://api.openai.com/v1/audio/transcriptions` with model `"whisper-1"` (defined as `OPENAI_ENDPOINT` and `OPENAI_MODEL`)

This function also implements retry logic for HTTP 429 rate limit responses and transient network failures, ensuring robust delivery regardless of backend choice.

## Response Normalization and Orchestration

After receiving the API response, `_segments_from_response()` parses the `verbose_json` output and normalizes it into Claude Video's internal format: a list of dictionaries containing `start`, `end`, and `text` keys.

The `transcribe_video()` function orchestrates the complete workflow:

1. Detects or receives backend credentials via `load_api_key()`
2. Extracts audio and determines whether to upload the whole file or chunked pieces
3. Invokes `_transcribe_file()` for each segment, which calls `_post_whisper()`
4. Merges chunk-level results using `shift_segments` to align timestamps with the original timeline
5. Returns the final segment list and the identifier of the backend used

## Practical Usage Examples

### Automatic Backend Detection

When running from the command line without specifying a backend, the script prioritizes Groq if `GROQ_API_KEY` is present:

```bash
python -m skills.watch.scripts.whisper video.mp4 audio.mp3

```

### Forcing a Specific Backend

To bypass automatic selection and use OpenAI explicitly:

```bash
python -m skills.watch.scripts.whisper video.mp4 audio.mp3 --backend openai

```

### Python API Integration

The "watch" skill calls the transcription logic programmatically:

```python
from skills.watch.scripts.whisper import transcribe_video
from pathlib import Path

# Automatic backend selection

segments, used_backend = transcribe_video(
    video_path="video.mp4",
    audio_out=Path("audio.mp3")
)

print(f"Transcribed with {used_backend}:")
for seg in segments:
    print(f"[{seg['start']:.2f}s → {seg['end']:.2f}s] {seg['text']}")

```

### Explicit Backend Configuration

Override the automatic detection by providing specific credentials:

```python
segments, backend = transcribe_video(
    video_path="video.mp4",
    audio_out=Path("audio.mp3"),
    backend="openai",
    api_key="sk-xxxxxx"
)

```

## Summary

- **Groq takes precedence**: The `load_api_key()` function in [`skills/watch/scripts/whisper.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/whisper.py) checks for `GROQ_API_KEY` before falling back to `OPENAI_API_KEY`, reflecting Groq's cost advantage for whisper-large-v3.
- **Pure stdlib implementation**: The integration avoids external HTTP libraries, manually constructing multipart requests and handling JSON parsing with built-in modules.
- **Automatic chunking**: Files exceeding 24 MiB are split into time-contiguous segments, transcribed separately, and reassembled with corrected timestamps.
- **Endpoint abstraction**: The `_post_whisper()` function seamlessly switches between Groq and OpenAI endpoints based on the selected backend, using appropriate model identifiers for each service.
- **CLI and API flexibility**: Users can auto-detect backends via environment variables or force specific providers via `--backend` flags or Python parameters.

## Frequently Asked Questions

### Which Whisper backend does Claude Video prefer by default?

Claude Video prefers Groq's Whisper-large-v3 over OpenAI's Whisper-1. According to the [`setup.py`](https://github.com/bradautomates/claude-video/blob/main/setup.py) configuration script, Groq offers significantly lower pricing for the same model quality. The `load_api_key()` function implements this preference by checking for `GROQ_API_KEY` before considering `OPENAI_API_KEY`.

### How does Claude Video handle audio files larger than the API upload limits?

The integration handles large files through automatic chunking. When `extract_audio()` produces a file exceeding `MAX_UPLOAD_BYTES` (24 MiB), the `plan_chunks()` function calculates optimal split points, and `split_audio()` divides the audio into time-contiguous segments. Each chunk is transcribed independently, and `shift_segments` adjusts the timestamps in the final merge to maintain synchronization with the original video timeline.

### Can I force Claude Video to use a specific transcription backend?

Yes. While the default behavior selects Groq when available, you can force a specific backend using the `--backend` CLI argument (accepting "groq" or "openai") or by passing the `backend` parameter to `transcribe_video()` in Python. When specified, the system validates only the corresponding API key and routes requests exclusively to that provider's endpoint.

### What audio format does the Whisper integration require?

The `extract_audio()` function in [`skills/watch/scripts/whisper.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/whisper.py) automatically converts input videos to mono 16 kHz MP3 format at approximately 64 kbps using `ffmpeg`. This standardization ensures compatibility with both Groq and OpenAI Whisper APIs while keeping file sizes manageable for the 24 MiB upload limit.