How Whisper Integration Works with Groq vs OpenAI Backends in Claude Video

Claude Video's "watch" skill uses a pure-stdlib Python script to transcribe video audio via either Groq's Whisper-large-v3 or OpenAI's Whisper-1 API, automatically selecting the backend based on available API keys with Groq taking precedence.

The bradautomates/claude-video repository provides a "watch" skill that converts video content into searchable text transcripts. At the heart of this capability lies whisper.py, a dependency-free Python module that interfaces with both Groq and OpenAI Whisper endpoints, giving users flexibility in choosing their transcription provider based on cost and availability.

Backend Selection and API Key Precedence

In skills/watch/scripts/whisper.py, the load_api_key() function implements a clear precedence logic for backend selection. The function first searches for GROQ_API_KEY in the environment; if absent, it falls back to OPENAI_API_KEY. This default behavior reflects the repository's cost optimization strategy, as Groq's whisper-large-v3 runs at a fraction of OpenAI's pricing.

Users can override this automatic detection by passing the --backend argument via CLI or the backend parameter via Python API. When explicitly set to "groq" or "openai", the function validates only that specific provider's key and returns a tuple (backend, api_key) for use throughout the transcription pipeline.

Audio Extraction and Chunking Strategy

Before API transmission, the extract_audio() function runs ffmpeg to convert input videos into mono 16 kHz MP3 files at approximately 64 kbps. This standardization ensures compatibility with both Groq and OpenAI upload requirements while minimizing bandwidth usage.

The script handles large files through a chunking mechanism governed by MAX_UPLOAD_BYTES (24 MiB). The plan_chunks() function calculates time-contiguous split points for files exceeding this limit, and split_audio() executes the actual segmentation. Each chunk is processed independently, with timestamps later adjusted to reflect the original video timeline.

Multipart Upload and Endpoint Routing

The _build_multipart() function constructs multipart/form-data payloads manually using only Python standard library modules, avoiding external dependencies like requests or vendor SDKs. This payload includes the audio file, model specification, and response format parameters.

The _post_whisper() function routes requests to backend-specific endpoints:

• Groq: Sends to https://api.groq.com/openai/v1/audio/transcriptions with model "whisper-large-v3" (defined as GROQ_ENDPOINT and GROQ_MODEL)
• OpenAI: Sends to https://api.openai.com/v1/audio/transcriptions with model "whisper-1" (defined as OPENAI_ENDPOINT and OPENAI_MODEL)

This function also implements retry logic for HTTP 429 rate limit responses and transient network failures, ensuring robust delivery regardless of backend choice.

Response Normalization and Orchestration

After receiving the API response, _segments_from_response() parses the verbose_json output and normalizes it into Claude Video's internal format: a list of dictionaries containing start, end, and text keys.

The transcribe_video() function orchestrates the complete workflow:

1. Detects or receives backend credentials via load_api_key()
2. Extracts audio and determines whether to upload the whole file or chunked pieces
3. Invokes _transcribe_file() for each segment, which calls _post_whisper()
4. Merges chunk-level results using shift_segments to align timestamps with the original timeline
5. Returns the final segment list and the identifier of the backend used

Practical Usage Examples

Automatic Backend Detection

When running from the command line without specifying a backend, the script prioritizes Groq if GROQ_API_KEY is present:

python -m skills.watch.scripts.whisper video.mp4 audio.mp3

Forcing a Specific Backend

To bypass automatic selection and use OpenAI explicitly:

python -m skills.watch.scripts.whisper video.mp4 audio.mp3 --backend openai

Python API Integration

The "watch" skill calls the transcription logic programmatically:

from skills.watch.scripts.whisper import transcribe_video
from pathlib import Path

# Automatic backend selection

segments, used_backend = transcribe_video(
    video_path="video.mp4",
    audio_out=Path("audio.mp3")
)

print(f"Transcribed with {used_backend}:")
for seg in segments:
    print(f"[{seg['start']:.2f}s → {seg['end']:.2f}s] {seg['text']}")

Explicit Backend Configuration

Override the automatic detection by providing specific credentials:

segments, backend = transcribe_video(
    video_path="video.mp4",
    audio_out=Path("audio.mp3"),
    backend="openai",
    api_key="sk-xxxxxx"
)

Summary

• Groq takes precedence: The load_api_key() function in skills/watch/scripts/whisper.py checks for GROQ_API_KEY before falling back to OPENAI_API_KEY, reflecting Groq's cost advantage for whisper-large-v3.
• Pure stdlib implementation: The integration avoids external HTTP libraries, manually constructing multipart requests and handling JSON parsing with built-in modules.
• Automatic chunking: Files exceeding 24 MiB are split into time-contiguous segments, transcribed separately, and reassembled with corrected timestamps.
• Endpoint abstraction: The _post_whisper() function seamlessly switches between Groq and OpenAI endpoints based on the selected backend, using appropriate model identifiers for each service.
• CLI and API flexibility: Users can auto-detect backends via environment variables or force specific providers via --backend flags or Python parameters.

Frequently Asked Questions

Which Whisper backend does Claude Video prefer by default?

Claude Video prefers Groq's Whisper-large-v3 over OpenAI's Whisper-1. According to the setup.py configuration script, Groq offers significantly lower pricing for the same model quality. The load_api_key() function implements this preference by checking for GROQ_API_KEY before considering OPENAI_API_KEY.

How does Claude Video handle audio files larger than the API upload limits?

The integration handles large files through automatic chunking. When extract_audio() produces a file exceeding MAX_UPLOAD_BYTES (24 MiB), the plan_chunks() function calculates optimal split points, and split_audio() divides the audio into time-contiguous segments. Each chunk is transcribed independently, and shift_segments adjusts the timestamps in the final merge to maintain synchronization with the original video timeline.

Can I force Claude Video to use a specific transcription backend?

Yes. While the default behavior selects Groq when available, you can force a specific backend using the --backend CLI argument (accepting "groq" or "openai") or by passing the backend parameter to transcribe_video() in Python. When specified, the system validates only the corresponding API key and routes requests exclusively to that provider's endpoint.

What audio format does the Whisper integration require?

The extract_audio() function in skills/watch/scripts/whisper.py automatically converts input videos to mono 16 kHz MP3 format at approximately 64 kbps using ffmpeg. This standardization ensures compatibility with both Groq and OpenAI Whisper APIs while keeping file sizes manageable for the 24 MiB upload limit.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}