# How Claude Video Handles Audio Extraction for Whisper Transcription

> Discover how Claude Video extracts audio for Whisper transcription with ffmpeg, handling large files and reconstructing accurate timelines for seamless analysis.

- Repository: [bradautomates/claude-video](https://github.com/bradautomates/claude-video)
- Tags: how-to-guide
- Published: 2026-07-28

---

**Claude Video uses ffmpeg to extract a lightweight mono 16kHz MP3 audio track from videos, automatically chunks files exceeding 24 MiB to respect Whisper API limits, and reconstructs accurate timelines by shifting segment offsets after transcription.**

The `bradautomates/claude-video` repository provides a `watch` skill that transforms video content into searchable text by leveraging OpenAI's Whisper API. The system handles everything from format optimization to large-file chunking within the [`skills/watch/scripts/whisper.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/whisper.py) module, ensuring reliable transcription regardless of video length.

## FFmpeg-Based Audio Extraction

The transcription pipeline begins in [`skills/watch/scripts/whisper.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/whisper.py) with the `extract_audio()` function. This utility invokes ffmpeg to strip the audio track from the source video and convert it to a **mono 16kHz MP3** encoded at approximately 64 kbps.

This specific configuration produces a compact file—roughly 480 kB per minute of audio—while maintaining the fidelity required for accurate speech recognition. The 16kHz sample rate aligns with Whisper's native expectations, eliminating unnecessary resampling overhead during API processing.

```python

# Conceptual usage within the transcription flow

from skills.watch.scripts.whisper import extract_audio
from pathlib import Path

video_path = Path("lecture.mp4")
audio_path = Path("lecture_audio.mp3")

# Extracts mono 16kHz MP3 optimized for Whisper

extract_audio(video_path, audio_path)

```

## Chunking Strategy for Large Files

Once extracted, the `transcribe_video()` function validates the audio file size against the `MAX_UPLOAD_BYTES` constant (24 MiB). If the audio exceeds this threshold, the system invokes `plan_chunks()` to calculate an optimal segmentation strategy, followed by `split_audio()` to divide the MP3 into smaller fragments.

Each chunk is processed independently through the `_post_whisper()` helper, which constructs a multipart/form-data payload using `_build_multipart()` and transmits it to the selected backend endpoint—either `GROQ_ENDPOINT` or `OPENAI_ENDPOINT`. This chunking mechanism ensures compliance with the 25 MB upload ceiling while maintaining transcription continuity across segment boundaries.

```python

# Simplified flow showing chunk handling

def transcribe_video(video_path: Path, audio_path: Path):
    extract_audio(video_path, audio_path)
    audio_bytes = audio_path.read_bytes()
    
    if len(audio_bytes) > MAX_UPLOAD_BYTES:  # 24 MiB limit

        chunks = plan_chunks(audio_path)  # Determine split points

        segments = []
        for chunk_file, offset in chunks:
            response = _post_whisper(chunk_file, backend="groq")
            chunk_segments = _segments_from_response(response)
            segments.extend(shift_segments(chunk_segments, offset))
        return segments
    else:
        return _post_whisper(audio_path, backend="openai")

```

## Timestamp Reconstruction and Alignment

Because Whisper processes each chunk independently, returned timestamps begin at zero for every segment. To restore chronological alignment with the original video, the pipeline employs `shift_segments()` within [`whisper.py`](https://github.com/bradautomates/claude-video/blob/main/whisper.py). This function offsets each transcription segment's start and end times by the chunk's original position in the audio timeline.

The result is a continuous, accurately timed transcript where the first word of the second chunk correctly aligns with the video timestamp where that chunk begins, rather than resetting to zero.

## Integration in the Watch Workflow

The extraction and transcription logic integrates seamlessly into the broader video processing workflow via [`skills/watch/scripts/watch.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/watch.py). This orchestration module imports `transcribe_video()` and executes it immediately after video download completion. The resulting segment list—containing precise text, start times, and end times—is then formatted for display or fed into downstream query-answering components.

```python

# From skills/watch/scripts/watch.py

from skills.watch.scripts.whisper import transcribe_video

def process_video(video_url: str):
    # ... download logic ...

    audio_file = Path("temp_audio.mp3")
    segments, backend_used = transcribe_video(downloaded_video, audio_file)
    return format_transcript(segments)

```

## Summary

- **Audio extraction** occurs via `extract_audio()` in [`whisper.py`](https://github.com/bradautomates/claude-video/blob/main/whisper.py), producing a mono 16kHz MP3 at ~64 kbps to minimize file size while preserving speech clarity.
- **Size validation** against `MAX_UPLOAD_BYTES` (24 MiB) triggers automatic chunking via `plan_chunks()` and `split_audio()` when videos exceed API upload limits.
- **API transmission** uses `_post_whisper()` with `_build_multipart()` to send audio to either Groq or OpenAI Whisper endpoints.
- **Timestamp alignment** is restored through `shift_segments()`, which offsets chunk-relative times to match the original video timeline.
- **Workflow integration** happens in [`watch.py`](https://github.com/bradautomates/claude-video/blob/main/watch.py), which orchestrates the entire pipeline from download to searchable transcript.

## Frequently Asked Questions

### What audio format does Claude Video use for Whisper transcription?

Claude Video extracts audio as a **mono MP3 file with a 16kHz sample rate** encoded at approximately 64 kbps. This format balances compression efficiency with the fidelity required for accurate speech recognition, resulting in files roughly 480 kB per minute.

### How does Claude Video handle videos longer than 25 MB?

When the extracted audio exceeds the `MAX_UPLOAD_BYTES` threshold of 24 MiB, the system automatically calculates a chunking plan using `plan_chunks()` and splits the audio into smaller segments with `split_audio()`. Each chunk is transcribed independently, and timestamps are reconstructed using `shift_segments()` to maintain chronological alignment.

### Why does Claude Video shift segment timestamps after transcription?

Whisper returns timestamps relative to the start of each audio chunk (always beginning at zero). Since Claude Video splits large files into multiple chunks, `shift_segments()` adds the original offset of each chunk to its respective transcription segments, ensuring the final transcript aligns accurately with the original video timeline.

### Which Whisper backends does Claude Video support?

The codebase supports both **Groq** and **OpenAI** Whisper endpoints. The `_post_whisper()` function in [`whisper.py`](https://github.com/bradautomates/claude-video/blob/main/whisper.py) constructs the appropriate multipart payload and routes it to the selected backend, allowing flexibility in API provider choice based on availability or pricing preferences.