# How Claude-Video's Video Processing Pipeline Works: From URL to Markdown Report

> Explore Claude-Video's video processing pipeline. Learn how it transforms video URLs into markdown reports using Python scripts for download, frame extraction, and transcription.

- Repository: [bradautomates/claude-video](https://github.com/bradautomates/claude-video)
- Tags: internals
- Published: 2026-08-09

---

**Claude-Video's video processing pipeline converts any video URL or local file into a markdown report by orchestrating download, frame extraction, and transcription through a sequence of pure-Python scripts culminating in [`watch.py`](https://github.com/bradautomates/claude-video/blob/main/watch.py).**

The `bradautomates/claude-video` repository implements a self-contained "watch" skill that transforms video content into structured markdown documentation. This video processing pipeline handles everything from YouTube downloads to AI-powered transcription, using a budget-conscious approach to frame sampling that adapts to video duration while maintaining predictable token usage.

## Input Acquisition and Media Handling

The pipeline begins in [`skills/watch/scripts/watch.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/watch.py), which delegates source detection and retrieval to [`download.py`](https://github.com/bradautomates/claude-video/blob/main/download.py). When a user provides input, the system first calls `is_url()` to determine whether to process a remote resource or resolve a local path via `resolve_local()`.

For remote URLs, the pipeline optimizes for efficiency by invoking `fetch_captions()` before downloading any video data. This retrieves metadata and available VTT subtitles—prioritizing manual captions over auto-generated ones—to potentially satisfy the request without fetching the full media file. If the video itself is required (either because captions are insufficient or the user specified `--timestamps`), `download_url()` leverages `yt-dlp` to retrieve the file, returning the video path, subtitle path, and metadata dictionary.

## Metadata Analysis and Frame Budgeting

Once the media is local, [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) takes over for technical analysis. The `get_metadata()` function invokes `ffprobe` to extract duration, resolution, codec information, and audio presence, which drives subsequent budgeting decisions.

The pipeline supports selective range processing through `parse_time()` and `format_time()`, which clamp user-supplied `--start` and `--end` arguments against the actual video duration. Based on the effective duration, the system calculates frame density using either `auto_fps()` for full-clip analysis or `auto_fps_focus()` for zoomed-in segments. These functions determine the target frames-per-second and total frame count while respecting the user-defined `--max-frames` cap, ensuring short clips receive dense coverage while long videos remain token-efficient.

## Intelligent Frame Extraction

Frame extraction operates through two complementary pathways. If the user provides `--timestamps`, `extract_at_timestamps()` captures high-quality frames at specific instants independent of the main extraction engine. These "cue frames" are stored separately as `cue_*.jpg` files.

The primary extraction flow uses the `--detail` flag to select one of three engines implemented in [`frames.py`](https://github.com/bradautomates/claude-video/blob/main/frames.py):

- **efficient** → `extract_keyframes()` retrieves only keyframe data for maximum speed (~50 frames)
- **balanced / token-burner** → `extract_scene_or_uniform()` performs scene-change detection with fallback to uniform sampling

Both engines utilize `ffmpeg` for extraction, apply `dedupe_perceptual()` to remove near-identical frames, and use `_even_sample()` to down-sample results to the calculated frame budget. Finally, `merge_frames()` combines cue frames with engine-extracted frames and re-indexes them chronologically for the final report.

## Transcript Processing and Fallback

Transcript generation begins with VTT subtitles obtained during the download phase. The [`skills/watch/scripts/transcribe.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/transcribe.py) module processes these through `filter_range()` to match the user's time window and `format_transcript()` to produce clean markdown.

When subtitles are unavailable and `--no-whisper` is not set, the pipeline falls back to [`skills/watch/scripts/whisper.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/whisper.py). This pure-stdlib implementation first calls `load_api_key()` to select between Groq or OpenAI backends. The `extract_audio()` function creates a mono 16kHz MP3, which `plan_chunks()` segments if larger than 24MiB. Each chunk is uploaded via `_post_whisper()`, with responses parsed through `_segments_from_response()` and timeline-adjusted using `shift_segments()` to maintain synchronization with the source video.

## Markdown Report Assembly

In the final stage, [`watch.py`](https://github.com/bradautomates/claude-video/blob/main/watch.py) aggregates all components into a structured markdown report. The output includes source information, video metadata, frame statistics with absolute paths to the `frames/` directory, and the formatted transcript. If transcription fails, the report includes a warning while still presenting available visual data. All intermediate files reside in a temporary working directory (or the user-specified `--out-dir`) for post-processing inspection.

## Practical Usage Examples

```bash

# Basic usage – download a YouTube video and get a balanced set of frames + transcript

watch https://www.youtube.com/watch?v=abcd1234

```

```bash

# Focus on a 30-second segment, extract up to 200 frames, and force the Groq Whisper backend

watch https://example.com/video.mp4 \
      --start 00:02:00 --end 00:02:30 \
      --max-frames 200 \
      --whisper groq

```

```bash

# Get only keyframe-based frames (fast, ~50 frames) without any transcript

watch local_video.mkv --detail efficient --no-whisper

```

```bash

# Pin exact moments via timestamps while still extracting scene frames

watch https://youtu.be/xyz \
      --timestamps 00:01:15,00:03:45,150 \
      --detail balanced

```

## Summary

- The video processing pipeline is orchestrated by [`skills/watch/scripts/watch.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/watch.py), which coordinates five specialized modules through pure-stdlib Python.
- Input handling in [`download.py`](https://github.com/bradautomates/claude-video/blob/main/download.py) intelligently pre-fetches captions before downloading video, optimizing for both speed and token efficiency.
- Frame extraction in [`frames.py`](https://github.com/bradautomates/claude-video/blob/main/frames.py) implements a budget-by-duration policy using `auto_fps()` and `auto_fps_focus()`, with three detail modes ranging from keyframe-only to scene-change detection.
- Transcription supports both VTT subtitles and Whisper API fallbacks via [`transcribe.py`](https://github.com/bradautomates/claude-video/blob/main/transcribe.py) and [`whisper.py`](https://github.com/bradautomates/claude-video/blob/main/whisper.py), with automatic chunking for large audio files.
- All external tools (`yt-dlp`, `ffmpeg`, `ffprobe`) are invoked via subprocesses, requiring no local GPU resources for the core pipeline.

## Frequently Asked Questions

### What video sources does Claude-Video support?

Claude-Video accepts any URL supported by `yt-dlp` (including YouTube, Vimeo, and direct MP4 links) as well as local file paths. The `is_url()` function in [`download.py`](https://github.com/bradautomates/claude-video/blob/main/download.py) automatically detects the source type and routes it to either `download_url()` for remote resources or `resolve_local()` for filesystem access.

### How does the frame budget calculation prevent token overflow?

The pipeline implements adaptive sampling through `auto_fps()` and `auto_fps_focus()` in [`frames.py`](https://github.com/bradautomates/claude-video/blob/main/frames.py), which calculate target frame counts based on video duration while strictly enforcing the `--max-frames` cap. Short clips receive dense coverage (high FPS), while long videos are down-sampled via `_even_sample()` to maintain predictable token usage regardless of input length.

### What is the difference between the efficient and balanced detail engines?

The **efficient** engine calls `extract_keyframes()` to retrieve only video keyframes, producing approximately 50 frames ideal for fast processing. The **balanced** and **token-burner** engines use `extract_scene_or_uniform()`, which attempts scene-change detection first and falls back to uniform sampling if no scene changes are detected, providing more comprehensive visual coverage at higher token cost.

### Does Claude-Video require local GPU resources for transcription?

No. The pipeline relies on external API services for transcription. When subtitles are unavailable, [`whisper.py`](https://github.com/bradautomates/claude-video/blob/main/whisper.py) uploads audio chunks to either Groq or OpenAI APIs using standard HTTP requests, making the transcription process entirely cloud-based without requiring local Whisper model inference.