How Claude-Video's Video Processing Pipeline Works: From URL to Markdown Report

Claude-Video's video processing pipeline converts any video URL or local file into a markdown report by orchestrating download, frame extraction, and transcription through a sequence of pure-Python scripts culminating in watch.py.

The bradautomates/claude-video repository implements a self-contained "watch" skill that transforms video content into structured markdown documentation. This video processing pipeline handles everything from YouTube downloads to AI-powered transcription, using a budget-conscious approach to frame sampling that adapts to video duration while maintaining predictable token usage.

Input Acquisition and Media Handling

The pipeline begins in skills/watch/scripts/watch.py, which delegates source detection and retrieval to download.py. When a user provides input, the system first calls is_url() to determine whether to process a remote resource or resolve a local path via resolve_local().

For remote URLs, the pipeline optimizes for efficiency by invoking fetch_captions() before downloading any video data. This retrieves metadata and available VTT subtitles—prioritizing manual captions over auto-generated ones—to potentially satisfy the request without fetching the full media file. If the video itself is required (either because captions are insufficient or the user specified --timestamps), download_url() leverages yt-dlp to retrieve the file, returning the video path, subtitle path, and metadata dictionary.

Metadata Analysis and Frame Budgeting

Once the media is local, skills/watch/scripts/frames.py takes over for technical analysis. The get_metadata() function invokes ffprobe to extract duration, resolution, codec information, and audio presence, which drives subsequent budgeting decisions.

The pipeline supports selective range processing through parse_time() and format_time(), which clamp user-supplied --start and --end arguments against the actual video duration. Based on the effective duration, the system calculates frame density using either auto_fps() for full-clip analysis or auto_fps_focus() for zoomed-in segments. These functions determine the target frames-per-second and total frame count while respecting the user-defined --max-frames cap, ensuring short clips receive dense coverage while long videos remain token-efficient.

Intelligent Frame Extraction

Frame extraction operates through two complementary pathways. If the user provides --timestamps, extract_at_timestamps() captures high-quality frames at specific instants independent of the main extraction engine. These "cue frames" are stored separately as cue_*.jpg files.

The primary extraction flow uses the --detail flag to select one of three engines implemented in frames.py:

• efficient → extract_keyframes() retrieves only keyframe data for maximum speed (~50 frames)
• balanced / token-burner → extract_scene_or_uniform() performs scene-change detection with fallback to uniform sampling

Both engines utilize ffmpeg for extraction, apply dedupe_perceptual() to remove near-identical frames, and use _even_sample() to down-sample results to the calculated frame budget. Finally, merge_frames() combines cue frames with engine-extracted frames and re-indexes them chronologically for the final report.

Transcript Processing and Fallback

Transcript generation begins with VTT subtitles obtained during the download phase. The skills/watch/scripts/transcribe.py module processes these through filter_range() to match the user's time window and format_transcript() to produce clean markdown.

When subtitles are unavailable and --no-whisper is not set, the pipeline falls back to skills/watch/scripts/whisper.py. This pure-stdlib implementation first calls load_api_key() to select between Groq or OpenAI backends. The extract_audio() function creates a mono 16kHz MP3, which plan_chunks() segments if larger than 24MiB. Each chunk is uploaded via _post_whisper(), with responses parsed through _segments_from_response() and timeline-adjusted using shift_segments() to maintain synchronization with the source video.

Markdown Report Assembly

In the final stage, watch.py aggregates all components into a structured markdown report. The output includes source information, video metadata, frame statistics with absolute paths to the frames/ directory, and the formatted transcript. If transcription fails, the report includes a warning while still presenting available visual data. All intermediate files reside in a temporary working directory (or the user-specified --out-dir) for post-processing inspection.

Practical Usage Examples

# Basic usage – download a YouTube video and get a balanced set of frames + transcript

watch https://www.youtube.com/watch?v=abcd1234

# Focus on a 30-second segment, extract up to 200 frames, and force the Groq Whisper backend

watch https://example.com/video.mp4 \
      --start 00:02:00 --end 00:02:30 \
      --max-frames 200 \
      --whisper groq

# Get only keyframe-based frames (fast, ~50 frames) without any transcript

watch local_video.mkv --detail efficient --no-whisper

# Pin exact moments via timestamps while still extracting scene frames

watch https://youtu.be/xyz \
      --timestamps 00:01:15,00:03:45,150 \
      --detail balanced

Summary

• The video processing pipeline is orchestrated by skills/watch/scripts/watch.py, which coordinates five specialized modules through pure-stdlib Python.
• Input handling in download.py intelligently pre-fetches captions before downloading video, optimizing for both speed and token efficiency.
• Frame extraction in frames.py implements a budget-by-duration policy using auto_fps() and auto_fps_focus(), with three detail modes ranging from keyframe-only to scene-change detection.
• Transcription supports both VTT subtitles and Whisper API fallbacks via transcribe.py and whisper.py, with automatic chunking for large audio files.
• All external tools (yt-dlp, ffmpeg, ffprobe) are invoked via subprocesses, requiring no local GPU resources for the core pipeline.

Frequently Asked Questions

What video sources does Claude-Video support?

Claude-Video accepts any URL supported by yt-dlp (including YouTube, Vimeo, and direct MP4 links) as well as local file paths. The is_url() function in download.py automatically detects the source type and routes it to either download_url() for remote resources or resolve_local() for filesystem access.

How does the frame budget calculation prevent token overflow?

The pipeline implements adaptive sampling through auto_fps() and auto_fps_focus() in frames.py, which calculate target frame counts based on video duration while strictly enforcing the --max-frames cap. Short clips receive dense coverage (high FPS), while long videos are down-sampled via _even_sample() to maintain predictable token usage regardless of input length.

What is the difference between the efficient and balanced detail engines?

The efficient engine calls extract_keyframes() to retrieve only video keyframes, producing approximately 50 frames ideal for fast processing. The balanced and token-burner engines use extract_scene_or_uniform(), which attempts scene-change detection first and falls back to uniform sampling if no scene changes are detected, providing more comprehensive visual coverage at higher token cost.

Does Claude-Video require local GPU resources for transcription?

No. The pipeline relies on external API services for transcription. When subtitles are unavailable, whisper.py uploads audio chunks to either Groq or OpenAI APIs using standard HTTP requests, making the transcription process entirely cloud-based without requiring local Whisper model inference.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}