How watch.py Orchestrates the Download, Frame Extraction, and Transcript Pipeline in claude-video

watch.py serves as the main entry point for the /watch skill, coordinating a 13-stage pipeline that transforms video URLs or local files into markdown reports containing extracted frames and searchable transcripts.

The watch.py script in the claude-video repository acts as the central orchestrator for the video processing workflow. It wires together specialized helper modules to handle everything from downloading content via yt-dlp to extracting frames based on scene changes and generating transcripts. Understanding how watch.py coordinates these operations reveals the architecture behind the /watch slash command used across Claude Code, Codex, and Cursor environments.

CLI Argument Parsing and Configuration Loading

The pipeline begins in skills/watch/scripts/watch.py with the main() function handling CLI argument parsing via argparse. This captures options for source location, resolution, detail level, timestamps, time ranges, and Whisper backend selection.

Immediately after parsing, watch.py loads user preferences through get_config and frame_cap from skills/watch/scripts/config.py. These functions resolve the --detail preset (efficient, balanced, or token-burner) and compute a frame budget (max_frames) that governs all subsequent extraction decisions. This budget mechanism ensures the tool respects token limits and processing constraints before any heavy operations begin.

Source Resolution and Caption Detection

Before downloading, watch.py determines whether the source is remote or local using is_url from skills/watch/scripts/download.py. For URL-based sources, the orchestrator attempts an optimization: fetch_captions pulls metadata and VTT subtitles via yt-dlp without downloading the actual video file.

If subtitles exist, parse_vtt from skills/watch/scripts/transcribe.py converts them into structured transcript_segments. This caption-only check allows the pipeline to bypass video download entirely when the user only needs transcript data, significantly reducing processing time and bandwidth.

Video Download and Metadata Extraction

When frame extraction is required or captions are unavailable, watch.py triggers download or download_url from skills/watch/scripts/download.py. This stage retrieves either the full video or audio-only streams depending on the detail mode requirements, returning a video_path and optional subtitle file.

Following successful download, the orchestrator calls get_metadata from skills/watch/scripts/frames.py, which invokes ffprobe to extract technical specifications including duration, resolution, codec information, and audio track presence. This metadata feeds into the frame budgeting system and validates time-range constraints.

Frame Budgeting and Time-Range Calculation

With video metadata available, watch.py processes temporal parameters. parse_time converts human-readable --start and --end arguments into seconds, while auto_fps or auto_fps_focus from skills/watch/scripts/frames.py calculates an appropriate frames-per-second rate that respects the max_frames budget across the effective duration.

This dynamic calculation ensures that whether analyzing a 30-second clip or a two-hour lecture, the frame extraction density adjusts to stay within configurable limits while maximizing coverage.

The Frame Extraction Pipeline

The orchestration sequences frame extraction in two distinct phases:

Cue-Frame Extraction

If the user specified --timestamps, watch.py reserves budget for cue frames using extract_at_timestamps from skills/watch/scripts/frames.py. This extracts single frames at precise transcript-cued moments (e.g., when the speaker says "look at this diagram"). These frames are reserved against the frame cap before the main extraction engine runs.

Detail-Engine Processing

Depending on the --detail preset, watch.py delegates to specific extraction engines in skills/watch/scripts/frames.py:

• Efficient mode: Uses extract_keyframes to grab only codec keyframes, minimizing processing.
• Balanced or Token-burner modes: Uses extract_scene_or_uniform, which attempts scene-change detection first, falling back to uniform sampling if scene detection fails or exhausts the budget.

Both engines respect the remaining detail_budget calculated earlier, ensuring the total frame count stays within limits.

Frame Merging and Deduplication

After extraction, merge_frames from skills/watch/scripts/frames.py combines cue frames with detail-engine frames, preserving chronological order and removing temporal duplicates. Unless --no-dedup is specified, watch.py runs deduplication logic to eliminate visually similar frames, which is particularly useful for slide-heavy presentations where minimal visual change occurs between frames.

Transcript Generation and Whisper Fallback

For transcripts, watch.py implements a cascading strategy. If fetch_captions succeeded earlier, those parsed segments flow directly to the report. Otherwise, the orchestrator checks if Whisper is disabled; if not, it proceeds to audio transcription.

The orchestrator calls load_api_key and transcribe_video from skills/watch/scripts/whisper.py. These functions select between Groq and OpenAI backends based on availability and user preference, then execute the Whisper API against the downloaded audio. This fallback mechanism ensures transcript availability even for videos without native captions.

Report Generation and Cleanup

In the final stage, watch.py assembles the markdown report using format_transcript and format_time helpers. The report includes source information, duration, resolution, frame statistics with extraction reasons (scene-change, uniform, transcript-cue), and the formatted transcript.

All temporary files reside in a working directory generated via tempfile.mkdtemp, which watch.py cleans up after report generation, leaving only the final markdown output containing embedded frame references and transcript data.

Practical Usage Examples

# Basic usage – download a YouTube video, extract balanced frames, and show captions

watch https://www.youtube.com/watch?v=xyz123

# Focus on a 30-second segment with higher resolution and custom fps

watch https://youtu.be/xyz123 --start 01:23 --end 01:53 \
      --resolution 1024 --fps 1.5 --detail balanced

# Extract precise frames at transcript-cued timestamps (e.g., "look here" moments)

watch https://youtu.be/xyz123 --timestamps 00:45,02:10,03:05

# Force Whisper fallback (Groq or OpenAI) when no subtitles are available

watch ./local_video.mp4 --whisper groq

# Disable near-duplicate removal (useful for slide decks)

watch ./presentation.mp4 --no-dedup

Key Architectural Components

File	Role
skills/watch/scripts/watch.py	Main orchestration script parsing CLI arguments and coordinating the full pipeline
skills/watch/scripts/download.py	URL detection, caption fetching, and video/audio download via yt-dlp
skills/watch/scripts/frames.py	Metadata probing, fps calculation, frame extraction, and deduplication logic
skills/watch/scripts/transcribe.py	VTT subtitle parsing and transcript formatting
skills/watch/scripts/whisper.py	API key management and Whisper backend selection
skills/watch/scripts/config.py	User-configurable defaults and detail presets

Summary

• watch.py acts as the linear orchestrator for the entire /watch skill, managing a 13-stage pipeline from argument parsing to final markdown output.
• The script prioritizes efficiency by attempting caption-only fetches before downloading full video files, falling back to yt-dlp downloads only when necessary.
• Frame extraction follows a strict budget system using max_frames and dynamic fps calculation, with separate handling for cue frames and detail-engine frames.
• All heavy lifting is delegated to specialized modules: download.py for acquisition, frames.py for visual processing, and whisper.py for audio transcription.
• The architecture remains host-agnostic, functioning uniformly across Claude Code, Codex, Cursor, and other environments without host-specific dependencies.

Frequently Asked Questions

How does watch.py decide whether to download the full video or just captions?

watch.py calls is_url to identify remote sources, then immediately attempts fetch_captions from skills/watch/scripts/download.py. If the video source provides VTT subtitles and the user only requested transcript data (or the detail mode allows it), the pipeline skips the download phase entirely. Full video download occurs only when frame extraction is required or when no captions exist and Whisper transcription is needed.

What happens if I specify timestamps that exceed the frame budget?

watch.py reserves the cue-frame budget first using extract_at_timestamps before running the detail engine. If timestamp requests exceed max_frames, the system prioritizes the explicitly requested cue frames and reduces sampling in the detail-engine phase. The auto_fps and auto_fps_focus functions recalculate sampling rates to accommodate both cue frames and scene-extraction within the remaining budget.

Can watch.py process local video files, or only URLs?

Yes, watch.py handles both sources. The is_url function from skills/watch/scripts/download.py returns false for local filesystem paths, causing the orchestrator to skip the download phase and proceed directly to metadata extraction via get_metadata in skills/watch/scripts/frames.py. Local processing supports all the same frame extraction and transcription features as URL-based workflows.

How does the deduplication logic work, and when should I disable it?

By default, watch.py runs deduplication after merge_frames to remove visually similar frames that fall within short temporal windows. This prevents waste when extracting from slide presentations or static scenes. Use the --no-dedup flag when you need to preserve every extracted frame, such as when analyzing subtle animation changes or when precise frame counts matter more than token efficiency.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}