Core Architecture of Claude-Video: Inside the Agent Skill Design

Claude-Video is a self-contained Agent Skill that orchestrates yt-dlp, ffmpeg, and optional Whisper APIs through pure-Python scripts to convert video URLs or local files into timestamped transcripts and curated JPEG frames.

The core architecture of Claude-Video is implemented in the bradautomates/claude-video repository as a modular skill living under skills/watch/. Unlike monolithic video processing applications, this system operates as a lightweight, host-agnostic pipeline that any compatible agent environment can invoke via a declarative contract.

Skill-Based Architecture Overview

Claude-Video follows the Agent Skill pattern, encapsulating all functionality within a single portable directory. The architecture centers on SKILL.md, which declares the /watch slash command and defines SKILL_DIR resolution for cross-platform compatibility.

All runtime components reside in skills/watch/scripts/ and communicate through pure Python data structures—lists and dictionaries—eliminating external state dependencies beyond a temporary working directory and a local configuration file. This design ensures the skill functions identically across Claude Code, Codex, Cursor, and other compatible hosts.

Component Breakdown and Execution Flow

The pipeline executes through seven coordinated Python modules, each handling a specific stage of video ingestion and analysis.

Setup and Environment Configuration

scripts/setup.py performs pre-flight validation on first invocation. It checks for ffmpeg, ffprobe, and yt-dlp binaries, auto-installing via Homebrew on macOS or providing exact installation commands for Linux/Windows. The script scaffolds ~/.config/watch/.env (mode 0600) to store GROQ_API_KEY or OPENAI_API_KEY preferences and default WATCH_DETAIL settings.

Entry Point and Orchestration

scripts/watch.py serves as the main orchestrator. It parses CLI arguments to determine the detail mode (transcript, efficient, balanced, or token-burner), computes automatic frame rates via auto_fps() or auto_fps_focus(), and coordinates the downstream pipeline. The script manages temporary directory creation and final markdown report assembly.

Download and Metadata Extraction

scripts/download.py wraps yt-dlp to fetch native subtitles when available and downloads video or audio streams based on the selected detail level. It returns critical metadata including duration_seconds and subtitle_path, enabling downstream components to make intelligent processing decisions.

Frame Extraction Engine

scripts/frames.py implements the visual analysis logic through four distinct extraction strategies:

• Efficient mode: Extracts keyframes only (ffmpeg -skip_frame nokey), capped at 50 frames via frame_cap('efficient')
• Balanced mode: Runs scene-change detection with uniform sampling fallback, capped at 100 frames
• Token-burner mode: Scene-aware extraction without frame caps
• Transcript mode: Skips frame extraction unless explicit timestamps are supplied

The engine respects user-supplied --timestamps and --start/--end ranges through extract_at_timestamps(), and optionally deduplicates near-identical frames. Each returned frame includes path, timestamp_seconds, and reason metadata.

Transcription Pipeline

scripts/transcribe.py parses VTT subtitle files into structured segments containing {timestamp, text} dictionaries. When native captions are unavailable, scripts/whisper.py extracts mono 16kHz audio via ffmpeg and sends it to either Groq or OpenAI Whisper endpoints based on the API key present in ~/.config/watch/.env. The module returns timestamped segments that merge seamlessly with existing subtitle data.

Configuration Management

scripts/config.py centralizes user preferences through get_config() and exposes utility functions like frame_cap(detail). It loads environment variables from the scoped .env file, ensuring API keys and default behaviors remain isolated from the host system.

Detail Modes and Frame Processing Logic

The architecture adapts resource consumption through configurable detail levels:

Transcript: Audio-only extraction optimized for speech-to-text analysis. Downloads minimal data and skips frame generation unless specific timestamps are requested.

Efficient: Keyframe-only extraction using ffmpeg's nokey skip filter. Ideal for long videos where scene boundaries provide sufficient visual context, strictly limited to 50 frames.

Balanced: Scene-change detection combined with uniform sampling to capture both contextual shifts and temporal progression. The frame_cap('balanced') function limits output to 100 frames.

Token-burner: Maximum visual fidelity using scene detection without frame caps, suitable for short videos requiring comprehensive frame analysis.

Cross-Platform and Host-Agnostic Design

Path resolution is harness-agnostic: SKILL.md defines SKILL_DIR as the directory containing the markdown file itself, with all scripts referenced as ${SKILL_DIR}/scripts/<script>.py. This eliminates host-specific environment variables and enables atomic skill migration between platforms.

The temporary working directory pattern ensures isolation—each invocation creates a unique path like /tmp/watch-XYZ/ containing frames and metadata, with cleanup deferred to the caller after report generation.

Practical Usage Examples

Run the skill from any compatible host using the SKILL_DIR resolution pattern:

# Basic usage with automatic detail selection

python3 "${SKILL_DIR}/scripts/watch.py" "https://youtu.be/abc123"

# Force keyframe-only extraction for long videos

python3 "${SKILL_DIR}/scripts/watch.py" "https://youtu.be/abc123" --detail efficient

# Analyze a specific segment at 2 frames per second

python3 "${SKILL_DIR}/scripts/watch.py" "https://youtu.be/abc123" \
  --start 1:30 --end 2:00 --fps 2

# Capture explicit timestamps mentioned in analysis

python3 "${SKILL_DIR}/scripts/watch.py" "https://youtu.be/abc123" \
  --detail balanced --timestamps 0:45,1:12,2:05

Initialize the environment on a fresh machine:

python3 "${SKILL_DIR}/scripts/setup.py"

Access extracted frames in agent sessions using the generated markdown report:

## Frames

- `/tmp/watch-XYZ/frames/frame_001.jpg` (t=00:12, reason=scene)
- `/tmp/watch-XYZ/frames/frame_002.jpg` (t=00:24, reason=scene)

Summary

• Claude-Video implements a modular Agent Skill architecture under skills/watch/, portable across multiple agent hosts via SKILL.md contract definitions.
• The pipeline orchestrates yt-dlp, ffmpeg, and Whisper APIs through seven specialized Python modules with no external state dependencies.
• Four detail modes (transcript, efficient, balanced, token-burner) control the trade-off between computational cost and visual comprehensiveness.
• Host-agnostic path resolution using SKILL_DIR ensures consistent execution without environment-specific configuration.
• All components communicate via pure Python data structures, with configuration isolated to ~/.config/watch/.env.

Frequently Asked Questions

What external dependencies does Claude-Video require?

Claude-Video requires yt-dlp for video downloading, ffmpeg and ffprobe for media processing, and optionally Groq or OpenAI API keys for Whisper transcription. The scripts/setup.py module validates these dependencies on first run and provides platform-specific installation instructions if binaries are missing.

How does Claude-Video handle videos without native subtitles?

When scripts/download.py fails to fetch native captions, the system falls back to scripts/whisper.py, which extracts a mono 16kHz audio track via ffmpeg and sends it to the configured Whisper endpoint. The resulting timestamped segments integrate seamlessly with the transcript pipeline, ensuring comprehensive coverage regardless of source material caption availability.

What is the difference between the 'efficient' and 'balanced' detail modes?

Efficient mode extracts only keyframes using ffmpeg -skip_frame nokey, capping output at 50 frames via frame_cap('efficient')—optimal for long videos where scene boundaries suffice. Balanced mode employs scene-change detection combined with uniform sampling, capped at 100 frames, providing both contextual transitions and temporal progression for medium-length content analysis.

Is Claude-Video compatible with agent hosts other than Claude Code?

Yes. The architecture is explicitly host-agnostic; SKILL.md defines standard SKILL_DIR resolution that works with Claude Code, Codex, Cursor, and any environment implementing the Agent Skill protocol. All scripts use relative paths based on SKILL_DIR, eliminating dependencies on host-specific environment variables or directory structures.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}