# Core Architecture of Claude-Video: Inside the Agent Skill Design

> Explore the core architecture of Claude-Video an Agent Skill using Python to convert video to transcripts and frames with yt-dlp ffmpeg and Whisper APIs.

- Repository: [bradautomates/claude-video](https://github.com/bradautomates/claude-video)
- Tags: architecture
- Published: 2026-08-09

---

**Claude-Video is a self-contained Agent Skill that orchestrates yt-dlp, ffmpeg, and optional Whisper APIs through pure-Python scripts to convert video URLs or local files into timestamped transcripts and curated JPEG frames.**

The **core architecture of Claude-Video** is implemented in the `bradautomates/claude-video` repository as a modular skill living under `skills/watch/`. Unlike monolithic video processing applications, this system operates as a lightweight, host-agnostic pipeline that any compatible agent environment can invoke via a declarative contract.

## Skill-Based Architecture Overview

Claude-Video follows the **Agent Skill pattern**, encapsulating all functionality within a single portable directory. The architecture centers on [`SKILL.md`](https://github.com/bradautomates/claude-video/blob/main/SKILL.md), which declares the `/watch` slash command and defines `SKILL_DIR` resolution for cross-platform compatibility.

All runtime components reside in `skills/watch/scripts/` and communicate through pure Python data structures—lists and dictionaries—eliminating external state dependencies beyond a temporary working directory and a local configuration file. This design ensures the skill functions identically across Claude Code, Codex, Cursor, and other compatible hosts.

## Component Breakdown and Execution Flow

The pipeline executes through seven coordinated Python modules, each handling a specific stage of video ingestion and analysis.

### Setup and Environment Configuration

[`scripts/setup.py`](https://github.com/bradautomates/claude-video/blob/main/scripts/setup.py) performs pre-flight validation on first invocation. It checks for `ffmpeg`, `ffprobe`, and `yt-dlp` binaries, auto-installing via Homebrew on macOS or providing exact installation commands for Linux/Windows. The script scaffolds `~/.config/watch/.env` (mode 0600) to store `GROQ_API_KEY` or `OPENAI_API_KEY` preferences and default `WATCH_DETAIL` settings.

### Entry Point and Orchestration

[`scripts/watch.py`](https://github.com/bradautomates/claude-video/blob/main/scripts/watch.py) serves as the main orchestrator. It parses CLI arguments to determine the **detail mode** (transcript, efficient, balanced, or token-burner), computes automatic frame rates via `auto_fps()` or `auto_fps_focus()`, and coordinates the downstream pipeline. The script manages temporary directory creation and final markdown report assembly.

### Download and Metadata Extraction

[`scripts/download.py`](https://github.com/bradautomates/claude-video/blob/main/scripts/download.py) wraps `yt-dlp` to fetch native subtitles when available and downloads video or audio streams based on the selected detail level. It returns critical metadata including `duration_seconds` and `subtitle_path`, enabling downstream components to make intelligent processing decisions.

### Frame Extraction Engine

[`scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/scripts/frames.py) implements the visual analysis logic through four distinct extraction strategies:

- **Efficient mode**: Extracts keyframes only (`ffmpeg -skip_frame nokey`), capped at 50 frames via `frame_cap('efficient')`
- **Balanced mode**: Runs scene-change detection with uniform sampling fallback, capped at 100 frames
- **Token-burner mode**: Scene-aware extraction without frame caps
- **Transcript mode**: Skips frame extraction unless explicit timestamps are supplied

The engine respects user-supplied `--timestamps` and `--start/--end` ranges through `extract_at_timestamps()`, and optionally deduplicates near-identical frames. Each returned frame includes `path`, `timestamp_seconds`, and `reason` metadata.

### Transcription Pipeline

[`scripts/transcribe.py`](https://github.com/bradautomates/claude-video/blob/main/scripts/transcribe.py) parses VTT subtitle files into structured segments containing `{timestamp, text}` dictionaries. When native captions are unavailable, [`scripts/whisper.py`](https://github.com/bradautomates/claude-video/blob/main/scripts/whisper.py) extracts mono 16kHz audio via ffmpeg and sends it to either **Groq** or **OpenAI** Whisper endpoints based on the API key present in `~/.config/watch/.env`. The module returns timestamped segments that merge seamlessly with existing subtitle data.

### Configuration Management

[`scripts/config.py`](https://github.com/bradautomates/claude-video/blob/main/scripts/config.py) centralizes user preferences through `get_config()` and exposes utility functions like `frame_cap(detail)`. It loads environment variables from the scoped `.env` file, ensuring API keys and default behaviors remain isolated from the host system.

## Detail Modes and Frame Processing Logic

The architecture adapts resource consumption through configurable detail levels:

**Transcript**: Audio-only extraction optimized for speech-to-text analysis. Downloads minimal data and skips frame generation unless specific timestamps are requested.

**Efficient**: Keyframe-only extraction using ffmpeg's nokey skip filter. Ideal for long videos where scene boundaries provide sufficient visual context, strictly limited to 50 frames.

**Balanced**: Scene-change detection combined with uniform sampling to capture both contextual shifts and temporal progression. The `frame_cap('balanced')` function limits output to 100 frames.

**Token-burner**: Maximum visual fidelity using scene detection without frame caps, suitable for short videos requiring comprehensive frame analysis.

## Cross-Platform and Host-Agnostic Design

Path resolution is **harness-agnostic**: [`SKILL.md`](https://github.com/bradautomates/claude-video/blob/main/SKILL.md) defines `SKILL_DIR` as the directory containing the markdown file itself, with all scripts referenced as `${SKILL_DIR}/scripts/<script>.py`. This eliminates host-specific environment variables and enables atomic skill migration between platforms.

The temporary working directory pattern ensures isolation—each invocation creates a unique path like `/tmp/watch-XYZ/` containing frames and metadata, with cleanup deferred to the caller after report generation.

## Practical Usage Examples

Run the skill from any compatible host using the SKILL_DIR resolution pattern:

```bash

# Basic usage with automatic detail selection

python3 "${SKILL_DIR}/scripts/watch.py" "https://youtu.be/abc123"

# Force keyframe-only extraction for long videos

python3 "${SKILL_DIR}/scripts/watch.py" "https://youtu.be/abc123" --detail efficient

# Analyze a specific segment at 2 frames per second

python3 "${SKILL_DIR}/scripts/watch.py" "https://youtu.be/abc123" \
  --start 1:30 --end 2:00 --fps 2

# Capture explicit timestamps mentioned in analysis

python3 "${SKILL_DIR}/scripts/watch.py" "https://youtu.be/abc123" \
  --detail balanced --timestamps 0:45,1:12,2:05

```

Initialize the environment on a fresh machine:

```bash
python3 "${SKILL_DIR}/scripts/setup.py"

```

Access extracted frames in agent sessions using the generated markdown report:

```markdown

## Frames

- `/tmp/watch-XYZ/frames/frame_001.jpg` (t=00:12, reason=scene)
- `/tmp/watch-XYZ/frames/frame_002.jpg` (t=00:24, reason=scene)

```

## Summary

- Claude-Video implements a **modular Agent Skill** architecture under `skills/watch/`, portable across multiple agent hosts via [`SKILL.md`](https://github.com/bradautomates/claude-video/blob/main/SKILL.md) contract definitions.
- The pipeline orchestrates **yt-dlp**, **ffmpeg**, and **Whisper APIs** through seven specialized Python modules with no external state dependencies.
- **Four detail modes** (transcript, efficient, balanced, token-burner) control the trade-off between computational cost and visual comprehensiveness.
- **Host-agnostic path resolution** using `SKILL_DIR` ensures consistent execution without environment-specific configuration.
- All components communicate via pure Python data structures, with configuration isolated to `~/.config/watch/.env`.

## Frequently Asked Questions

### What external dependencies does Claude-Video require?

Claude-Video requires **yt-dlp** for video downloading, **ffmpeg** and **ffprobe** for media processing, and optionally **Groq** or **OpenAI API keys** for Whisper transcription. The [`scripts/setup.py`](https://github.com/bradautomates/claude-video/blob/main/scripts/setup.py) module validates these dependencies on first run and provides platform-specific installation instructions if binaries are missing.

### How does Claude-Video handle videos without native subtitles?

When [`scripts/download.py`](https://github.com/bradautomates/claude-video/blob/main/scripts/download.py) fails to fetch native captions, the system falls back to [`scripts/whisper.py`](https://github.com/bradautomates/claude-video/blob/main/scripts/whisper.py), which extracts a mono 16kHz audio track via ffmpeg and sends it to the configured Whisper endpoint. The resulting timestamped segments integrate seamlessly with the transcript pipeline, ensuring comprehensive coverage regardless of source material caption availability.

### What is the difference between the 'efficient' and 'balanced' detail modes?

**Efficient** mode extracts only keyframes using `ffmpeg -skip_frame nokey`, capping output at 50 frames via `frame_cap('efficient')`—optimal for long videos where scene boundaries suffice. **Balanced** mode employs scene-change detection combined with uniform sampling, capped at 100 frames, providing both contextual transitions and temporal progression for medium-length content analysis.

### Is Claude-Video compatible with agent hosts other than Claude Code?

Yes. The architecture is explicitly **host-agnostic**; [`SKILL.md`](https://github.com/bradautomates/claude-video/blob/main/SKILL.md) defines standard `SKILL_DIR` resolution that works with Claude Code, Codex, Cursor, and any environment implementing the Agent Skill protocol. All scripts use relative paths based on `SKILL_DIR`, eliminating dependencies on host-specific environment variables or directory structures.