# Frame Extraction Pipeline Architecture in claude-video: A 10-Stage Technical Breakdown

> Explore the 10-stage frame extraction pipeline architecture in claude-video. Learn about its modular design, detail engines, and frame budget management for efficient video processing.

- Repository: [bradautomates/claude-video](https://github.com/bradautomates/claude-video)
- Tags: architecture
- Published: 2026-08-07

---

**The frame extraction pipeline architecture in claude-video is a modular 10-stage system orchestrated by [`watch.py`](https://github.com/bradautomates/claude-video/blob/main/watch.py) and implemented in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py), supporting three detail engines—uniform, scene-aware, and keyframe—while managing frame budgets through metadata probing, adaptive FPS selection, perceptual deduplication, and cue-frame merging.**

The `bradautomates/claude-video` repository implements a sophisticated frame extraction pipeline architecture designed to convert video content into optimized image sequences for large language model consumption. Located primarily in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) and orchestrated by [`watch.py`](https://github.com/bradautomates/claude-video/blob/main/watch.py), this system balances computational efficiency with comprehensive visual coverage through budget-aware processing and three distinct extraction modes.

## Stage-by-Stage Pipeline Architecture

### 1. Video Metadata Probing

The pipeline begins with `get_metadata()` in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) at line 86, which invokes `ffprobe` to extract duration, resolution, codec information, and audio presence. This data informs all subsequent budget calculations and extraction parameters.

### 2. FPS and Frame-Budget Selection

Two helper functions determine the extraction rate: `auto_fps()` (line 22) for full-clip scans and `auto_fps_focus()` (line 41) for user-specified time ranges. These respect the global `MAX_FPS` cap of approximately 2 fps while scaling dynamically with video length to maintain predictable token usage.

### 3. Uniform Extraction (Fallback Mode)

The `extract()` function at line 70 performs simple `ffmpeg` passes that capture frames at the computed FPS, scaling each to the requested resolution. This serves as the baseline extraction method and fallback when advanced detection yields insufficient results.

### 4. Scene-Change Detection

For the `balanced` and `token-burner` detail engines, `extract_scene_candidates()` (line 176) runs `ffmpeg` with a `select='gt(scene,THRESH)'` filter to capture the first frame plus every detected scene cut. Results are optionally limited by the `max_frames` parameter to respect budget constraints.

### 5. Key-Frame Extraction (Efficient Mode)

The `efficient` detail engine invokes `extract_keyframes()` at line 76, which decodes only I-frames using `-skip_frame nokey`. This method leverages natural video compression boundaries to identify significant visual changes without expensive full-frame analysis.

### 6. Near-Duplicate Deduplication

The `dedupe_perceptual()` function (line 63) down-scales each JPEG to a 16×16 grayscale thumbnail and greedily drops frames whose mean per-pixel delta falls below `DEDUP_THRESHOLD`. This eliminates redundant visual information before final sampling.

### 7. Even-Sampling to Budget

After deduplication, `_even_sample()` (line 93) selects *n* evenly-spaced frames while preserving the first and last frames to honor the user- or engine-imposed frame cap. This ensures consistent coverage regardless of video length.

### 8. Timestamp-Cue Extraction

When users specify transcript-highlighted moments via `--timestamps`, `extract_at_timestamps()` (line 124) uses `ffmpeg` to seek to each absolute timestamp and extract single frames (`cue_*.jpg`). Over-budget cues are evenly trimmed to maintain constraints.

### 9. Cue and Detail Frame Merging

The `merge_frames()` function (line 112) guarantees that cue frames are never dropped, merging them chronologically with the engine's output to preserve critical narrative moments alongside automated selections.

### 10. Orchestration and Engine Selection

[`watch.py`](https://github.com/bradautomates/claude-video/blob/main/watch.py) (lines 99-127) serves as the orchestration layer, invoking `extract_scene_or_uniform()` (line 110) to select the appropriate engine based on the `--detail` flag, presence of cues, and focus windows. It handles automatic fallback to uniform extraction when scene or keyframe data proves insufficient.

## Execution Flow and Implementation Details

The pipeline executes through a deterministic sequence controlled by [`watch.py`](https://github.com/bradautomates/claude-video/blob/main/watch.py). First, metadata probing establishes the video's characteristics. Budget calculators then determine the target frame count. If timestamp cues are provided, these extract immediately via `extract_at_timestamps()`.

The detail engine selection follows the `--detail` flag:

- **efficient**: Routes to `extract_keyframes()` for I-frame-only decoding
- **balanced** / **token-burner**: Routes to `extract_scene_or_uniform()`, which may invoke `extract_scene_candidates()` followed by `dedupe_perceptual()` and `_even_sample()`
- **fallback**: Defaults to `extract()` when scene detection fails

Unless `--no-dedup` is specified, deduplication runs automatically. Finally, `merge_frames()` combines cue frames with detail frames before [`watch.py`](https://github.com/bradautomates/claude-video/blob/main/watch.py) generates the markdown summary report.

## Configuration and Budget Management

Frame budgets are enforced through [`config.py`](https://github.com/bradautomates/claude-video/blob/main/config.py), which defines default caps per detail level via `frame_cap()`. The system maintains a global `MAX_FPS` ceiling of approximately 2 fps to prevent token overconsumption, while `auto_fps_focus()` enables higher temporal resolution within user-specified time windows without exceeding total frame quotas.

## Code Examples

### Extracting Uniform Frames

To extract a uniform 2 fps sample from a local file:

```bash
python -m skills.watch.scripts.frames extract \
    path/to/video.mp4 ./out \
    --fps 2 \
    --resolution 720 \
    --max-frames 50

```

This executes the `extract()` function in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) at line 70.

### Running the Full Pipeline

To use the high-level orchestration with scene detection and timestamp cues:

```bash
python -m skills.watch.scripts.watch \
    "https://www.youtube.com/watch?v=abc123" \
    --detail balanced \
    --resolution 640 \
    --timestamps "00:30,01:45,02:10"

```

This triggers the complete pipeline: metadata extraction, FPS calculation, cue extraction, scene-aware frame detection, deduplication, merging, and markdown reporting as implemented in [`watch.py`](https://github.com/bradautomates/claude-video/blob/main/watch.py).

## Summary

- The **frame extraction pipeline architecture** centers on [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py), providing 10 discrete stages from metadata probing to frame merging.
- Three **detail engines**—efficient (keyframes), balanced (scene-aware), and uniform—offer trade-offs between computational cost and visual comprehensiveness.
- **Perceptual deduplication** via `dedupe_perceptual()` eliminates redundant frames using 16×16 grayscale thumbnails before final sampling.
- **Budget-aware extraction** through `auto_fps()` and `auto_fps_focus()` ensures frame counts remain within LLM token limits regardless of video duration.
- **Cue frame prioritization** guarantees that user-specified timestamps are preserved and merged with algorithmically selected frames.

## Frequently Asked Questions

### What is the difference between the three detail engines in claude-video?

The **efficient** engine uses `extract_keyframes()` to decode only I-frames, minimizing processing overhead by leveraging existing video compression keyframes. The **balanced** and **token-burner** engines use `extract_scene_candidates()` to detect scene changes via ffmpeg's scene filter, with token-burner allowing higher frame caps. If scene detection yields insufficient frames, both fall back to uniform extraction via `extract()`.

### How does the deduplication algorithm prevent redundant frames?

The `dedupe_perceptual()` function in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) resamples each extracted JPEG to a 16×16 grayscale thumbnail and calculates the mean per-pixel delta between consecutive frames. Frames with deltas below the `DEDUP_THRESHOLD` are discarded greedily, ensuring only perceptually distinct images proceed to the final sampling stage.

### Can I extract frames at specific timestamps instead of using automated detection?

Yes. The `extract_at_timestamps()` function (line 124) accepts absolute timestamps and uses ffmpeg to seek and extract single frames at those positions. These **cue frames** are prioritized during the merge stage via `merge_frames()`, ensuring they appear in the final output even when the automated engine would otherwise skip those time positions.

### What happens when scene detection fails to find enough frames?

When `extract_scene_or_uniform()` detects insufficient scene changes, it automatically falls back to `extract()` for uniform frame sampling. This fallback mechanism, orchestrated in [`watch.py`](https://github.com/bradautomates/claude-video/blob/main/watch.py), guarantees that the pipeline always delivers the requested number of frames (up to the budget cap) regardless of the video's visual complexity or scene density.