Frame Extraction Pipeline Architecture in claude-video: A 10-Stage Technical Breakdown

The frame extraction pipeline architecture in claude-video is a modular 10-stage system orchestrated by watch.py and implemented in skills/watch/scripts/frames.py, supporting three detail engines—uniform, scene-aware, and keyframe—while managing frame budgets through metadata probing, adaptive FPS selection, perceptual deduplication, and cue-frame merging.

The bradautomates/claude-video repository implements a sophisticated frame extraction pipeline architecture designed to convert video content into optimized image sequences for large language model consumption. Located primarily in skills/watch/scripts/frames.py and orchestrated by watch.py, this system balances computational efficiency with comprehensive visual coverage through budget-aware processing and three distinct extraction modes.

Stage-by-Stage Pipeline Architecture

1. Video Metadata Probing

The pipeline begins with get_metadata() in skills/watch/scripts/frames.py at line 86, which invokes ffprobe to extract duration, resolution, codec information, and audio presence. This data informs all subsequent budget calculations and extraction parameters.

2. FPS and Frame-Budget Selection

Two helper functions determine the extraction rate: auto_fps() (line 22) for full-clip scans and auto_fps_focus() (line 41) for user-specified time ranges. These respect the global MAX_FPS cap of approximately 2 fps while scaling dynamically with video length to maintain predictable token usage.

3. Uniform Extraction (Fallback Mode)

The extract() function at line 70 performs simple ffmpeg passes that capture frames at the computed FPS, scaling each to the requested resolution. This serves as the baseline extraction method and fallback when advanced detection yields insufficient results.

4. Scene-Change Detection

For the balanced and token-burner detail engines, extract_scene_candidates() (line 176) runs ffmpeg with a select='gt(scene,THRESH)' filter to capture the first frame plus every detected scene cut. Results are optionally limited by the max_frames parameter to respect budget constraints.

5. Key-Frame Extraction (Efficient Mode)

The efficient detail engine invokes extract_keyframes() at line 76, which decodes only I-frames using -skip_frame nokey. This method leverages natural video compression boundaries to identify significant visual changes without expensive full-frame analysis.

6. Near-Duplicate Deduplication

The dedupe_perceptual() function (line 63) down-scales each JPEG to a 16×16 grayscale thumbnail and greedily drops frames whose mean per-pixel delta falls below DEDUP_THRESHOLD. This eliminates redundant visual information before final sampling.

7. Even-Sampling to Budget

After deduplication, _even_sample() (line 93) selects n evenly-spaced frames while preserving the first and last frames to honor the user- or engine-imposed frame cap. This ensures consistent coverage regardless of video length.

8. Timestamp-Cue Extraction

When users specify transcript-highlighted moments via --timestamps, extract_at_timestamps() (line 124) uses ffmpeg to seek to each absolute timestamp and extract single frames (cue_*.jpg). Over-budget cues are evenly trimmed to maintain constraints.

9. Cue and Detail Frame Merging

The merge_frames() function (line 112) guarantees that cue frames are never dropped, merging them chronologically with the engine's output to preserve critical narrative moments alongside automated selections.

10. Orchestration and Engine Selection

watch.py (lines 99-127) serves as the orchestration layer, invoking extract_scene_or_uniform() (line 110) to select the appropriate engine based on the --detail flag, presence of cues, and focus windows. It handles automatic fallback to uniform extraction when scene or keyframe data proves insufficient.

Execution Flow and Implementation Details

The pipeline executes through a deterministic sequence controlled by watch.py. First, metadata probing establishes the video's characteristics. Budget calculators then determine the target frame count. If timestamp cues are provided, these extract immediately via extract_at_timestamps().

The detail engine selection follows the --detail flag:

• efficient: Routes to extract_keyframes() for I-frame-only decoding
• balanced / token-burner: Routes to extract_scene_or_uniform(), which may invoke extract_scene_candidates() followed by dedupe_perceptual() and _even_sample()
• fallback: Defaults to extract() when scene detection fails

Unless --no-dedup is specified, deduplication runs automatically. Finally, merge_frames() combines cue frames with detail frames before watch.py generates the markdown summary report.

Configuration and Budget Management

Frame budgets are enforced through config.py, which defines default caps per detail level via frame_cap(). The system maintains a global MAX_FPS ceiling of approximately 2 fps to prevent token overconsumption, while auto_fps_focus() enables higher temporal resolution within user-specified time windows without exceeding total frame quotas.

Code Examples

Extracting Uniform Frames

To extract a uniform 2 fps sample from a local file:

python -m skills.watch.scripts.frames extract \
    path/to/video.mp4 ./out \
    --fps 2 \
    --resolution 720 \
    --max-frames 50

This executes the extract() function in skills/watch/scripts/frames.py at line 70.

Running the Full Pipeline

To use the high-level orchestration with scene detection and timestamp cues:

python -m skills.watch.scripts.watch \
    "https://www.youtube.com/watch?v=abc123" \
    --detail balanced \
    --resolution 640 \
    --timestamps "00:30,01:45,02:10"

This triggers the complete pipeline: metadata extraction, FPS calculation, cue extraction, scene-aware frame detection, deduplication, merging, and markdown reporting as implemented in watch.py.

Summary

• The frame extraction pipeline architecture centers on skills/watch/scripts/frames.py, providing 10 discrete stages from metadata probing to frame merging.
• Three detail engines—efficient (keyframes), balanced (scene-aware), and uniform—offer trade-offs between computational cost and visual comprehensiveness.
• Perceptual deduplication via dedupe_perceptual() eliminates redundant frames using 16×16 grayscale thumbnails before final sampling.
• Budget-aware extraction through auto_fps() and auto_fps_focus() ensures frame counts remain within LLM token limits regardless of video duration.
• Cue frame prioritization guarantees that user-specified timestamps are preserved and merged with algorithmically selected frames.

Frequently Asked Questions

What is the difference between the three detail engines in claude-video?

The efficient engine uses extract_keyframes() to decode only I-frames, minimizing processing overhead by leveraging existing video compression keyframes. The balanced and token-burner engines use extract_scene_candidates() to detect scene changes via ffmpeg's scene filter, with token-burner allowing higher frame caps. If scene detection yields insufficient frames, both fall back to uniform extraction via extract().

How does the deduplication algorithm prevent redundant frames?

The dedupe_perceptual() function in skills/watch/scripts/frames.py resamples each extracted JPEG to a 16×16 grayscale thumbnail and calculates the mean per-pixel delta between consecutive frames. Frames with deltas below the DEDUP_THRESHOLD are discarded greedily, ensuring only perceptually distinct images proceed to the final sampling stage.

Can I extract frames at specific timestamps instead of using automated detection?

Yes. The extract_at_timestamps() function (line 124) accepts absolute timestamps and uses ffmpeg to seek and extract single frames at those positions. These cue frames are prioritized during the merge stage via merge_frames(), ensuring they appear in the final output even when the automated engine would otherwise skip those time positions.

What happens when scene detection fails to find enough frames?

When extract_scene_or_uniform() detects insufficient scene changes, it automatically falls back to extract() for uniform frame sampling. This fallback mechanism, orchestrated in watch.py, guarantees that the pipeline always delivers the requested number of frames (up to the budget cap) regardless of the video's visual complexity or scene density.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}