Claude Video Frame Extraction Modes Performance Comparison: Speed vs. Accuracy Trade-offs

Key-frame extraction is fastest but least precise, while scene-based extraction offers highest accuracy at the cost of full video decoding, with uniform extraction providing a balanced middle ground.

The bradautomates/claude-video repository implements multiple frame extraction strategies in skills/watch/scripts/frames.py that trade off processing speed against visual fidelity. Understanding the performance differences between Claude Video's frame extraction modes helps developers choose the optimal strategy for their specific use case, whether prioritizing rapid preview generation or precise scene boundary detection.

Understanding Claude Video's Four Frame Extraction Modes

Uniform Extraction (extract)

The extract function performs uniform extraction by invoking ffmpeg with a simple filter chain (-vf fps=…,scale=…). This mode decodes frames at a fixed fps rate using a single pass through the video file. It is most efficient when processing short clips or when using low frame rates (≤ 2 fps), as it only decodes the specific frames needed to match the target rate.

Scene-Based Extraction (extract_scene_or_uniform)

The extract_scene_or_uniform function implements the most computationally intensive approach. It first runs a full-decode pass using ffmpeg's scene detection filter (-vf select='eq(n\,0)+gt(scene,…)') to identify cut points by computing per-frame histogram differences. If the algorithm detects fewer than SCENE_MIN_FRAMES (8) scene cuts, it automatically falls back to uniform extraction, incurring the cost of both passes. This mode excels when videos contain many rapid cuts, as it isolates semantically significant frames.

Key-Frame Extraction (extract_keyframes)

The extract_keyframes function provides the fastest extraction by leveraging ffmpeg's -skip_frame nokey option to decode only I-frames (keyframes) while skipping P/B frames entirely. This lightweight approach works best when videos contain at least KEYFRAME_MIN (4) keyframes. The method approximates scene cuts with minimal CPU overhead, making it ideal for rapid video overviews.

Timestamp-Driven Extraction (extract_at_timestamps)

The extract_at_timestamps function targets specific temporal points by invoking ffmpeg with the -ss … seek parameter for each user-provided timestamp. This mode generates one ffmpeg invocation per timestamp, making it highly efficient when only a handful of explicit cue points (such as transcript timestamps) are required.

Performance Analysis: Decoding Workload and CPU Impact

The performance differences between Claude Video's frame extraction modes stem from four primary technical factors:

1. Decoding workload – Key-frame mode decodes only I-frames, uniform mode decodes frames matching the target fps, and scene mode decodes every frame once to evaluate the scene filter.
2. Filter complexity – The scene-selection filter forces ffmpeg to compute histogram differences for every frame, adding significant CPU overhead compared to the simple fps filter used in uniform extraction.
3. Post-processing overhead – Both scene and key-frame modes may invoke dedupe_perceptual and _even_sample functions, though these operations are negligible compared to the initial decode pass.
4. Fallback logic – Scene extraction incurs a double penalty when it falls back to uniform extraction after failing to find sufficient cuts (fewer than 8 frames).

Overall, the performance ranking from fastest to slowest is: key-frame extraction → uniform extraction → scene extraction.

Implementation Details in the Source Code

The frame extraction engines are centralized in skills/watch/scripts/frames.py, which contains the core implementations, FPS heuristics, and de-duplication logic. The skills/watch/scripts/watch.py module orchestrates the end-to-end /watch command, selecting the appropriate mode based on user options. Configuration defaults (including MAX_FPS and SCENE_THRESHOLD) reside in skills/watch/scripts/config.py, while tests/test_frames.py validates each extraction path and measures frame counts.

Practical Code Examples

from pathlib import Path
from skills.watch.scripts.frames import (
    extract,
    extract_keyframes,
    extract_scene_or_uniform,
)

video = Path("example.mp4")
out_dir = Path("frames/")

# 1️⃣ Uniform extraction (default 2 fps, 512 px)

uniform_frames = extract(video, out_dir, fps=2.0)

# 2️⃣ Key‑frame extraction (quickest)

key_frames, key_meta = extract_keyframes(video, out_dir, max_frames=50)

# 3️⃣ Scene‑based extraction (most accurate)

scene_frames, scene_meta = extract_scene_or_uniform(
    video,
    out_dir,
    fps=2.0,
    target_frames=100,
    max_frames=100,
    dedup=True,
)

Summary

• Key-frame extraction delivers the highest speed by decoding only I-frames via -skip_frame nokey, making it ideal for rapid previews when exact scene boundaries are not required.
• Uniform extraction provides a balanced middle ground, using a simple fps filter to extract evenly-spaced frames with moderate CPU usage.
• Scene-based extraction offers the highest accuracy by analyzing every frame for cuts, but requires full video decoding and histogram computation, making it the most computationally intensive.
• Timestamp-driven extraction is optimal for extracting specific frames at exact moments, using direct seeking to minimize unnecessary decoding.

Frequently Asked Questions

Which frame extraction mode is fastest in Claude Video?

Key-frame extraction is the fastest mode because it uses ffmpeg's -skip_frame nokey to decode only I-frames while skipping all P/B frames. This approach minimizes CPU usage and works best when the video contains at least 4 keyframes, providing a rapid overview without analyzing every frame.

When should I use scene-based extraction over uniform extraction?

Use scene-based extraction when you need frames that correspond to actual semantic boundaries or cuts in the video, such as for detailed analysis or chapter detection. While uniform extraction provides evenly-spaced frames regardless of content, scene extraction identifies true transition points by computing histogram differences for every frame.

What causes the scene extraction mode to fall back to uniform extraction?

The extract_scene_or_uniform function automatically falls back to uniform extraction when it detects fewer than SCENE_MIN_FRAMES (8) scene cuts in the video. This ensures the system returns a sufficient number of frames for analysis, though it incurs the computational cost of both the initial scene-detection pass and the subsequent uniform extraction pass.

How does timestamp-driven extraction handle videos with few I-frames?

Timestamp-driven extraction operates independently of keyframe distribution, using ffmpeg's -ss parameter to seek directly to specified timestamps. This mode is optimal when you need exact frames at specific temporal points (such as transcript cues), regardless of the video's GOP structure or keyframe density.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}