# Claude Video Frame Extraction Modes Performance Comparison: Speed vs. Accuracy Trade-offs

> Compare Claude Video frame extraction modes. Discover speed vs accuracy trade-offs for key-frame, scene-based, and uniform extraction to optimize your video analysis.

- Repository: [bradautomates/claude-video](https://github.com/bradautomates/claude-video)
- Tags: performance
- Published: 2026-07-13

---

**Key-frame extraction is fastest but least precise, while scene-based extraction offers highest accuracy at the cost of full video decoding, with uniform extraction providing a balanced middle ground.**

The `bradautomates/claude-video` repository implements multiple frame extraction strategies in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) that trade off processing speed against visual fidelity. Understanding the performance differences between Claude Video's frame extraction modes helps developers choose the optimal strategy for their specific use case, whether prioritizing rapid preview generation or precise scene boundary detection.

## Understanding Claude Video's Four Frame Extraction Modes

### Uniform Extraction (`extract`)

The `extract` function performs **uniform extraction** by invoking ffmpeg with a simple filter chain (`-vf fps=…,scale=…`). This mode decodes frames at a fixed `fps` rate using a single pass through the video file. It is most efficient when processing short clips or when using low frame rates (≤ 2 fps), as it only decodes the specific frames needed to match the target rate.

### Scene-Based Extraction (`extract_scene_or_uniform`)

The `extract_scene_or_uniform` function implements the most computationally intensive approach. It first runs a full-decode pass using ffmpeg's scene detection filter (`-vf select='eq(n\,0)+gt(scene,…)'`) to identify cut points by computing per-frame histogram differences. If the algorithm detects fewer than `SCENE_MIN_FRAMES` (8) scene cuts, it automatically falls back to uniform extraction, incurring the cost of both passes. This mode excels when videos contain many rapid cuts, as it isolates semantically significant frames.

### Key-Frame Extraction (`extract_keyframes`)

The `extract_keyframes` function provides the fastest extraction by leveraging ffmpeg's `-skip_frame nokey` option to decode only I-frames (keyframes) while skipping P/B frames entirely. This lightweight approach works best when videos contain at least `KEYFRAME_MIN` (4) keyframes. The method approximates scene cuts with minimal CPU overhead, making it ideal for rapid video overviews.

### Timestamp-Driven Extraction (`extract_at_timestamps`)

The `extract_at_timestamps` function targets specific temporal points by invoking ffmpeg with the `-ss …` seek parameter for each user-provided timestamp. This mode generates one ffmpeg invocation per timestamp, making it highly efficient when only a handful of explicit cue points (such as transcript timestamps) are required.

## Performance Analysis: Decoding Workload and CPU Impact

The performance differences between Claude Video's frame extraction modes stem from four primary technical factors:

1. **Decoding workload** – Key-frame mode decodes only I-frames, uniform mode decodes frames matching the target fps, and scene mode decodes every frame once to evaluate the scene filter.
2. **Filter complexity** – The scene-selection filter forces ffmpeg to compute histogram differences for every frame, adding significant CPU overhead compared to the simple `fps` filter used in uniform extraction.
3. **Post-processing overhead** – Both scene and key-frame modes may invoke `dedupe_perceptual` and `_even_sample` functions, though these operations are negligible compared to the initial decode pass.
4. **Fallback logic** – Scene extraction incurs a double penalty when it falls back to uniform extraction after failing to find sufficient cuts (fewer than 8 frames).

Overall, the performance ranking from fastest to slowest is: **key-frame extraction → uniform extraction → scene extraction**.

## Implementation Details in the Source Code

The frame extraction engines are centralized in **[`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py)**, which contains the core implementations, FPS heuristics, and de-duplication logic. The **[`skills/watch/scripts/watch.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/watch.py)** module orchestrates the end-to-end `/watch` command, selecting the appropriate mode based on user options. Configuration defaults (including `MAX_FPS` and `SCENE_THRESHOLD`) reside in **[`skills/watch/scripts/config.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/config.py)**, while **[`tests/test_frames.py`](https://github.com/bradautomates/claude-video/blob/main/tests/test_frames.py)** validates each extraction path and measures frame counts.

## Practical Code Examples

```python
from pathlib import Path
from skills.watch.scripts.frames import (
    extract,
    extract_keyframes,
    extract_scene_or_uniform,
)

video = Path("example.mp4")
out_dir = Path("frames/")

# 1️⃣ Uniform extraction (default 2 fps, 512 px)

uniform_frames = extract(video, out_dir, fps=2.0)

# 2️⃣ Key‑frame extraction (quickest)

key_frames, key_meta = extract_keyframes(video, out_dir, max_frames=50)

# 3️⃣ Scene‑based extraction (most accurate)

scene_frames, scene_meta = extract_scene_or_uniform(
    video,
    out_dir,
    fps=2.0,
    target_frames=100,
    max_frames=100,
    dedup=True,
)

```

## Summary

- **Key-frame extraction** delivers the highest speed by decoding only I-frames via `-skip_frame nokey`, making it ideal for rapid previews when exact scene boundaries are not required.
- **Uniform extraction** provides a balanced middle ground, using a simple `fps` filter to extract evenly-spaced frames with moderate CPU usage.
- **Scene-based extraction** offers the highest accuracy by analyzing every frame for cuts, but requires full video decoding and histogram computation, making it the most computationally intensive.
- **Timestamp-driven extraction** is optimal for extracting specific frames at exact moments, using direct seeking to minimize unnecessary decoding.

## Frequently Asked Questions

### Which frame extraction mode is fastest in Claude Video?

**Key-frame extraction** is the fastest mode because it uses ffmpeg's `-skip_frame nokey` to decode only I-frames while skipping all P/B frames. This approach minimizes CPU usage and works best when the video contains at least 4 keyframes, providing a rapid overview without analyzing every frame.

### When should I use scene-based extraction over uniform extraction?

Use **scene-based extraction** when you need frames that correspond to actual semantic boundaries or cuts in the video, such as for detailed analysis or chapter detection. While uniform extraction provides evenly-spaced frames regardless of content, scene extraction identifies true transition points by computing histogram differences for every frame.

### What causes the scene extraction mode to fall back to uniform extraction?

The `extract_scene_or_uniform` function automatically falls back to uniform extraction when it detects fewer than `SCENE_MIN_FRAMES` (8) scene cuts in the video. This ensures the system returns a sufficient number of frames for analysis, though it incurs the computational cost of both the initial scene-detection pass and the subsequent uniform extraction pass.

### How does timestamp-driven extraction handle videos with few I-frames?

**Timestamp-driven extraction** operates independently of keyframe distribution, using ffmpeg's `-ss` parameter to seek directly to specified timestamps. This mode is optimal when you need exact frames at specific temporal points (such as transcript cues), regardless of the video's GOP structure or keyframe density.