# Keyframe Extraction vs Scene-Detection in Claude-Video: Technical Differences and Usage Guide

> Discover the technical differences between keyframe extraction and scene-detection in claude-video. Learn when to use each for optimal video analysis and performance.

- Repository: [bradautomates/claude-video](https://github.com/bradautomates/claude-video)
- Tags: deep-dive
- Published: 2026-07-07

---

**Keyframe extraction decodes only I-frames using ffmpeg's `-skip_frame nokey` for minimal CPU cost, while scene-detection performs a full decode with ffmpeg's `scene` filter to catch perceptual changes exceeding a 0.20 threshold, trading performance for higher accuracy.**

Claude-Video converts video into representative frames before transcription using two distinct engines defined in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py). Understanding the differences between keyframe extraction and scene-detection helps you optimize for either speed or visual fidelity when processing video content. Each approach uses different underlying ffmpeg strategies, computational budgets, and fallback behaviors.

## Core Technical Differences

The two engines differ fundamentally in how they select frames from the input video stream.

### Keyframe Extraction

**Keyframe extraction** relies on the video encoder's existing I-frame markers. In [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py), the `extract_keyframes` function builds an ffmpeg command using `-skip_frame nokey`, which instructs the decoder to skip all non-keyframes and extract only frames that the encoder already flagged as keyframes.

This approach captures frames that the encoder deemed necessary for random access, which usually correspond to scene cuts but may also include bitrate management insertions. Because ffmpeg skips the majority of frames during decoding, this method is extremely fast and cheap on CPU.

### Scene-Detection

**Scene-detection** uses ffmpeg's built-in scene-change detector implemented in `extract_scene_candidates`. The function constructs a filtergraph with `select='eq(n\\,0)+gt(scene\\,THRESH)'`, which evaluates every frame to detect where the visual difference between consecutive frames exceeds `SCENE_THRESHOLD` (default 0.20).

This method always includes the first frame (`eq(n\\,0)`) and catches any perceptible visual change, even if the encoder did not insert a keyframe at that location. Because it requires decoding every frame to evaluate the scene metric, this approach is significantly more expensive than keyframe extraction.

## Performance and Cost Comparison

The computational cost differential between these two methods is substantial.

**Keyframe extraction** performs minimal work by decoding only I-frames. This makes it ideal for fast previews or processing well-keyframed content like most YouTube uploads. The function returns quickly because ffmpeg skips intervening frames at the decoder level.

**Scene-detection** requires a full decode of the entire video to calculate frame-to-frame differences. The additional processing catches subtle visual changes—such as slide transitions in screen recordings or videos with long GOPs (Group of Pictures)—at the cost of higher CPU utilization and longer processing times.

Both methods support configurable `max_frames`, `resolution`, and optional `dedup` parameters to control output size and quality.

## Fallback Behavior and Edge Cases

Both engines implement automatic fallback logic when the primary method yields insufficient frames.

### Keyframe Minimum Threshold

If `extract_keyframes` finds fewer than `KEYFRAME_MIN` (4) keyframes, the function discards the candidate frames and automatically calls the uniform sampler (`extract`) as a fallback. This handles videos with extremely sparse keyframe placement or static content.

### Scene Detection Minimum Threshold

Similarly, `extract_scene_or_uniform` checks if the scene detector found fewer than `SCENE_MIN_FRAMES` (8) cuts. When the count is insufficient, the pipeline falls back to uniform sampling (`extract`) to ensure adequate frame coverage.

### Uniform Extraction

The shared fallback method (`extract`) requests frames at a fixed FPS rate (capped by `MAX_FPS = 2.0`) and writes JPEGs. This provides consistent output when the content is too static for keyframe or scene-based methods to capture meaningful variation.

## Implementation Details in claude-video

The frame extraction logic resides in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py), which defines the constants and pipeline orchestration.

### Configuration Constants

The module defines the thresholds controlling each engine at lines 19-30:

```python
MAX_FPS = 2.0
SCENE_THRESHOLD = 0.20           # scene-change detection sensitivity

SCENE_MIN_FRAMES = 8             # minimum scene cuts to stay in scene mode

KEYFRAME_MIN = 4                 # minimum keyframes before falling back to uniform

```

These values determine when the system switches from the primary extraction method to uniform sampling.

### The Deduplication Pipeline

Regardless of which engine generates the frames, both paths funnel through `dedupe_perceptual` to remove near-duplicates. This function creates 16×16 pixel thumbnails of each extracted JPEG and calculates the mean absolute pixel difference between consecutive frames. Frames with a difference ≤ `DEDUP_THRESHOLD` (2.0) are discarded, ensuring that visually identical content is not over-represented in the final frame set.

## Practical Usage Examples

The following examples demonstrate how to invoke each extraction engine from the claude-video codebase.

### Keyframe Extraction

Use `extract_keyframes` for fast processing when you trust the video's existing keyframe structure:

```python
from pathlib import Path
from skills.watch.scripts import frames

video = "sample.mp4"
out_dir = Path("keyframes")

frames_key, meta_key = frames.extract_keyframes(
    video_path=video,
    out_dir=out_dir,
    resolution=512,
    max_frames=30,          # request up to 30 frames

    start_seconds=None,
    end_seconds=None,
    dedup=True,
)

print("Keyframe engine:", meta_key["engine"])
print("Frames returned:", len(frames_key))

```

### Scene-Detection Extraction

Use `extract_scene_or_uniform` when you need higher recall of visual changes, such as for slide decks or screen recordings:

```python
out_dir = Path("scene")

frames_scene, meta_scene = frames.extract_scene_or_uniform(
    video_path=video,
    out_dir=out_dir,
    fps=1.5,                # ignored when scene detection succeeds

    target_frames=30,
    resolution=512,
    max_frames=30,
    start_seconds=None,
    end_seconds=None,
    dedup=True,
)

print("Scene engine:", meta_scene["engine"])
print("Frames returned:", len(frames_scene))

```

### Uniform Fallback

You can also invoke the uniform extractor directly when you need predictable temporal sampling:

```python
out_dir = Path("uniform")

frames_uni, meta_uni = frames.extract(
    video_path=video,
    out_dir=out_dir,
    fps=1.0,
    resolution=512,
    max_frames=30,
    start_seconds=None,
    end_seconds=None,
)

print("Uniform engine:", meta_uni["engine"])

```

## Summary

- **Keyframe extraction** uses ffmpeg's `-skip_frame nokey` to cheaply decode only I-frames, making it ideal for fast previews of well-keyframed videos.
- **Scene-detection** performs a full decode using ffmpeg's `scene` filter with a default threshold of 0.20, catching perceptual changes that keyframes might miss but requiring significantly more CPU time.
- Both methods automatically fall back to uniform sampling when they detect fewer than the minimum required frames (`KEYFRAME_MIN` = 4 or `SCENE_MIN_FRAMES` = 8).
- Both paths apply `dedupe_perceptual` to remove visually identical frames using 16×16 pixel thumbnails and a mean absolute difference threshold of 2.0.
- The implementations reside in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py), with `extract_keyframes` and `extract_scene_or_uniform` serving as the primary entry points.

## Frequently Asked Questions

### When should I use keyframe extraction over scene-detection?

Use **keyframe extraction** when processing speed is critical and the video already contains sufficient I-frames, such as standard YouTube uploads or cinematic content with natural scene cuts. Use **scene-detection** when working with screen recordings, slide presentations, or videos with long GOPs where important visual changes might not align with encoder keyframes.

### Why does scene-detection take longer to process?

Scene-detection requires ffmpeg to decode every frame in the video to calculate the `scene` metric—the difference between consecutive frames. In contrast, keyframe extraction uses `-skip_frame nokey` to decode only I-frames, allowing ffmpeg to skip the vast majority of frames. The full decode necessary for scene analysis consumes significantly more CPU cycles.

### What happens if a video has no scene changes?

If the scene detector finds fewer than `SCENE_MIN_FRAMES` (8) cuts, `extract_scene_or_uniform` automatically falls back to the uniform sampler (`extract`), which samples frames at a fixed FPS rate. Similarly, if keyframe extraction yields fewer than `KEYFRAME_MIN` (4) frames, it also falls back to uniform sampling to ensure adequate frame coverage.

### How does the deduplication step work?

After extraction, both engines pass frames through `dedupe_perceptual`, which resizes each JPEG to a 16×16 pixel thumbnail and compares it to the previous frame using mean absolute pixel difference. If the difference is ≤ `DEDUP_THRESHOLD` (2.0), the frame is discarded as a near-duplicate. This prevents over-representation of static or minimally changing content in the final frame set.