# Scene Detection vs Keyframe Extraction in Claude Video: How They Differ and When to Use Each

> Understand scene detection vs keyframe extraction in Claude Video. Learn how keyframe extraction prioritizes speed while scene detection offers accuracy for perceptual changes.

- Repository: [bradautomates/claude-video](https://github.com/bradautomates/claude-video)
- Tags: deep-dive
- Published: 2026-07-11

---

**Keyframe extraction decodes only encoder-flagged I-frames for speed, while scene detection performs a full video decode to catch perceptual changes using ffmpeg’s scene filter, trading performance for higher accuracy.**

The `bradautomates/claude-video` repository offers two independent engines for converting video into representative frames before transcription. While both methods populate the [`frames.py`](https://github.com/bradautomates/claude-video/blob/main/frames.py) pipeline, they differ in underlying ffmpeg commands, computational cost, and their ability to detect subtle visual changes. Choosing between **scene detection** and **keyframe extraction** depends on whether your priority is processing speed or capturing every visual transition.

## Core Technical Differences

### What Keyframe Extraction Captures

Keyframe extraction leverages the encoder’s existing I-frame markers by passing `-skip_frame nokey` to ffmpeg. This instructs the decoder to skip all non-keyframes, processing only frames that the video encoder already flagged as complete reference frames. According to the source in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py), these I-frames usually correspond to scene cuts but can also be inserted for bitrate management reasons, potentially missing subtle transitions in videos with long GOP structures.

### What Scene Detection Captures

Scene detection uses ffmpeg’s `select='eq(n\\,0)+gt(scene\\,THRESH)'` filter to evaluate every frame’s visual difference from its predecessor. When the calculated scene score exceeds `SCENE_THRESHOLD` (default **0.20**), the frame is retained. This method catches any perceptible change—including slide transitions in screen recordings or subtle lighting shifts—even when the encoder did not insert a keyframe at that boundary.

### Performance and Cost Comparison

**Keyframe extraction** is computationally cheap because ffmpeg skips the majority of frames during decode. **Scene detection** requires a full decode of the entire video to evaluate the scene-change filter, consuming significantly more CPU resources. Both methods enforce minimum thresholds: if `extract_keyframes` finds fewer than `KEYFRAME_MIN` (**4**) frames, or if scene detection finds fewer than `SCENE_MIN_FRAMES` (**8**) cuts, the pipeline automatically falls back to uniform sampling via the `extract` function.

## Implementation Details in [`frames.py`](https://github.com/bradautomates/claude-video/blob/main/frames.py)

### Keyframe Extraction with `extract_keyframes`

Located in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py), the `extract_keyframes` function builds an ffmpeg command that applies the `-skip_frame nokey` flag. The pipeline parses ffmpeg’s `showinfo` filter output to collect timestamps of the decoded I-frames. If the resulting list contains fewer than 4 frames, the function discards the candidates and invokes the uniform extractor (`extract`) as a fallback, ensuring the transcription pipeline always receives sufficient visual context.

### Scene Detection with `extract_scene_candidates`

The `extract_scene_candidates` function implements the perceptual detection logic. It always includes the first frame (`eq(n\\,0)`) and uses the `scene` filter to identify cuts where visual difference exceeds the threshold. The caller, `extract_scene_or_uniform`, validates the count against `SCENE_MIN_FRAMES` (8). When sufficient cuts exist, the pipeline proceeds to `dedupe_perceptual` before down-sampling to the requested `max_frames`.

### Shared Deduplication and Fallback Logic

Both engines share common post-processing steps defined in the same module. After extraction, frames pass through `dedupe_perceptual`, which thumbnails each JPEG to 16×16 pixels and drops frames whose mean absolute pixel difference is ≤ **2.0** (`DEDUP_THRESHOLD`). This prevents near-identical visual content from over-representing static scenes. The uniform fallback (`extract`) simply requests a fixed FPS and writes JPEGs, serving as the safety net when both specialized engines encounter insufficient variation.

## Practical Usage Examples

### Fast Keyframe Extraction

Use this when processing well-keyframed content like typical YouTube uploads where speed matters more than catching every visual nuance.

```python
from pathlib import Path
from skills.watch.scripts import frames

video = "sample.mp4"
out_dir = Path("keyframes")

frames_key, meta_key = frames.extract_keyframes(
    video_path=video,
    out_dir=out_dir,
    resolution=512,
    max_frames=30,
    start_seconds=None,
    end_seconds=None,
    dedup=True,
)
print("Engine:", meta_key["engine"])  # "keyframe"

print("Frames:", len(frames_key))

```

### Accurate Scene Detection

Use this for screen captures, presentations, or videos with long GOPs where visual changes might not align with encoder keyframes.

```python
out_dir = Path("scenes")

frames_scene, meta_scene = frames.extract_scene_or_uniform(
    video_path=video,
    out_dir=out_dir,
    fps=1.5,                # Ignored when scene detection succeeds

    target_frames=30,
    resolution=512,
    max_frames=30,
    start_seconds=None,
    end_seconds=None,
    dedup=True,
)
print("Engine:", meta_scene["engine"])  # "scene" or "uniform"

print("Frames:", len(frames_scene))

```

### Uniform Fallback (Automatic)

This runs automatically when the primary engines fail to meet minimum frame thresholds, ensuring robustness across all video types.

```python
out_dir = Path("uniform")

frames_uni, meta_uni = frames.extract(
    video_path=video,
    out_dir=out_dir,
    fps=1.0,
    resolution=512,
    max_frames=30,
    start_seconds=None,
    end_seconds=None,
)
print("Engine:", meta_uni["engine"])  # "uniform"

```

## Summary

- **Keyframe extraction** uses `-skip_frame nokey` to cheaply decode only I-frames, falling back to uniform sampling if fewer than 4 keyframes are found.
- **Scene detection** uses ffmpeg’s `scene` filter with a default threshold of 0.20 to catch perceptual changes during a full decode, requiring `SCENE_MIN_FRAMES` (8) cuts to avoid fallback.
- Both engines apply `dedupe_perceptual` to remove near-duplicate frames using a 16×16 pixel thumbnail comparison with a threshold of 2.0.
- Keyframe extraction is preferred for speed and typical web video; scene detection is preferred for screen recordings and content with sparse keyframes.

## Frequently Asked Questions

### When should I use keyframe extraction over scene detection?

Use **keyframe extraction** when processing speed is critical and the video source uses standard encoding with frequent I-frames, such as most YouTube uploads or streaming content. It is significantly cheaper because it skips non-keyframes during decode. Use **scene detection** when working with screen recordings, presentation slides, or videos using long GOPs where important visual changes might occur between encoded keyframes.

### Why does scene detection take longer to process?

Scene detection requires a **full decode** of every video frame to calculate the `scene` filter value in ffmpeg, comparing each frame against its predecessor to detect perceptual differences. In contrast, keyframe extraction uses `-skip_frame nokey` to decode only the pre-flagged I-frames, drastically reducing CPU cycles. The trade-off is accuracy: scene detection catches subtle visual transitions that keyframe extraction might miss.

### What happens if a video has no scene cuts or keyframes?

If `extract_keyframes` finds fewer than `KEYFRAME_MIN` (4) frames, or if scene detection identifies fewer than `SCENE_MIN_FRAMES` (8) cuts, Claude Video automatically invokes the **uniform sampling** engine (`extract`). This fallback extracts frames at a fixed FPS (typically 1.0–2.0) to ensure the transcription pipeline receives the minimum required visual context regardless of the video’s encoding characteristics.

### How does Claude Video handle duplicate frames?

After both extraction engines, frames pass through `dedupe_perceptual` in [`frames.py`](https://github.com/bradautomates/claude-video/blob/main/frames.py). This function creates 16×16 pixel thumbnails of each JPEG and calculates the mean absolute pixel difference between consecutive frames. If the difference is ≤ **2.0** (`DEDUP_THRESHOLD`), the frame is discarded as a near-duplicate. This deduplication step runs by default but can be disabled by setting `dedup=False` in the extraction calls.