# How Keyframe Extraction Differs from Scene-Change Detection in claude-video

> Discover the key differences between keyframe extraction and scene-change detection in claude-video. Understand their unique methods for video analysis and sampling.

- Repository: [bradautomates/claude-video](https://github.com/bradautomates/claude-video)
- Tags: deep-dive
- Published: 2026-07-12

---

**Keyframe extraction decodes only I-frames using FFmpeg's `-skip_frame nokey` for fast, low-cost sampling, while scene-change detection performs a full decode with the `select='gt(scene,THRESH)'` filter to identify visual discontinuities exceeding a 0.20 threshold.**

The `claude-video` skill implements two distinct visual sampling strategies in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) to convert video into LLM-ready frames. Understanding how **keyframe extraction** differs from **scene-change detection** helps you choose between the **"efficient"** detail mode for speed or the **"balanced"** and **"token-burner"** modes for richer visual context.

## Core Technical Differences

Both methods generate candidate frames for Claude's context window, but they use fundamentally different FFmpeg approaches to identify which frames to extract.

### Keyframe Extraction: I-Frame Sampling

In [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) (lines 776-815), the `extract_keyframes` function uses the FFmpeg argument `-skip_frame nokey` to **decode only I-frames** (keyframes). This leverages the video encoder's naturally placed keyframes, which often align with scene cuts but require significantly less processing since FFmpeg skips non-keyframe data during decoding.

The function parses FFmpeg's `showinfo` log (lines 822-834) to capture timestamps for each keyframe candidate. This approach produces a lightweight list of frames representing structural boundaries in the video stream.

### Scene-Change Detection: Visual Difference Analysis

Conversely, scene-change detection in `extract_scene_candidates` (lines 226-254) uses the FFmpeg filter `select='gt(scene,THRESH)'` with a default **SCENE_THRESHOLD of 0.20**. This performs a full decode of the video stream, calculating the visual difference between consecutive frames to detect actual content changes.

This method produces a list of scene-change candidates including the first frame plus every detected cut where the scene metric exceeds the threshold (lines 268-280). Unlike keyframe extraction, this captures genuine visual transitions even when they don't align with encoder keyframes, providing richer context for analysis.

## Fallback and Deduplication Strategies

Both pipelines share identical post-processing logic but trigger different fallback thresholds when initial sampling proves insufficient.

### Uniform Frame Fallback

If the keyframe pipeline finds fewer than **KEYFRAME_MIN (4)** frames, it falls back to uniform frame extraction (lines 836-863). Similarly, if scene-change detection yields fewer than **SCENE_MIN_FRAMES (8)** cuts, it also resorts to uniform extraction (lines 510-525). This ensures a minimum viable sample regardless of video content.

### Perceptual Deduplication Pipeline

After initial candidate generation, both methods run `dedupe_perceptual` to remove near-identical frames, followed by `_even_sample` to respect the frame cap (lines 870-882 for keyframes, lines 528-543 for scene detection). This shared deduplication ensures token efficiency regardless of which extraction engine generated the initial candidates.

## Selecting the Right Mode

The [`skills/watch/scripts/watch.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/watch.py) entry point (lines 4-5 and 4-6) maps detail modes to extraction engines:

- **Efficient mode** (`--detail efficient`): Uses keyframe extraction when speed and low token cost are prioritized.
- **Balanced and Token-burner modes** (`--detail balanced` or `--detail token-burner`): Use scene-change detection to capture richer visual context at the cost of higher processing time.

## Practical Implementation Examples

Both extraction methods return identical data structures—a list of dictionaries containing `index`, `timestamp_seconds`, `path`, and `reason`—but differ in their underlying candidate generation.

### Extracting Keyframes for Efficient Mode

```python
from pathlib import Path
from skills.watch.scripts.frames import extract_keyframes

video = "example.mp4"
out_dir = Path("/tmp/keyframes")
frames, meta = extract_keyframes(
    video_path=video,
    out_dir=out_dir,
    resolution=512,
    max_frames=50,          # cap for efficient mode

    start_seconds=None,
    end_seconds=None,
    dedup=True,
)

print("Engine:", meta["engine"])          # → "keyframe"

print("Selected frames:", len(frames))

```

### Extracting Scene-Change Frames for Balanced Mode

```python
from pathlib import Path
from skills.watch.scripts.frames import extract_scene_or_uniform, auto_fps

video = "example.mp4"
out_dir = Path("/tmp/scene")
duration = 120.0                 # seconds, e.g. from get_metadata()

fps, target = auto_fps(duration, max_frames=100)

frames, meta = extract_scene_or_uniform(
    video_path=video,
    out_dir=out_dir,
    fps=fps,
    target_frames=target,
    resolution=512,
    max_frames=100,               # cap for balanced mode

    start_seconds=None,
    end_seconds=None,
    dedup=True,
)

print("Engine:", meta["engine"])          # → "scene" or "uniform" (fallback)

print("Detected shots:", meta["candidate_count"])

```

## Summary

- **Keyframe extraction** uses `-skip_frame nokey` to decode only I-frames (lines 776-815), making it fast but dependent on encoder placement.
- **Scene-change detection** uses `select='gt(scene,0.20)'` to calculate visual differences (lines 226-254), capturing genuine content changes through full decode.
- Both methods fall back to uniform extraction if minimum thresholds aren't met (4 for keyframes, 8 for scenes).
- Both pipelines share `dedupe_perceptual` and `_even_sample` logic (lines 870-882 and 528-543).
- **Efficient mode** selects keyframes; **balanced/token-burner modes** select scene-change detection.

## Frequently Asked Questions

### What is the minimum number of frames required before falling back to uniform extraction?

Keyframe extraction requires at least **4** frames (`KEYFRAME_MIN`) before falling back to uniform sampling, while scene-change detection requires **8** frames (`SCENE_MIN_FRAMES`). These thresholds are defined in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) at lines 836-863 and 510-525 respectively.

### How does the deduplication process work for both extraction methods?

Both methods use the `dedupe_perceptual` function to remove visually similar frames, followed by `_even_sample` to evenly distribute the final frame selection according to the `max_frames` cap. This occurs at lines 870-882 for keyframes and lines 528-543 for scene-change detection.

### Which detail mode should I use for analyzing fast-paced video content?

For fast-paced content with rapid scene changes, use **balanced** or **token-burner** mode. These modes trigger scene-change detection, which calculates actual visual differences rather than relying on encoder keyframes that may not align with rapid content changes.

### What FFmpeg threshold is used for scene-change detection?

The scene-change detection uses a threshold of **0.20** defined by the `SCENE_THRESHOLD` constant. The FFmpeg filter `select='gt(scene,0.20)'` compares consecutive frames and extracts those where the scene metric exceeds this value, indicating a significant visual discontinuity.