# Scene-Change Detection vs Keyframe Extraction in claude‑video: When to Use Each Method

> Learn when to use scene-change detection versus keyframe extraction in claude-video. Understand their distinct use cases for efficient video analysis.

- Repository: [bradautomates/claude-video](https://github.com/bradautomates/claude-video)
- Tags: deep-dive
- Published: 2026-08-03

---

**The `claude-video` skill uses keyframe extraction for fast, token-efficient analysis and scene-change detection when richer visual context is needed**, with both methods falling back to uniform sampling when video content is sparse.

The `claude-video` repository by bradautomates implements two distinct frame-extraction strategies in [[`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py)](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py). Understanding when **scene-change detection** versus **keyframe extraction** is triggered helps you optimize for speed, cost, or comprehension depending on your use case.

## How Each Method Works at the FFmpeg Level

Both approaches rely on FFmpeg but use fundamentally different filters and decode strategies.

### Keyframe Extraction: Decode-Only I-Frames

The `extract_keyframes` function in [[`frames.py`](https://github.com/bradautomates/claude-video/blob/main/frames.py)](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) (lines 776–815) skips unnecessary decoding by targeting frames the encoder already marked as keyframes:

```python

# From frames.py L776-L815

cmd = [
    ffmpeg_path,
    "-hide_banner", "-nostats",
    "-skip_frame", "nokey",      # Only decode I-frames

    "-i", video_path,
    "-an",                       # No audio

    "-vf", f"showinfo,scale={resolution}:-1:flags=fast_bilinear",
    str(out_template),
]

```

The `-skip_frame nokey` flag tells FFmpeg to **decode only I-frames**—compressed frames that don't depend on neighboring frames for reconstruction. Timestamps are scraped from FFmpeg's `showinfo` log output (lines 822–834).

### Scene-Change Detection: Full Decode with Visual Analysis

The `extract_scene_candidates` function (lines 226–254) performs a complete decode and computes frame-to-frame differences:

```python

# From frames.py L226-L254

scene_filter = f"select='gt(scene\\,{scene_threshold})'"
cmd = [
    ffmpeg_path,
    "-hide_banner",
    "-i", video_path,
    "-an",
    "-vf", f"{scene_filter},showinfo,scale={resolution}:-1:flags=fast_bilinear",
    "-vsync", "vfr",             # Variable frame rate output

    str(out_template),
]

```

This uses FFmpeg's `scene` metric to detect cuts whenever visual difference exceeds `SCENE_THRESHOLD = 0.20`. The first frame is always captured, then every subsequent scene cut (lines 268–280).

## When Each Method Is Selected: The `--detail` Modes

The CLI entry point in [[`watch.py`](https://github.com/bradautomates/claude-video/blob/main/watch.py)](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/watch.py) maps user intent to extraction engines:

| `--detail` Mode | Engine Used | Typical Use Case |
|-----------------|-------------|----------------|
| `efficient` | **Keyframe extraction** | Quick summaries, low token spend, speed priority |
| `balanced` | **Scene-change detection** | Moderate detail with contextual scene boundaries |
| `token-burner` | **Scene-change detection** | Maximum comprehension, highest token budget |

This means **scene-change detection versus keyframe extraction** is not a manual choice but a consequence of your selected detail level.

## Fallback Behavior: When Neither Method Finds Enough Frames

Both engines protect against sparse video content through identical fallback logic.

### Keyframe Fallback

If fewer than `KEYFRAME_MIN` (4) keyframes are found, `extract_keyframes` switches to uniform FPS-based sampling (lines 836–863):

```python

# After keyframe extraction attempt

if len(frames) < KEYFRAME_MIN:
    # Fallback to uniform extraction across video duration

    return _uniform_extraction(...)

```

### Scene-Change Fallback

Similarly, `extract_scene_or_uniform` falls back if `SCENE_MIN_FRAMES` (8) scene cuts aren't detected (lines 510–525):

```python

# From frames.py L510-L525

if candidate_count < SCENE_MIN_FRAMES:
    logger.info("Scene detection sparse; falling back to uniform extraction")
    return _uniform_extraction(...)

```

## Shared Post-Processing Pipeline

After candidate generation, both methods converge on identical deduplication and capping logic:

1. **`dedupe_perceptual`** – Removes near-identical frames using perceptual hashing
2. **`_even_sample`** – Evenly subsamples to respect `max_frames` limit

For keyframes: lines 870–882.  
For scene detection: lines 528–543.

## Practical Code Examples

### Efficient Mode: Keyframe Extraction

```python
from pathlib import Path
from skills.watch.scripts.frames import extract_keyframes

video = "example.mp4"
out_dir = Path("/tmp/keyframes")

frames, meta = extract_keyframes(
    video_path=video,
    out_dir=out_dir,
    resolution=512,
    max_frames=50,
    start_seconds=None,
    end_seconds=None,
    dedup=True,
)

print("Engine:", meta["engine"])          # → "keyframe"

print("Selected frames:", len(frames))

```

### Balanced Mode: Scene-Change Detection

```python
from pathlib import Path
from skills.watch.scripts.frames import extract_scene_or_uniform, auto_fps

video = "example.mp4"
out_dir = Path("/tmp/scene")
duration = 120.0

fps, target = auto_fps(duration, max_frames=100)

frames, meta = extract_scene_or_uniform(
    video_path=video,
    out_dir=out_dir,
    fps=fps,
    target_frames=target,
    resolution=512,
    max_frames=100,
    start_seconds=None,
    end_seconds=None,
    dedup=True,
)

print("Engine:", meta["engine"])          # → "scene" or "uniform"

print("Detected shots:", meta["candidate_count"])

```

Both return identical structures: `frames` contains dictionaries with `index`, `timestamp_seconds`, `path`, and `reason`.

## Performance and Cost Trade-offs

| Factor | Keyframe Extraction | Scene-Change Detection |
|--------|---------------------|------------------------|
| Decode cost | Minimal (I-frames only) | Full video decode |
| Accuracy | Coarse (encoder-dependent) | Precise (visual difference threshold) |
| Token efficiency | Higher | Lower |
| Scene boundary precision | Variable (keyframes ≠ cuts) | Exact (configurable threshold) |
| Best for | Talking heads, static shots | Fast cuts, dynamic content |

## Summary

- **Keyframe extraction** is selected automatically for `--detail efficient`, using FFmpeg's `-skip_frame nokey` for minimal decode cost with fallback to uniform sampling if fewer than 4 keyframes exist.

- **Scene-change detection** powers `--detail balanced` and `--detail token-burner`, performing full decode with FFmpeg's `select='gt(scene,THRESH)'` filter and falling back if fewer than 8 cuts are detected.

- Both pipelines share deduplication (`dedupe_perceptual`) and capping (`_even_sample`) logic in [[`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py)](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py).

- The `claude-video` skill abstracts this choice through the `--detail` flag in [[`watch.py`](https://github.com/bradautomates/claude-video/blob/main/watch.py)](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/watch.py), letting users optimize for speed or comprehension without managing FFmpeg filters directly.

## Frequently Asked Questions

### Can I force scene-change detection instead of keyframes in efficient mode?

No. The engine selection is hardcoded to the `--detail` parameter in [[`watch.py`](https://github.com/bradautomates/claude-video/blob/main/watch.py)](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/watch.py). To use scene detection, specify `--detail balanced`. You could modify [`watch.py`](https://github.com/bradautomates/claude-video/blob/main/watch.py) lines 45–46 to override the mapping if needed.

### Why do both methods need a fallback to uniform extraction?

Some videos lack sufficient natural keyframes or scene cuts—think static timelapses, single-shot interviews, or slideshow-style content. The `KEYFRAME_MIN` (4) and `SCENE_MIN_FRAMES` (8) thresholds ensure the skill always returns a usable frame set rather than failing or returning too few images for context.

### How does the scene threshold value affect results?

The default `SCENE_THRESHOLD = 0.20` in [[`frames.py`](https://github.com/bradautomates/claude-video/blob/main/frames.py)](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) balances sensitivity: lower values detect more subtle transitions (risking false positives), while higher values miss gradual changes. This is passed to FFmpeg's `select` filter as `gt(scene,0.20)` and is not currently exposed as a CLI flag.

### Are extracted frames stored permanently?

Frames are written to the `out_dir` path specified in the function call. The return metadata includes `path` for each frame, but cleanup is caller-dependent. The test suite in [[`tests/test_frames.py`](https://github.com/bradautomates/claude-video/blob/main/tests/test_frames.py)](https://github.com/bradautomates/claude-video/blob/main/tests/test_frames.py) uses temporary directories for automatic cleanup.