# Difference Between Scene-Change Detection and Keyframe Extraction in Video Analysis

> Understand the difference between scene-change detection and keyframe extraction in video analysis. Learn about precision vs speed in video processing.

- Repository: [bradautomates/claude-video](https://github.com/bradautomates/claude-video)
- Tags: deep-dive
- Published: 2026-07-31

---

**Scene-change detection performs a full pixel-level comparison between consecutive frames to identify visual discontinuities, while keyframe extraction simply decodes existing I-frames that the video encoder embedded for compression purposes; the former captures semantic scene boundaries with higher precision but requires more processing power, whereas the latter offers speed at the cost of potentially missing subtle transitions.**

The `bradautomates/claude-video` repository implements both approaches in its video analysis skill, allowing users to choose between computational efficiency and analytical depth. Understanding the distinction between these methods is essential for optimizing token costs and visual context when processing video content with large language models.

## Core Technical Mechanisms

The fundamental difference lies in how each method identifies "important" frames within the video stream.

### How Keyframe Extraction Works

In [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py), the `extract_keyframes()` function relies on the video codec's existing structure. It invokes FFmpeg with the `-skip_frame nokey` flag, which instructs the decoder to **decode only I-frames** (keyframes) while skipping predictive frames as defined in lines 776-815. These I-frames represent complete images that the encoder placed at intervals for seeking and compression efficiency, not necessarily at semantic scene boundaries.

### How Scene-Change Detection Works

Conversely, scene-change detection utilizes FFmpeg's `select='gt(scene,THRESH)'` filter to evaluate the visual difference between every consecutive frame pair according to lines 226-254. This method calculates a scene-change metric based on pixel histogram differences; when the value exceeds the `SCENE_THRESHOLD` (set to 0.20), the frame is flagged as a scene boundary. Unlike keyframe extraction, this approach requires **full decoding** of the video stream to compute inter-frame differences.

## Implementation Differences in claude-video

The repository exposes these mechanisms through distinct Python functions with different operational characteristics.

### Candidate Generation Logic

**Keyframe extraction** produces candidate timestamps by parsing FFmpeg's `showinfo` log output, extracting the presentation timestamps of decoded I-frames as implemented in lines 822-834. This yields a list of keyframe candidates that may or may not align with actual content changes.

**Scene-change detection** generates candidates by identifying the first frame and every subsequent frame where the scene metric exceeds the threshold, creating a list of detected cuts as shown in lines 268-280.

### Fallback Thresholds

Both implementations include safety mechanisms when initial sampling proves insufficient. Keyframe extraction falls back to **uniform frame extraction** (regular fps-based sampling) if fewer than `KEYFRAME_MIN` (4) keyframes are discovered in lines 836-863. Scene-change detection triggers the same uniform fallback when fewer than `SCENE_MIN_FRAMES` (8) scene cuts are detected according to lines 510-525.

### Post-Processing Pipeline

Despite their different extraction methods, both pipelines converge on identical deduplication logic. After initial candidate selection, both run `dedupe_perceptual()` to eliminate near-identical frames, followed by `_even_sample()` to enforce the maximum frame cap as referenced in lines 870-882 and 528-543.

## Performance Characteristics and Use Cases

The `claude-video` skill selects between these engines based on the `--detail` parameter passed through [`watch.py`](https://github.com/bradautomates/claude-video/blob/main/watch.py).

**Efficient Mode** (`--detail efficient`) activates keyframe extraction, prioritizing speed and minimal token consumption. This mode is ideal for long videos where rapid sampling outweighs the need for precise scene alignment.

**Balanced and Token-Burner Modes** (`--detail balanced` or `--detail token-burner`) engage scene-change detection to capture richer visual context at scene boundaries. These modes suit content analysis requiring accurate shot detection, such as narrative structure analysis or detailed visual Q&A.

## Practical Implementation Examples

The following examples demonstrate calling each extraction method directly from the [`frames.py`](https://github.com/bradautomates/claude-video/blob/main/frames.py) module.

### Extracting Keyframes (Efficient Mode)

```python
from pathlib import Path
from skills.watch.scripts.frames import extract_keyframes

video = "example.mp4"
out_dir = Path("/tmp/keyframes")
frames, meta = extract_keyframes(
    video_path=video,
    out_dir=out_dir,
    resolution=512,
    max_frames=50,          # cap for efficient mode

    start_seconds=None,
    end_seconds=None,
    dedup=True,
)

print("Engine:", meta["engine"])          # → "keyframe"

print("Selected frames:", len(frames))

```

### Extracting Scene-Change Frames (Balanced Mode)

```python
from pathlib import Path
from skills.watch.scripts.frames import extract_scene_or_uniform, auto_fps

video = "example.mp4"
out_dir = Path("/tmp/scene")
duration = 120.0                 # seconds, e.g. from get_metadata()

fps, target = auto_fps(duration, max_frames=100)

frames, meta = extract_scene_or_uniform(
    video_path=video,
    out_dir=out_dir,
    fps=fps,
    target_frames=target,
    resolution=512,
    max_frames=100,               # cap for balanced mode

    start_seconds=None,
    end_seconds=None,
    dedup=True,
)

print("Engine:", meta["engine"])          # → "scene" or "uniform" (fallback)

print("Detected shots:", meta["candidate_count"])

```

Both functions return a list of frame dictionaries containing `index`, `timestamp_seconds`, `path`, and `reason`, though they populate these lists through fundamentally different candidate generation strategies.

## Summary

- **Keyframe extraction** decodes only I-frames using `-skip_frame nokey`, offering speed but potentially missing scene boundaries that fall between encoder keyframes.
- **Scene-change detection** uses the `select='gt(scene,0.20)'` filter to perform full-frame analysis, accurately identifying visual cuts but requiring greater computational resources.
- Both methods in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) fall back to uniform fps-based sampling when initial extraction yields insufficient candidates (minimums of 4 and 8 frames respectively).
- The deduplication and capping pipeline is shared between both approaches, ensuring consistent output formats regardless of the extraction engine selected.

## Frequently Asked Questions

### Which method is faster for video analysis?

Keyframe extraction is significantly faster because it leverages the video codec's existing structure and skips predictive frames entirely. Scene-change detection requires full decoding and pixel-level comparison between consecutive frames, consuming more CPU cycles per minute of video processed.

### Can scene-change detection miss cuts that keyframe extraction catches?

While uncommon, keyframe extraction may capture frames at regular intervals that happen to align with scene cuts by coincidence, even if the encoder placed them for compression rather than content reasons. However, scene-change detection specifically targets visual discontinuities and will reliably catch every cut where the inter-frame difference exceeds the 0.20 threshold, making it more accurate for semantic boundary detection despite the higher computational cost.

### How does the uniform fallback work in claude-video?

When either `extract_keyframes()` finds fewer than 4 keyframes or `extract_scene_or_uniform()` detects fewer than 8 scene changes, the respective function abandons its primary strategy and switches to uniform frame extraction. This samples frames at regular intervals based on the video's frame rate and duration, ensuring a minimum viable set of visual data even when the video content lacks clear scene divisions or encoder keyframes.

### What is the SCENE_THRESHOLD parameter?

`SCENE_THRESHOLD` is a floating-point value set to 0.20 in [`frames.py`](https://github.com/bradautomates/claude-video/blob/main/frames.py) that determines the sensitivity of scene-change detection. It represents the threshold for FFmpeg's scene detection metric, where values closer to 0.0 detect almost any change (including minor camera movements) and values approaching 1.0 require dramatic visual differences to trigger a cut. The 0.20 default provides a balanced detection of meaningful content transitions while filtering out subtle fluctuations.