# Keyframe Extraction vs Scene Detection in Claude Video: What's the Difference and When to Use Each

> Understand keyframe extraction vs scene detection in Claude Video. Learn when to use each for optimal performance and visual change detection.

- Repository: [bradautomates/claude-video](https://github.com/bradautomates/claude-video)
- Tags: deep-dive
- Published: 2026-08-02

---

**Keyframe extraction relies on decoder-embedded I-frames for speed, while scene detection performs a full video decode using ffmpeg's `scene` filter to catch every visual change—each with distinct performance trade-offs and fallback behaviors.**

Claude Video offers two independent engines for converting videos into representative frames prior to transcription. Understanding the difference between **keyframe extraction** and **scene detection** helps you choose the right approach for your video type and performance requirements. This article examines the implementation in `bradautomates/claude-video`, tracing the actual source code in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) to explain how each engine works, when it falls back, and how to configure it.

## How Keyframe Extraction Works

The **keyframe extraction** engine is the faster, encoder-driven option. It leverages the fact that video encoders already mark certain frames as "keyframes" (I-frames) that can be decoded without reference to other frames.

### Underlying FFmpeg Command

In `extract_keyframes` (lines 45-90 of [`frames.py`](https://github.com/bradautomates/claude-video/blob/main/frames.py)), the code constructs an ffmpeg command using `-skip_frame nokey`:

```bash
ffmpeg -skip_frame nokey -i input.mp4 -vf "showinfo,scale=512:-1" ...

```

This flag instructs ffmpeg to **skip every non-keyframe** during decoding. The decoder only touches I-frames, dramatically reducing CPU work.

### What Keyframes Actually Capture

Keyframes are inserted by the encoder for two reasons:

- **Scene cuts** — when visual content changes significantly
- **Bitrate management** — to reset the prediction chain and control file size

This means keyframe extraction often captures scene boundaries, but **misses subtle visual changes** that didn't trigger a keyframe insertion. Videos with long GOP (Group of Pictures) structures may have sparse keyframes despite frequent visual changes.

### Fallback Behavior

If the keyframe count falls below `KEYFRAME_MIN` (4 frames), the engine abandons its results and calls `extract()` for uniform sampling:

```python

# From frames.py lines 78-85

if len(frames) < KEYFRAME_MIN:
    logger.warning(f"Only {len(frames)} keyframes found, falling back to uniform sampling")
    return extract(
        video_path=video_path,
        out_dir=out_dir,
        fps=1.0,  # Default uniform rate

        # ... additional parameters

    )

```

This ensures you always receive usable frames even from static videos.

## How Scene Detection Works

The **scene detection** engine prioritizes **recall over speed**. It analyzes every frame to detect perceptual changes, regardless of encoder keyframe placement.

### Underlying FFmpeg Command

In `extract_scene_candidates` (lines 115-165), the engine uses ffmpeg's `select` filter with the scene detection expression:

```bash
ffmpeg -i input.mp4 -vf "select='eq(n\,0)+gt(scene\,0.20)',showinfo,scale=512:-1" ...

```

The filter `gt(scene\,THRESH)` outputs frames where the **scene-change metric exceeds the threshold** (default 0.20). The `eq(n\,0)` term always includes the first frame.

### Full Decode Requirement

Unlike keyframe extraction, this approach **decode every frame** to compute inter-frame differences. The `scene` filter analyzes pixel-level changes between consecutive frames, which requires complete video decoding.

This makes scene detection **significantly more expensive** for long or high-resolution videos, but it catches:

- Slide transitions in presentations
- Screen recording changes without keyframes
- Subtle visual shifts that encoders optimize away

### Fallback and Post-Processing

The `extract_scene_or_uniform` wrapper (lines 168-210) validates results against `SCENE_MIN_FRAMES` (8 cuts minimum):

```python

# From frames.py lines 185-195

if len(scene_frames) < SCENE_MIN_FRAMES:
    logger.info(f"Scene detection found {len(scene_frames)} frames, using uniform sampling")
    return extract(
        video_path=video_path,
        out_dir=out_dir,
        fps=fps,
        # ... additional parameters

    )

```

When sufficient scenes exist, the pipeline applies `dedupe_perceptual()` to remove near-identical frames, then evenly samples down to `max_frames`.

## Shared Infrastructure: Deduplication and Uniform Fallback

Both engines converge on common utilities for final processing.

### Perceptual Deduplication

After any extraction path, frames pass through `dedupe_perceptual()` (lines 220-275). This function:

1. Resizes each JPEG to 16×16 pixels
2. Computes **mean absolute pixel difference** between consecutive frames
3. Drops frames where difference ≤ `DEDUP_THRESHOLD` (2.0)

```python
from skills.watch.scripts import frames

frames, meta = frames.extract_keyframes(
    video_path="talk.mp4",
    out_dir="./frames",
    dedup=True,  # Enable perceptual deduplication (default)

)

```

### Uniform Sampling Fallback

The `extract()` function (lines 35-70) serves as the reliable baseline. It requests frames at a fixed FPS without scene analysis:

```python
frames, meta = frames.extract(
    video_path="static.mp4",
    out_dir="./frames",
    fps=1.0,        # One frame per second

    max_frames=30,
    resolution=512,
)

```

Both keyframe and scene engines delegate to this function when their primary method yields insufficient frames.

## Configuration Constants

The behavior of both engines is controlled by constants defined at the top of [`frames.py`](https://github.com/bradautomates/claude-video/blob/main/frames.py) (lines 19-30):

```python
MAX_FPS = 2.0
SCENE_THRESHOLD = 0.20      # scene-change detection sensitivity (0.0-1.0)

SCENE_MIN_FRAMES = 8        # minimum scene cuts before trusting scene mode

KEYFRAME_MIN = 4            # minimum keyframes before falling back

DEDUP_THRESHOLD = 2.0       # mean pixel difference for deduplication

```

## Practical Code Examples

### Fast Keyframe Extraction

Use this for typical web videos with reasonable keyframe density:

```python
from pathlib import Path
from skills.watch.scripts import frames

frames, meta = frames.extract_keyframes(
    video_path="youtube_video.mp4",
    out_dir=Path("keyframes"),
    resolution=512,
    max_frames=30,
    dedup=True,
)
print(f"Engine: {meta['engine']}")  # "keyframe"

print(f"Frames: {len(frames)}")

```

### Precision Scene Detection

Use this for presentations, screen recordings, or videos with long GOPs:

```python
frames, meta = frames.extract_scene_or_uniform(
    video_path="webinar_recording.mp4",
    out_dir=Path("scenes"),
    fps=1.5,                    # Ignored if scene detection succeeds

    target_frames=30,
    resolution=512,
    max_frames=30,
    dedup=True,
)
print(f"Engine: {meta['engine']}")  # "scene" or "uniform"

```

### Direct Uniform Sampling

Force consistent frame spacing without content analysis:

```python
frames, meta = frames.extract(
    video_path="security_footage.mp4",
    out_dir=Path("uniform"),
    fps=0.5,                    # One frame every 2 seconds

    resolution=512,
    max_frames=30,
)
print(f"Engine: {meta['engine']}")  # "uniform"

```

## Key Differences Summary

| Aspect | Keyframe Extraction | Scene Detection |
|--------|---------------------|-----------------|
| **Speed** | Very fast (decoder skips non-keyframes) | Slower (full decode required) |
| **Accuracy** | Encoder-dependent; may miss changes | Catches all perceptual changes |
| **Best for** | Standard web videos, fast previews | Presentations, screen recordings, analysis |
| **Fallback trigger** | < 4 keyframes found | < 8 scene cuts detected |
| **Unique parameter** | None (simple on/off) | `threshold` (0.0-1.0 sensitivity) |

## Summary

- **Keyframe extraction** uses `ffmpeg -skip_frame nokey` to decode only I-frames, making it the fastest option for well-keyframed videos.
- **Scene detection** employs `ffmpeg select='gt(scene,THRESH)'` with full decoding to catch every visual change at higher CPU cost.
- Both engines fall back to uniform `fps`-based sampling when their primary method yields insufficient frames (`KEYFRAME_MIN` = 4, `SCENE_MIN_FRAMES` = 8).
- All paths apply `dedupe_perceptual()` to remove redundant frames using 16×16 thumbnail comparison.
- Choose **keyframes** for speed on typical content; choose **scene detection** when visual completeness matters more than processing time.

## Frequently Asked Questions

### Can I use both keyframe extraction and scene detection on the same video?

No — these are mutually exclusive engines in Claude Video's current architecture. You call either `extract_keyframes()` or `extract_scene_or_uniform()` for a given extraction task. If you need both approaches for comparison, run them as separate operations and merge the results manually.

### How do I adjust the sensitivity of scene detection?

Pass the `threshold` parameter to `extract_scene_or_uniform()`. The default is 0.20 (defined by `SCENE_THRESHOLD` in [`frames.py`](https://github.com/bradautomates/claude-video/blob/main/frames.py)). Lower values (0.10-0.15) detect more subtle changes; higher values (0.30-0.40) require more dramatic scene cuts. Values outside 0.0-1.0 may produce unpredictable results.

### Why does keyframe extraction sometimes fall back to uniform sampling despite visible scene changes?

The encoder may have placed keyframes sparsely for bitrate efficiency, or the video may genuinely lack visual variety. The `KEYFRAME_MIN = 4` safeguard ensures you receive usable frames rather than a sparse, potentially unrepresentative set. Uniform sampling guarantees coverage across the video timeline.

### Does deduplication run before or after max_frames is applied?

Deduplication (`dedupe_perceptual`) runs **before** the final frame count reduction in the scene detection pipeline, but the timing varies by path. For keyframe extraction, deduplication filters the raw keyframes, then results are returned directly. For scene detection, deduplication occurs before `max_frames` caps the result set through even sampling.