# Keyframe Extraction vs Scene-Aware Selection in Claude-Video: 2 Algorithms Compared

> Compare keyframe extraction and scene-aware selection in Claude-Video. Discover how each algorithm processes video for efficient analysis and understand their speed and accuracy trade-offs. Learn which method suits your needs.

- Repository: [bradautomates/claude-video](https://github.com/bradautomates/claude-video)
- Tags: deep-dive
- Published: 2026-07-10

---

**Keyframe extraction leverages ffmpeg's `-skip_frame nokey` to pull only I-frames for maximum speed, while scene-aware selection uses ffmpeg's `scene` filter to analyze visual content and detect actual scene boundaries at the cost of higher processing time.**

The `bradautomates/claude-video` repository implements two distinct strategies for selecting representative frames from video content. Understanding the differences between **keyframe extraction and scene-aware selection** helps you choose the right approach for your specific video analysis pipeline, whether you prioritize processing speed or visual accuracy.

## Core Algorithmic Differences

### Keyframe Extraction: The Fast I-Frame Approach

The keyframe extraction strategy relies on the video encoder's existing **I-frames** (keyframes). Using ffmpeg's `-skip_frame nokey` flag, the algorithm instructs the decoder to skip all non-key frames entirely, extracting only frames that the encoder already marked as scene boundaries during compression.

This approach is extremely fast because ffmpeg avoids decoding frames that are not keyframes, making it the "cheap, near-instant tier" suitable for quick previews.

### Scene-Aware Selection: Content-Based Detection

The scene-aware selection algorithm takes a content-analysis approach. It uses ffmpeg's `scene` filter with the expression `select='gt(scene,THRESHOLD)'` to evaluate every frame's visual difference from the previous one. When the scene-change metric exceeds the configured threshold, the frame is collected as a candidate.

This method also explicitly includes the first frame via `eq(n\,0)` in the filter string, ensuring coverage even if the video starts with a static scene. Because it requires full frame decoding, this approach is significantly slower but yields higher fidelity to actual visual boundaries.

## Implementation in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py)

Both algorithms are implemented in the core frame extraction module.

**Keyframe extraction** resides in the `extract_keyframes` function (lines 76-82 and 85-90). The implementation constructs an ffmpeg command including `-skip_frame nokey` and `showinfo` to capture timestamps, followed by a call to `dedupe_perceptual` (lines 85-90) to remove near-identical frames.

**Scene-aware selection** is implemented across two functions: `extract_scene_candidates` (lines 17-25 and 52-58) handles the raw detection, while `extract_scene_or_uniform` (lines 110-124) orchestrates the full pipeline including fallback logic. The ffmpeg filter string is built as `select='eq(n\\,0)+gt(scene\\,THRESHOLD)'` to combine first-frame retention with scene detection.

## Fallback Thresholds and Edge Cases

Both algorithms implement protective fallbacks to **uniform frame extraction** when their respective strategies yield insufficient coverage:

- **Keyframe extraction** falls back to uniform sampling if fewer than **4** keyframes are found (`KEYFRAME_MIN = 4`, line 29).
- **Scene-aware selection** falls back if fewer than **8** scenes are detected (`SCENE_MIN_FRAMES = 8`, line 26).

These thresholds protect against static videos—such as screen recordings or talking-head footage—where scene detection or keyframe density might otherwise return almost nothing.

## Post-Processing Pipeline

Despite their different selection strategies, both algorithms share a common post-processing pipeline:

1. **Perceptual deduplication**: Both use `dedupe_perceptual` to remove visually similar frames.
2. **Even sampling**: If the candidate set exceeds the requested `max_frames`, both apply `_even_sample` to distribute selections evenly across the timeline.

In the final output, keyframe-extracted frames are labeled with `"reason": "keyframe"`, while scene-aware selections receive `"reason": "scene-change"` or `"reason": "first-frame"` for the initial frame.

## Performance Characteristics and Use Cases

**Keyframe extraction** is ideal when:
- Speed is the primary constraint
- You need coarse coverage of distinct moments
- Processing resources are limited

**Scene-aware selection** is preferred when:
- You need accurate alignment with actual content changes
- Working with narrative or action-heavy clips with frequent cuts
- Visual fidelity matters more than processing time

## Practical Code Examples

Use `extract_keyframes` for fast I-frame extraction with automatic fallback:

```python
from pathlib import Path
from skills.watch.scripts.frames import extract_keyframes

video = "sample.mp4"
out_dir = Path("./output")

# Fast keyframe extraction (falls back to uniform if <4 keyframes)

frames_kf, meta_kf = extract_keyframes(
    video_path=video,
    out_dir=out_dir,
    resolution=512,
    max_frames=50,
    dedup=True,
)
print("Engine:", meta_kf["engine"])  # "keyframe" or "uniform"

print("Frames:", len(frames_kf))

```

Use `extract_scene_or_uniform` for content-aware detection:

```python
from skills.watch.scripts.frames import extract_scene_or_uniform

# Scene-aware selection (falls back to uniform if <8 scenes)

frames_sc, meta_sc = extract_scene_or_uniform(
    video_path=video,
    out_dir=out_dir,
    fps=2.0,
    target_frames=100,
    resolution=512,
    max_frames=80,
    dedup=True,
)
print("Engine:", meta_sc["engine"])  # "scene" or "uniform"

print("Frames:", len(frames_sc))

```

## Summary

- **Keyframe extraction** uses ffmpeg's `-skip_frame nokey` to pull only I-frames, making it extremely fast but dependent on encoder decisions.
- **Scene-aware selection** analyzes every frame with ffmpeg's `scene` filter to detect actual visual changes, providing higher accuracy at the cost of processing time.
- Both strategies implement safeguards: keyframe extraction requires a minimum of 4 frames, while scene detection requires 8 scenes before falling back to uniform sampling.
- Both pipelines apply `dedupe_perceptual` and even sampling to optimize the final frame set.
- The implementations reside in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py), with `extract_keyframes` handling I-frame extraction and `extract_scene_or_uniform` managing content-aware detection.

## Frequently Asked Questions

### Which algorithm performs faster on long videos?

**Keyframe extraction** performs significantly faster because it skips decoding non-key frames entirely using ffmpeg's `-skip_frame nokey` flag. Scene-aware selection must decode every frame to calculate the scene-change metric, making it slower but more accurate for detecting actual visual boundaries.

### When should I choose scene-aware selection over keyframe extraction?

Choose **scene-aware selection** when you need frames that align with actual visual content changes rather than compression artifacts. This is essential for narrative videos, action-heavy clips, or any content where the encoder's keyframe placement might not match semantic scene boundaries. The algorithm detects cuts via the `scene` filter and labels frames with `"reason": "scene-change"`.

### What happens if my video has no scene changes?

If scene detection yields fewer than 8 frames (`SCENE_MIN_FRAMES = 8` in line 26 of [`frames.py`](https://github.com/bradautomates/claude-video/blob/main/frames.py)), the engine automatically falls back to **uniform frame extraction** to guarantee coverage. This protects against static content like screen recordings or talking-head videos where the `scene` filter would otherwise return almost nothing.

### How does the deduplication process work for both methods?

Both algorithms pass their raw candidates through `dedupe_perceptual` (referenced in lines 85-90 for keyframes and used in the scene pipeline) to remove visually similar frames before applying the final even sampling. This ensures the final `max_frames` limit is met with perceptually distinct images regardless of the initial extraction strategy.