Keyframe Extraction vs Scene-Aware Selection in Claude-Video: 2 Algorithms Compared

Keyframe extraction leverages ffmpeg's -skip_frame nokey to pull only I-frames for maximum speed, while scene-aware selection uses ffmpeg's scene filter to analyze visual content and detect actual scene boundaries at the cost of higher processing time.

The bradautomates/claude-video repository implements two distinct strategies for selecting representative frames from video content. Understanding the differences between keyframe extraction and scene-aware selection helps you choose the right approach for your specific video analysis pipeline, whether you prioritize processing speed or visual accuracy.

Core Algorithmic Differences

Keyframe Extraction: The Fast I-Frame Approach

The keyframe extraction strategy relies on the video encoder's existing I-frames (keyframes). Using ffmpeg's -skip_frame nokey flag, the algorithm instructs the decoder to skip all non-key frames entirely, extracting only frames that the encoder already marked as scene boundaries during compression.

This approach is extremely fast because ffmpeg avoids decoding frames that are not keyframes, making it the "cheap, near-instant tier" suitable for quick previews.

Scene-Aware Selection: Content-Based Detection

The scene-aware selection algorithm takes a content-analysis approach. It uses ffmpeg's scene filter with the expression select='gt(scene,THRESHOLD)' to evaluate every frame's visual difference from the previous one. When the scene-change metric exceeds the configured threshold, the frame is collected as a candidate.

This method also explicitly includes the first frame via eq(n\,0) in the filter string, ensuring coverage even if the video starts with a static scene. Because it requires full frame decoding, this approach is significantly slower but yields higher fidelity to actual visual boundaries.

Implementation in skills/watch/scripts/frames.py

Both algorithms are implemented in the core frame extraction module.

Keyframe extraction resides in the extract_keyframes function (lines 76-82 and 85-90). The implementation constructs an ffmpeg command including -skip_frame nokey and showinfo to capture timestamps, followed by a call to dedupe_perceptual (lines 85-90) to remove near-identical frames.

Scene-aware selection is implemented across two functions: extract_scene_candidates (lines 17-25 and 52-58) handles the raw detection, while extract_scene_or_uniform (lines 110-124) orchestrates the full pipeline including fallback logic. The ffmpeg filter string is built as select='eq(n\\,0)+gt(scene\\,THRESHOLD)' to combine first-frame retention with scene detection.

Fallback Thresholds and Edge Cases

Both algorithms implement protective fallbacks to uniform frame extraction when their respective strategies yield insufficient coverage:

• Keyframe extraction falls back to uniform sampling if fewer than 4 keyframes are found (KEYFRAME_MIN = 4, line 29).
• Scene-aware selection falls back if fewer than 8 scenes are detected (SCENE_MIN_FRAMES = 8, line 26).

These thresholds protect against static videos—such as screen recordings or talking-head footage—where scene detection or keyframe density might otherwise return almost nothing.

Post-Processing Pipeline

Despite their different selection strategies, both algorithms share a common post-processing pipeline:

1. Perceptual deduplication: Both use dedupe_perceptual to remove visually similar frames.
2. Even sampling: If the candidate set exceeds the requested max_frames, both apply _even_sample to distribute selections evenly across the timeline.

In the final output, keyframe-extracted frames are labeled with "reason": "keyframe", while scene-aware selections receive "reason": "scene-change" or "reason": "first-frame" for the initial frame.

Performance Characteristics and Use Cases

Keyframe extraction is ideal when:

• Speed is the primary constraint
• You need coarse coverage of distinct moments
• Processing resources are limited

Scene-aware selection is preferred when:

• You need accurate alignment with actual content changes
• Working with narrative or action-heavy clips with frequent cuts
• Visual fidelity matters more than processing time

Practical Code Examples

Use extract_keyframes for fast I-frame extraction with automatic fallback:

from pathlib import Path
from skills.watch.scripts.frames import extract_keyframes

video = "sample.mp4"
out_dir = Path("./output")

# Fast keyframe extraction (falls back to uniform if <4 keyframes)

frames_kf, meta_kf = extract_keyframes(
    video_path=video,
    out_dir=out_dir,
    resolution=512,
    max_frames=50,
    dedup=True,
)
print("Engine:", meta_kf["engine"])  # "keyframe" or "uniform"

print("Frames:", len(frames_kf))

Use extract_scene_or_uniform for content-aware detection:

from skills.watch.scripts.frames import extract_scene_or_uniform

# Scene-aware selection (falls back to uniform if <8 scenes)

frames_sc, meta_sc = extract_scene_or_uniform(
    video_path=video,
    out_dir=out_dir,
    fps=2.0,
    target_frames=100,
    resolution=512,
    max_frames=80,
    dedup=True,
)
print("Engine:", meta_sc["engine"])  # "scene" or "uniform"

print("Frames:", len(frames_sc))

Summary

• Keyframe extraction uses ffmpeg's -skip_frame nokey to pull only I-frames, making it extremely fast but dependent on encoder decisions.
• Scene-aware selection analyzes every frame with ffmpeg's scene filter to detect actual visual changes, providing higher accuracy at the cost of processing time.
• Both strategies implement safeguards: keyframe extraction requires a minimum of 4 frames, while scene detection requires 8 scenes before falling back to uniform sampling.
• Both pipelines apply dedupe_perceptual and even sampling to optimize the final frame set.
• The implementations reside in skills/watch/scripts/frames.py, with extract_keyframes handling I-frame extraction and extract_scene_or_uniform managing content-aware detection.

Frequently Asked Questions

Which algorithm performs faster on long videos?

Keyframe extraction performs significantly faster because it skips decoding non-key frames entirely using ffmpeg's -skip_frame nokey flag. Scene-aware selection must decode every frame to calculate the scene-change metric, making it slower but more accurate for detecting actual visual boundaries.

When should I choose scene-aware selection over keyframe extraction?

Choose scene-aware selection when you need frames that align with actual visual content changes rather than compression artifacts. This is essential for narrative videos, action-heavy clips, or any content where the encoder's keyframe placement might not match semantic scene boundaries. The algorithm detects cuts via the scene filter and labels frames with "reason": "scene-change".

What happens if my video has no scene changes?

If scene detection yields fewer than 8 frames (SCENE_MIN_FRAMES = 8 in line 26 of frames.py), the engine automatically falls back to uniform frame extraction to guarantee coverage. This protects against static content like screen recordings or talking-head videos where the scene filter would otherwise return almost nothing.

How does the deduplication process work for both methods?

Both algorithms pass their raw candidates through dedupe_perceptual (referenced in lines 85-90 for keyframes and used in the scene pipeline) to remove visually similar frames before applying the final even sampling. This ensures the final max_frames limit is met with perceptually distinct images regardless of the initial extraction strategy.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}