Keyframe Extraction vs Scene-Change Detection in Claude-Video: A Technical Comparison

Keyframe extraction leverages existing I-frames via decoder-skip logic for speed, while scene-change detection analyzes pixel differences between frames to identify visual cuts regardless of encoder settings.

The bradautomates/claude-video repository provides a robust frames module for intelligent video sampling located in skills/watch/scripts/frames.py. When building video analysis pipelines, understanding the difference between keyframe extraction and scene-change detection determines both processing speed and frame coverage quality. This article examines the implementation differences, performance characteristics, and selection criteria for each approach based on the actual source code.

How Keyframe Extraction Works

Keyframe extraction utilizes ffmpeg's -skip_frame nokey flag to decode only I-frames that the encoder has already inserted into the video stream. In skills/watch/scripts/frames.py, the extract_keyframes() function (lines 776-885) implements this strategy by instructing the decoder to bypass non-keyframe data entirely. After decoding, the function optionally deduplicates near-identical frames and applies even-sampling to respect configured max_frames limits.

How Scene-Change Detection Works

Scene-change detection operates through ffmpeg's select filter evaluating the scene metric using the expression gt(scene, threshold). The extract_scene_candidates() function (lines 174-226) forces a full decode of every frame to calculate pixel-level differences between consecutive images. When the difference exceeds SCENE_THRESHOLD (default 0.20), ffmpeg emits that frame as a scene boundary candidate. The implementation always preserves the first frame and subsequently deduplicates and samples the results.

Key Technical Differences

Data Source

• Keyframe extraction uses only pre-existing I-frames baked into the video container by the encoder.
• Scene-change detection recomputes cuts by analyzing actual frame-to-frame visual differences rather than metadata markers.

Performance Profile

• Keyframe extraction avoids full decoding, completing in milliseconds for most files regardless of video length.
• Scene-change detection requires decoding every frame, significantly increasing CPU usage and processing time proportional to video duration.

Reliability Characteristics

• Keyframe extraction may return insufficient frames for videos with sparse keyframe spacing, such as long screen recordings or low-motion content.
• Scene-change detection works regardless of encoder settings because it evaluates pixel differences rather than relying on encoder insertion points.

Configuration Options

• Keyframe extraction offers no tunable threshold parameter—it accepts whatever keyframes exist in the stream.
• Scene-change detection exposes configurable sensitivity through the threshold parameter (defaulting to 0.20) that tunes the gt(scene, threshold) filter.

Fallback Logic and Orchestration

The extract_scene_or_uniform() function (lines 511-574) implements intelligent fallback behavior when scene detection yields fewer than SCENE_MIN_FRAMES results. In this scenario, the system automatically switches to uniform sampling across the video timeline to guarantee minimum frame coverage. The orchestration layer in skills/watch/scripts/watch.py coordinates metadata collection, fps calculation, and engine selection to ensure optimal frame extraction across diverse video types.

Practical Implementation Examples

Extract frames using the keyframe engine for maximum speed:

from pathlib import Path
from skills.watch.scripts.frames import extract_keyframes

video = "example.mp4"
out_dir = Path("frames")

# Fast extraction using only encoder-generated keyframes

keyframes, key_meta = extract_keyframes(
    video_path=video,
    out_dir=out_dir,
    resolution=512,
    max_frames=50,
    dedup=True,
)
print(f"Engine: {key_meta['engine']}, Frames extracted: {len(keyframes)}")

Use scene-change detection for content-aware sampling:

from skills.watch.scripts.frames import extract_scene_candidates

# Content-aware extraction with full frame analysis

scene_frames, scene_meta = extract_scene_candidates(
    video_path=video,
    out_dir=out_dir,
    resolution=512,
    max_frames=100,
    threshold=0.20,  # Same as SCENE_THRESHOLD default

)
print(f"Engine: {scene_meta['engine']}, Frames extracted: {len(scene_frames)}")

Both functions return a list of dictionaries and metadata. Each frame dictionary contains:

• index: Position in the ordered frame list
• timestamp_seconds: Temporal offset in the source video
• path: Filesystem path to the extracted JPEG
• reason: Classification string ("keyframe", "scene-change", or "first-frame")

Summary

• Keyframe extraction in extract_keyframes() (lines 776-885) provides maximum speed by decoding only existing I-frames using -skip_frame nokey, making it ideal for quick previews of properly encoded content.
• Scene-change detection in extract_scene_candidates() (lines 174-226) performs pixel-level analysis via ffmpeg's gt(scene, threshold) filter, offering superior accuracy for videos with irregular keyframe distribution at the cost of higher computational overhead.
• The extract_scene_or_uniform() fallback mechanism (lines 511-574) ensures reliable frame coverage by switching to uniform sampling when scene detection returns fewer than SCENE_MIN_FRAMES results.
• Source files skills/watch/scripts/frames.py and skills/watch/scripts/watch.py contain the complete implementation for both extraction strategies and workflow orchestration.

Frequently Asked Questions

When should I use keyframe extraction over scene-change detection?

Use keyframe extraction when processing speed is critical and the source video contains frequent scene cuts or regular keyframe intervals. This approach works best with professionally encoded content where the encoder has already inserted I-frames at logical boundaries. Avoid keyframe extraction for long screen recordings or low-motion videos where keyframes may be spaced minutes apart, as the function may return insufficient frames for analysis.

What happens if scene-change detection finds too few frames?

The library automatically triggers fallback logic in extract_scene_or_uniform() located at lines 511-574 of skills/watch/scripts/frames.py. When scene detection returns fewer frames than SCENE_MIN_FRAMES, the function switches to uniform sampling across the video timeline to ensure adequate coverage for downstream analysis tasks.

Can I adjust the sensitivity of scene-change detection?

Yes, the extract_scene_candidates() function accepts a threshold parameter that controls the ffmpeg scene detection sensitivity. The default value matches SCENE_THRESHOLD = 0.20, where higher values require greater pixel difference between frames to trigger a cut. Lower values detect more subtle transitions but may increase false positives in videos with gradual lighting changes.

What information does the extraction output contain?

Both extraction functions return a tuple containing a list of frame dictionaries and metadata. Each frame dictionary includes the index (ordered position), timestamp_seconds (source video time), path (JPEG location), and reason (extraction classification such as "keyframe", "scene-change", or "first-frame"). This structure enables precise correlation between extracted frames and their original temporal positions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}