Scene-Change Detection vs Keyframe Extraction in Claude-Video: Technical Differences Explained

Scene-change detection performs pixel-level analysis across every decoded frame to identify visual transitions, while keyframe extraction rapidly selects only pre-existing I-frames already embedded by the video encoder, trading computational accuracy for processing speed.

Choosing the right frame extraction strategy in bradautomates/claude-video directly impacts both the quality of your video analysis and processing overhead. The repository's frames module, located at skills/watch/scripts/frames.py, implements these two approaches through distinct ffmpeg-based pipelines designed for different content types and performance requirements.

How Keyframe Extraction Works

The I-Frame Selection Pipeline

The extract_keyframes() function operates by invoking ffmpeg with the -skip_frame nokey flag, which instructs the decoder to process only I-frames (keyframes) already present in the video stream. These frames are inserted by the encoder at scene cuts or regular intervals during the original compression.

Implemented in lines 776-885 of skills/watch/scripts/frames.py, this method optionally deduplicates near-identical frames and applies even-sampling to respect a max_frames limit. Because ffmpeg skips non-key frames entirely without decoding them, this approach completes in milliseconds for most files, making it ideal for quick previews when the video already contains sufficient encoder-generated cuts.

How Scene-Change Detection Works

Content-Aware Frame Analysis

In contrast, extract_scene_candidates() (lines 174-226) runs ffmpeg with a select filter that evaluates the scene metric using the expression gt(scene, threshold). Each time the per-frame pixel difference exceeds the configured threshold—defaulting to 0.20 (SCENE_THRESHOLD)—ffmpeg emits that frame as a scene boundary.

This method forces ffmpeg to decode every frame in the video, analyzing actual visual content rather than relying on encoder metadata. The function always retains the first frame regardless of threshold, then applies deduplication and sampling to the results. This approach excels with content that has few or irregular keyframes, such as screen recordings or static slide presentations.

Critical Differences Between Approaches

Understanding the technical distinctions helps you select the appropriate method for your specific use case:

• Data Source: Keyframe extraction uses only existing I-frames baked into the video stream, while scene-change detection recomputes cuts by analyzing frame-to-frame pixel differences.
• Processing Cost: Keyframe extraction avoids full decoding, consuming minimal CPU. Scene detection decodes every frame, creating significant computational overhead but yielding finer-grained coverage.
• Encoder Independence: If a video contains few keyframes (common in long screen recordings), keyframe extraction may return insufficient frames. Scene detection works regardless of encoder settings because it evaluates actual pixel changes.
• Configurability: Scene detection exposes a threshold parameter for tuning sensitivity to motion, while keyframe extraction offers no comparable content-based parameter.

The Fallback Strategy: extract_scene_or_uniform()

The library implements intelligent fallback logic in extract_scene_or_uniform() (lines 511-574). This function first attempts scene detection; if fewer than SCENE_MIN_FRAMES are found, it automatically falls back to uniform sampling across the timeline.

This ensures robust behavior for static recordings where scene detection would return almost nothing, guaranteeing a usable set of representative frames even when visual content changes minimally over time.

Practical Implementation Examples

Both extraction methods follow a consistent API pattern, returning lists of frame metadata dictionaries containing index, timestamp_seconds, path, and reason fields:

from pathlib import Path
from skills.watch.scripts.frames import extract_keyframes, extract_scene_candidates

video = "example.mp4"
out_dir = Path("frames")

# Strategy 1: Keyframe extraction (fast, encoder-dependent)

keyframes, key_meta = extract_keyframes(
    video_path=video,
    out_dir=out_dir,
    resolution=512,
    max_frames=50,
    dedup=True,
)
print(f"Engine: {key_meta['engine']} | Found: {len(keyframes)} frames")

# Strategy 2: Scene-change detection (slower, content-aware)

scene_frames, scene_meta = extract_scene_candidates(
    video_path=video,
    out_dir=out_dir,
    resolution=512,
    max_frames=100,
    threshold=0.20,  # Adjust sensitivity (0.0-1.0)

)
print(f"Engine: {scene_meta['engine']} | Found: {len(scene_frames)} frames")

Integration with the Processing Pipeline

The skills/watch/scripts/watch.py file orchestrates the end-to-end workflow, selecting the appropriate extraction engine based on video characteristics. While watch.py handles metadata collection and fps calculation, the underlying frame extraction logic remains contained within skills/watch/scripts/frames.py, making these functions reusable for standalone frame processing tasks separate from the main pipeline.

Summary

• Keyframe extraction in claude-video leverages existing I-frames via ffmpeg's -skip_frame nokey for maximum speed, defined in extract_keyframes() at lines 776-885 of skills/watch/scripts/frames.py.
• Scene-change detection analyzes pixel differences using ffmpeg's select filter with a configurable threshold (default 0.20), implemented in extract_scene_candidates() at lines 174-226.
• Scene detection requires full frame decoding, making it slower but more reliable for videos with sparse or irregular keyframes.
• The extract_scene_or_uniform() function (lines 511-574) provides automatic fallback to uniform sampling when scene detection yields fewer than SCENE_MIN_FRAMES.
• Both methods return standardized metadata dictionaries compatible with downstream captioning and thumbnail generation workflows.

Frequently Asked Questions

Which method should I use for screen recordings?

Use scene-change detection. Screen recordings often contain long static sections with few keyframes, making keyframe extraction return an insufficient number of frames. The extract_scene_candidates() function analyzes actual pixel changes to detect meaningful transitions regardless of encoder settings.

Can I adjust the sensitivity of scene detection?

Yes. The threshold parameter in extract_scene_candidates() accepts values between 0.0 and 1.0, with the default set to 0.20 (SCENE_THRESHOLD). Lower values detect subtle transitions, while higher values only capture major scene cuts.

Why does keyframe extraction run faster than scene detection?

Keyframe extraction uses ffmpeg's -skip_frame nokey flag to decode only I-frames, skipping all intermediate frames. Scene detection requires decoding every frame to calculate pixel differences, which consumes significantly more CPU cycles but provides content-aware accuracy.

What happens if a video has no scene changes?

The extract_scene_or_uniform() function automatically falls back to uniform temporal sampling when fewer than SCENE_MIN_FRAMES are detected, ensuring you always receive a usable set of representative frames spread across the video timeline.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}