How Keyframe Extraction (--detail efficient) Works Compared to Scene Detection

Keyframe extraction (--detail efficient) uses ffmpeg's I-frame skipping for fast, low-cost sampling, while scene detection (--detail scene) performs full video decoding to find visual cuts, trading speed for granularity.

In the bradautomates/claude-video repository, the watch skill provides multiple strategies for extracting representative frames from video files. Understanding the difference between --detail efficient and --detail scene helps you choose the right balance between processing speed and visual granularity when analyzing video content.

What Is Keyframe Extraction (--detail efficient)?

The --detail efficient mode extracts only keyframes (I-frames) already embedded in the video stream. These frames represent complete images that don't depend on other frames for decoding, making them natural candidates for video summarization.

Implementation Details

The core logic resides in extract_keyframes() within skills/watch/scripts/frames.py (lines 1006–1014). This function invokes ffmpeg with the -skip_frame nokey flag, which instructs the decoder to skip all non-keyframes and extract only I-frames:

# From frames.py - keyframe extraction logic

ffmpeg -skip_frame nokey -i input.mp4 ...

After extraction, timestamps are parsed from ffmpeg's showinfo output. The pipeline then applies dedupe_perceptual() to remove near-identical frames and optionally applies _even_sample() to respect the max_frames cap while ensuring the first and last frames are always preserved.

Fallback Behavior

If fewer than KEYFRAME_MIN (approximately 4) keyframes are detected, the engine automatically falls back to uniform-budget extraction by calling extract() (lines 1036–1069). This ensures you always receive a usable set of frames even in videos with sparse keyframe placement.

What Is Scene Detection (--detail scene)?

The --detail scene mode identifies scene cuts—points where visual content changes significantly between consecutive frames—regardless of whether those frames are keyframes.

Implementation Details

Implemented in extract_scene_candidates() within skills/watch/scripts/frames.py (lines 1052–1056), this method uses ffmpeg's select filter with a scene detection threshold:

ffmpeg -i input.mp4 -vf "select='eq(n\,0)+gt(scene\,THRESHOLD)'" ...

This forces a full decode of the entire video, computing the scene change metric for every frame. When the metric exceeds SCENE_THRESHOLD (default 0.20), ffmpeg emits that frame. The first frame is always included to ensure coverage from the video start.

Processing Overhead

Unlike keyframe extraction, scene detection requires decoding every frame to calculate visual differences. This results in significantly higher CPU usage and longer processing times, making it more expensive for large video files or batch processing.

Fallback Behavior

If the detected scene count falls below SCENE_MIN_FRAMES (approximately 8), the system falls back to uniform sampling via extract() (lines 1050–1054). This prevents under-sampling in videos with minimal visual changes.

Key Differences and Performance Implications

Speed vs. Granularity: --detail efficient is ideal for quick previews or processing many videos, as it only touches the keyframe index table. --detail scene yields richer frame sets for videos with subtle cuts but requires full-video decoding.

Frame Distribution: Keyframe extraction relies on encoder decisions about where to place I-frames, which typically occur at natural cut points but may miss rapid visual transitions. Scene detection captures any significant visual change, potentially providing more granular coverage.

Resource Consumption: Keyframe extraction is very cheap—ffmpeg seeks directly to keyframe locations without full decoding. Scene detection is more expensive, requiring complete frame processing to compute the scene metric.

Practical Usage Examples

Both modes respect the --max-frames parameter and apply perceptual deduplication before returning results. The entry point in skills/watch/scripts/watch.py parses the --detail flag and dispatches to the appropriate extraction function.

Efficient keyframe extraction for quick analysis:

python -m skills.watch.scripts.frames \
    /path/to/video.mp4 /tmp/out \
    --detail efficient \
    --max-frames 50

Scene-change detection for detailed visual analysis:

python -m skills.watch.scripts.frames \
    /path/to/video.mp4 /tmp/out \
    --detail scene \
    --max-frames 80

Summary

• Keyframe extraction (--detail efficient) uses ffmpeg -skip_frame nokey to extract only I-frames, offering fast, low-cost processing with minimal CPU overhead.
• Scene detection (--detail scene) uses ffmpeg's select filter with a scene threshold to find visual cuts, requiring full-video decoding but providing finer granularity.
• Both modes apply dedupe_perceptual() and _even_sample() to remove duplicates and respect max_frames limits.
• Fallback mechanisms ensure usability: KEYFRAME_MIN (4) for efficient mode and SCENE_MIN_FRAMES (8) for scene mode trigger uniform sampling if insufficient frames are detected.
• The implementation resides in skills/watch/scripts/frames.py, with CLI handling in skills/watch/scripts/watch.py.

Frequently Asked Questions

When should I use --detail efficient versus --detail scene?

Use --detail efficient when processing speed and low resource usage are priorities, such as when analyzing many videos or generating quick previews. Use --detail scene when you need to capture subtle visual transitions or rapid cuts that might occur between keyframes, and can afford the additional processing time.

How does the fallback mechanism work?

According to the source code in skills/watch/scripts/frames.py, if keyframe extraction finds fewer than 4 frames (KEYFRAME_MIN), or scene detection finds fewer than 8 frames (SCENE_MIN_FRAMES), the system automatically switches to uniform sampling via extract(). This ensures you always receive a representative set of frames even when the primary extraction strategy yields sparse results.

What is perceptual deduplication?

After initial frame extraction, both modes call dedupe_perceptual() to remove visually similar frames that might represent the same scene. This prevents redundant analysis of nearly identical images before the final _even_sample() step distributes frames evenly across the video timeline.

Can I control the number of extracted frames?

Yes. Both extraction modes respect the --max-frames parameter passed via the CLI in watch.py. After deduplication, the _even_sample() function ensures the final frame count doesn't exceed your specified maximum while preserving the first and last frames of the video for complete temporal coverage.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}