How Keyframe Extraction Differs from Scene-Change Detection in claude-video

Keyframe extraction decodes only I-frames using FFmpeg's -skip_frame nokey for fast, low-cost sampling, while scene-change detection performs a full decode with the select='gt(scene,THRESH)' filter to identify visual discontinuities exceeding a 0.20 threshold.

The claude-video skill implements two distinct visual sampling strategies in skills/watch/scripts/frames.py to convert video into LLM-ready frames. Understanding how keyframe extraction differs from scene-change detection helps you choose between the "efficient" detail mode for speed or the "balanced" and "token-burner" modes for richer visual context.

Core Technical Differences

Both methods generate candidate frames for Claude's context window, but they use fundamentally different FFmpeg approaches to identify which frames to extract.

Keyframe Extraction: I-Frame Sampling

In skills/watch/scripts/frames.py (lines 776-815), the extract_keyframes function uses the FFmpeg argument -skip_frame nokey to decode only I-frames (keyframes). This leverages the video encoder's naturally placed keyframes, which often align with scene cuts but require significantly less processing since FFmpeg skips non-keyframe data during decoding.

The function parses FFmpeg's showinfo log (lines 822-834) to capture timestamps for each keyframe candidate. This approach produces a lightweight list of frames representing structural boundaries in the video stream.

Scene-Change Detection: Visual Difference Analysis

Conversely, scene-change detection in extract_scene_candidates (lines 226-254) uses the FFmpeg filter select='gt(scene,THRESH)' with a default SCENE_THRESHOLD of 0.20. This performs a full decode of the video stream, calculating the visual difference between consecutive frames to detect actual content changes.

This method produces a list of scene-change candidates including the first frame plus every detected cut where the scene metric exceeds the threshold (lines 268-280). Unlike keyframe extraction, this captures genuine visual transitions even when they don't align with encoder keyframes, providing richer context for analysis.

Fallback and Deduplication Strategies

Both pipelines share identical post-processing logic but trigger different fallback thresholds when initial sampling proves insufficient.

Uniform Frame Fallback

If the keyframe pipeline finds fewer than KEYFRAME_MIN (4) frames, it falls back to uniform frame extraction (lines 836-863). Similarly, if scene-change detection yields fewer than SCENE_MIN_FRAMES (8) cuts, it also resorts to uniform extraction (lines 510-525). This ensures a minimum viable sample regardless of video content.

Perceptual Deduplication Pipeline

After initial candidate generation, both methods run dedupe_perceptual to remove near-identical frames, followed by _even_sample to respect the frame cap (lines 870-882 for keyframes, lines 528-543 for scene detection). This shared deduplication ensures token efficiency regardless of which extraction engine generated the initial candidates.

Selecting the Right Mode

The skills/watch/scripts/watch.py entry point (lines 4-5 and 4-6) maps detail modes to extraction engines:

• Efficient mode (--detail efficient): Uses keyframe extraction when speed and low token cost are prioritized.
• Balanced and Token-burner modes (--detail balanced or --detail token-burner): Use scene-change detection to capture richer visual context at the cost of higher processing time.

Practical Implementation Examples

Both extraction methods return identical data structures—a list of dictionaries containing index, timestamp_seconds, path, and reason—but differ in their underlying candidate generation.

Extracting Keyframes for Efficient Mode

from pathlib import Path
from skills.watch.scripts.frames import extract_keyframes

video = "example.mp4"
out_dir = Path("/tmp/keyframes")
frames, meta = extract_keyframes(
    video_path=video,
    out_dir=out_dir,
    resolution=512,
    max_frames=50,          # cap for efficient mode

    start_seconds=None,
    end_seconds=None,
    dedup=True,
)

print("Engine:", meta["engine"])          # → "keyframe"

print("Selected frames:", len(frames))

Extracting Scene-Change Frames for Balanced Mode

from pathlib import Path
from skills.watch.scripts.frames import extract_scene_or_uniform, auto_fps

video = "example.mp4"
out_dir = Path("/tmp/scene")
duration = 120.0                 # seconds, e.g. from get_metadata()

fps, target = auto_fps(duration, max_frames=100)

frames, meta = extract_scene_or_uniform(
    video_path=video,
    out_dir=out_dir,
    fps=fps,
    target_frames=target,
    resolution=512,
    max_frames=100,               # cap for balanced mode

    start_seconds=None,
    end_seconds=None,
    dedup=True,
)

print("Engine:", meta["engine"])          # → "scene" or "uniform" (fallback)

print("Detected shots:", meta["candidate_count"])

Summary

• Keyframe extraction uses -skip_frame nokey to decode only I-frames (lines 776-815), making it fast but dependent on encoder placement.
• Scene-change detection uses select='gt(scene,0.20)' to calculate visual differences (lines 226-254), capturing genuine content changes through full decode.
• Both methods fall back to uniform extraction if minimum thresholds aren't met (4 for keyframes, 8 for scenes).
• Both pipelines share dedupe_perceptual and _even_sample logic (lines 870-882 and 528-543).
• Efficient mode selects keyframes; balanced/token-burner modes select scene-change detection.

Frequently Asked Questions

What is the minimum number of frames required before falling back to uniform extraction?

Keyframe extraction requires at least 4 frames (KEYFRAME_MIN) before falling back to uniform sampling, while scene-change detection requires 8 frames (SCENE_MIN_FRAMES). These thresholds are defined in skills/watch/scripts/frames.py at lines 836-863 and 510-525 respectively.

How does the deduplication process work for both extraction methods?

Both methods use the dedupe_perceptual function to remove visually similar frames, followed by _even_sample to evenly distribute the final frame selection according to the max_frames cap. This occurs at lines 870-882 for keyframes and lines 528-543 for scene-change detection.

Which detail mode should I use for analyzing fast-paced video content?

For fast-paced content with rapid scene changes, use balanced or token-burner mode. These modes trigger scene-change detection, which calculates actual visual differences rather than relying on encoder keyframes that may not align with rapid content changes.

What FFmpeg threshold is used for scene-change detection?

The scene-change detection uses a threshold of 0.20 defined by the SCENE_THRESHOLD constant. The FFmpeg filter select='gt(scene,0.20)' compares consecutive frames and extracts those where the scene metric exceeds this value, indicating a significant visual discontinuity.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}