Scene-Change Detection vs Keyframe Extraction in claude‑video: When to Use Each Method

The claude-video skill uses keyframe extraction for fast, token-efficient analysis and scene-change detection when richer visual context is needed, with both methods falling back to uniform sampling when video content is sparse.

The claude-video repository by bradautomates implements two distinct frame-extraction strategies in [skills/watch/scripts/frames.py](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py). Understanding when scene-change detection versus keyframe extraction is triggered helps you optimize for speed, cost, or comprehension depending on your use case.

How Each Method Works at the FFmpeg Level

Both approaches rely on FFmpeg but use fundamentally different filters and decode strategies.

Keyframe Extraction: Decode-Only I-Frames

The extract_keyframes function in [frames.py](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) (lines 776–815) skips unnecessary decoding by targeting frames the encoder already marked as keyframes:

# From frames.py L776-L815

cmd = [
    ffmpeg_path,
    "-hide_banner", "-nostats",
    "-skip_frame", "nokey",      # Only decode I-frames

    "-i", video_path,
    "-an",                       # No audio

    "-vf", f"showinfo,scale={resolution}:-1:flags=fast_bilinear",
    str(out_template),
]

The -skip_frame nokey flag tells FFmpeg to decode only I-frames—compressed frames that don't depend on neighboring frames for reconstruction. Timestamps are scraped from FFmpeg's showinfo log output (lines 822–834).

Scene-Change Detection: Full Decode with Visual Analysis

The extract_scene_candidates function (lines 226–254) performs a complete decode and computes frame-to-frame differences:

# From frames.py L226-L254

scene_filter = f"select='gt(scene\\,{scene_threshold})'"
cmd = [
    ffmpeg_path,
    "-hide_banner",
    "-i", video_path,
    "-an",
    "-vf", f"{scene_filter},showinfo,scale={resolution}:-1:flags=fast_bilinear",
    "-vsync", "vfr",             # Variable frame rate output

    str(out_template),
]

This uses FFmpeg's scene metric to detect cuts whenever visual difference exceeds SCENE_THRESHOLD = 0.20. The first frame is always captured, then every subsequent scene cut (lines 268–280).

When Each Method Is Selected: The --detail Modes

The CLI entry point in [watch.py](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/watch.py) maps user intent to extraction engines:

--detail Mode	Engine Used	Typical Use Case
efficient	Keyframe extraction	Quick summaries, low token spend, speed priority
balanced	Scene-change detection	Moderate detail with contextual scene boundaries
token-burner	Scene-change detection	Maximum comprehension, highest token budget

This means scene-change detection versus keyframe extraction is not a manual choice but a consequence of your selected detail level.

Fallback Behavior: When Neither Method Finds Enough Frames

Both engines protect against sparse video content through identical fallback logic.

Keyframe Fallback

If fewer than KEYFRAME_MIN (4) keyframes are found, extract_keyframes switches to uniform FPS-based sampling (lines 836–863):

# After keyframe extraction attempt

if len(frames) < KEYFRAME_MIN:
    # Fallback to uniform extraction across video duration

    return _uniform_extraction(...)

Scene-Change Fallback

Similarly, extract_scene_or_uniform falls back if SCENE_MIN_FRAMES (8) scene cuts aren't detected (lines 510–525):

# From frames.py L510-L525

if candidate_count < SCENE_MIN_FRAMES:
    logger.info("Scene detection sparse; falling back to uniform extraction")
    return _uniform_extraction(...)

Shared Post-Processing Pipeline

After candidate generation, both methods converge on identical deduplication and capping logic:

1. dedupe_perceptual – Removes near-identical frames using perceptual hashing
2. _even_sample – Evenly subsamples to respect max_frames limit

For keyframes: lines 870–882.
For scene detection: lines 528–543.

Practical Code Examples

Efficient Mode: Keyframe Extraction

from pathlib import Path
from skills.watch.scripts.frames import extract_keyframes

video = "example.mp4"
out_dir = Path("/tmp/keyframes")

frames, meta = extract_keyframes(
    video_path=video,
    out_dir=out_dir,
    resolution=512,
    max_frames=50,
    start_seconds=None,
    end_seconds=None,
    dedup=True,
)

print("Engine:", meta["engine"])          # → "keyframe"

print("Selected frames:", len(frames))

Balanced Mode: Scene-Change Detection

from pathlib import Path
from skills.watch.scripts.frames import extract_scene_or_uniform, auto_fps

video = "example.mp4"
out_dir = Path("/tmp/scene")
duration = 120.0

fps, target = auto_fps(duration, max_frames=100)

frames, meta = extract_scene_or_uniform(
    video_path=video,
    out_dir=out_dir,
    fps=fps,
    target_frames=target,
    resolution=512,
    max_frames=100,
    start_seconds=None,
    end_seconds=None,
    dedup=True,
)

print("Engine:", meta["engine"])          # → "scene" or "uniform"

print("Detected shots:", meta["candidate_count"])

Both return identical structures: frames contains dictionaries with index, timestamp_seconds, path, and reason.

Performance and Cost Trade-offs

Factor	Keyframe Extraction	Scene-Change Detection
Decode cost	Minimal (I-frames only)	Full video decode
Accuracy	Coarse (encoder-dependent)	Precise (visual difference threshold)
Token efficiency	Higher	Lower
Scene boundary precision	Variable (keyframes ≠ cuts)	Exact (configurable threshold)
Best for	Talking heads, static shots	Fast cuts, dynamic content

Summary

• Keyframe extraction is selected automatically for --detail efficient, using FFmpeg's -skip_frame nokey for minimal decode cost with fallback to uniform sampling if fewer than 4 keyframes exist.

• Scene-change detection powers --detail balanced and --detail token-burner, performing full decode with FFmpeg's select='gt(scene,THRESH)' filter and falling back if fewer than 8 cuts are detected.

• Both pipelines share deduplication (dedupe_perceptual) and capping (_even_sample) logic in [skills/watch/scripts/frames.py](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py).

• The claude-video skill abstracts this choice through the --detail flag in [watch.py](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/watch.py), letting users optimize for speed or comprehension without managing FFmpeg filters directly.

Frequently Asked Questions

Can I force scene-change detection instead of keyframes in efficient mode?

No. The engine selection is hardcoded to the --detail parameter in [watch.py](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/watch.py). To use scene detection, specify --detail balanced. You could modify watch.py lines 45–46 to override the mapping if needed.

Why do both methods need a fallback to uniform extraction?

Some videos lack sufficient natural keyframes or scene cuts—think static timelapses, single-shot interviews, or slideshow-style content. The KEYFRAME_MIN (4) and SCENE_MIN_FRAMES (8) thresholds ensure the skill always returns a usable frame set rather than failing or returning too few images for context.

How does the scene threshold value affect results?

The default SCENE_THRESHOLD = 0.20 in [frames.py](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) balances sensitivity: lower values detect more subtle transitions (risking false positives), while higher values miss gradual changes. This is passed to FFmpeg's select filter as gt(scene,0.20) and is not currently exposed as a CLI flag.

Are extracted frames stored permanently?

Frames are written to the out_dir path specified in the function call. The return metadata includes path for each frame, but cleanup is caller-dependent. The test suite in [tests/test_frames.py](https://github.com/bradautomates/claude-video/blob/main/tests/test_frames.py) uses temporary directories for automatic cleanup.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}