Keyframe Extraction vs Scene-Detection in Claude-Video: Technical Differences and Usage Guide

Keyframe extraction decodes only I-frames using ffmpeg's -skip_frame nokey for minimal CPU cost, while scene-detection performs a full decode with ffmpeg's scene filter to catch perceptual changes exceeding a 0.20 threshold, trading performance for higher accuracy.

Claude-Video converts video into representative frames before transcription using two distinct engines defined in skills/watch/scripts/frames.py. Understanding the differences between keyframe extraction and scene-detection helps you optimize for either speed or visual fidelity when processing video content. Each approach uses different underlying ffmpeg strategies, computational budgets, and fallback behaviors.

Core Technical Differences

The two engines differ fundamentally in how they select frames from the input video stream.

Keyframe Extraction

Keyframe extraction relies on the video encoder's existing I-frame markers. In skills/watch/scripts/frames.py, the extract_keyframes function builds an ffmpeg command using -skip_frame nokey, which instructs the decoder to skip all non-keyframes and extract only frames that the encoder already flagged as keyframes.

This approach captures frames that the encoder deemed necessary for random access, which usually correspond to scene cuts but may also include bitrate management insertions. Because ffmpeg skips the majority of frames during decoding, this method is extremely fast and cheap on CPU.

Scene-Detection

Scene-detection uses ffmpeg's built-in scene-change detector implemented in extract_scene_candidates. The function constructs a filtergraph with select='eq(n\\,0)+gt(scene\\,THRESH)', which evaluates every frame to detect where the visual difference between consecutive frames exceeds SCENE_THRESHOLD (default 0.20).

This method always includes the first frame (eq(n\\,0)) and catches any perceptible visual change, even if the encoder did not insert a keyframe at that location. Because it requires decoding every frame to evaluate the scene metric, this approach is significantly more expensive than keyframe extraction.

Performance and Cost Comparison

The computational cost differential between these two methods is substantial.

Keyframe extraction performs minimal work by decoding only I-frames. This makes it ideal for fast previews or processing well-keyframed content like most YouTube uploads. The function returns quickly because ffmpeg skips intervening frames at the decoder level.

Scene-detection requires a full decode of the entire video to calculate frame-to-frame differences. The additional processing catches subtle visual changes—such as slide transitions in screen recordings or videos with long GOPs (Group of Pictures)—at the cost of higher CPU utilization and longer processing times.

Both methods support configurable max_frames, resolution, and optional dedup parameters to control output size and quality.

Fallback Behavior and Edge Cases

Both engines implement automatic fallback logic when the primary method yields insufficient frames.

Keyframe Minimum Threshold

If extract_keyframes finds fewer than KEYFRAME_MIN (4) keyframes, the function discards the candidate frames and automatically calls the uniform sampler (extract) as a fallback. This handles videos with extremely sparse keyframe placement or static content.

Scene Detection Minimum Threshold

Similarly, extract_scene_or_uniform checks if the scene detector found fewer than SCENE_MIN_FRAMES (8) cuts. When the count is insufficient, the pipeline falls back to uniform sampling (extract) to ensure adequate frame coverage.

Uniform Extraction

The shared fallback method (extract) requests frames at a fixed FPS rate (capped by MAX_FPS = 2.0) and writes JPEGs. This provides consistent output when the content is too static for keyframe or scene-based methods to capture meaningful variation.

Implementation Details in claude-video

The frame extraction logic resides in skills/watch/scripts/frames.py, which defines the constants and pipeline orchestration.

Configuration Constants

The module defines the thresholds controlling each engine at lines 19-30:

MAX_FPS = 2.0
SCENE_THRESHOLD = 0.20           # scene-change detection sensitivity

SCENE_MIN_FRAMES = 8             # minimum scene cuts to stay in scene mode

KEYFRAME_MIN = 4                 # minimum keyframes before falling back to uniform

These values determine when the system switches from the primary extraction method to uniform sampling.

The Deduplication Pipeline

Regardless of which engine generates the frames, both paths funnel through dedupe_perceptual to remove near-duplicates. This function creates 16×16 pixel thumbnails of each extracted JPEG and calculates the mean absolute pixel difference between consecutive frames. Frames with a difference ≤ DEDUP_THRESHOLD (2.0) are discarded, ensuring that visually identical content is not over-represented in the final frame set.

Practical Usage Examples

The following examples demonstrate how to invoke each extraction engine from the claude-video codebase.

Keyframe Extraction

Use extract_keyframes for fast processing when you trust the video's existing keyframe structure:

from pathlib import Path
from skills.watch.scripts import frames

video = "sample.mp4"
out_dir = Path("keyframes")

frames_key, meta_key = frames.extract_keyframes(
    video_path=video,
    out_dir=out_dir,
    resolution=512,
    max_frames=30,          # request up to 30 frames

    start_seconds=None,
    end_seconds=None,
    dedup=True,
)

print("Keyframe engine:", meta_key["engine"])
print("Frames returned:", len(frames_key))

Scene-Detection Extraction

Use extract_scene_or_uniform when you need higher recall of visual changes, such as for slide decks or screen recordings:

out_dir = Path("scene")

frames_scene, meta_scene = frames.extract_scene_or_uniform(
    video_path=video,
    out_dir=out_dir,
    fps=1.5,                # ignored when scene detection succeeds

    target_frames=30,
    resolution=512,
    max_frames=30,
    start_seconds=None,
    end_seconds=None,
    dedup=True,
)

print("Scene engine:", meta_scene["engine"])
print("Frames returned:", len(frames_scene))

Uniform Fallback

You can also invoke the uniform extractor directly when you need predictable temporal sampling:

out_dir = Path("uniform")

frames_uni, meta_uni = frames.extract(
    video_path=video,
    out_dir=out_dir,
    fps=1.0,
    resolution=512,
    max_frames=30,
    start_seconds=None,
    end_seconds=None,
)

print("Uniform engine:", meta_uni["engine"])

Summary

• Keyframe extraction uses ffmpeg's -skip_frame nokey to cheaply decode only I-frames, making it ideal for fast previews of well-keyframed videos.
• Scene-detection performs a full decode using ffmpeg's scene filter with a default threshold of 0.20, catching perceptual changes that keyframes might miss but requiring significantly more CPU time.
• Both methods automatically fall back to uniform sampling when they detect fewer than the minimum required frames (KEYFRAME_MIN = 4 or SCENE_MIN_FRAMES = 8).
• Both paths apply dedupe_perceptual to remove visually identical frames using 16×16 pixel thumbnails and a mean absolute difference threshold of 2.0.
• The implementations reside in skills/watch/scripts/frames.py, with extract_keyframes and extract_scene_or_uniform serving as the primary entry points.

Frequently Asked Questions

When should I use keyframe extraction over scene-detection?

Use keyframe extraction when processing speed is critical and the video already contains sufficient I-frames, such as standard YouTube uploads or cinematic content with natural scene cuts. Use scene-detection when working with screen recordings, slide presentations, or videos with long GOPs where important visual changes might not align with encoder keyframes.

Why does scene-detection take longer to process?

Scene-detection requires ffmpeg to decode every frame in the video to calculate the scene metric—the difference between consecutive frames. In contrast, keyframe extraction uses -skip_frame nokey to decode only I-frames, allowing ffmpeg to skip the vast majority of frames. The full decode necessary for scene analysis consumes significantly more CPU cycles.

What happens if a video has no scene changes?

If the scene detector finds fewer than SCENE_MIN_FRAMES (8) cuts, extract_scene_or_uniform automatically falls back to the uniform sampler (extract), which samples frames at a fixed FPS rate. Similarly, if keyframe extraction yields fewer than KEYFRAME_MIN (4) frames, it also falls back to uniform sampling to ensure adequate frame coverage.

How does the deduplication step work?

After extraction, both engines pass frames through dedupe_perceptual, which resizes each JPEG to a 16×16 pixel thumbnail and compares it to the previous frame using mean absolute pixel difference. If the difference is ≤ DEDUP_THRESHOLD (2.0), the frame is discarded as a near-duplicate. This prevents over-representation of static or minimally changing content in the final frame set.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}