Keyframe Extraction vs Scene Detection in Claude Video: What's the Difference and When to Use Each

Keyframe extraction relies on decoder-embedded I-frames for speed, while scene detection performs a full video decode using ffmpeg's scene filter to catch every visual change—each with distinct performance trade-offs and fallback behaviors.

Claude Video offers two independent engines for converting videos into representative frames prior to transcription. Understanding the difference between keyframe extraction and scene detection helps you choose the right approach for your video type and performance requirements. This article examines the implementation in bradautomates/claude-video, tracing the actual source code in skills/watch/scripts/frames.py to explain how each engine works, when it falls back, and how to configure it.

How Keyframe Extraction Works

The keyframe extraction engine is the faster, encoder-driven option. It leverages the fact that video encoders already mark certain frames as "keyframes" (I-frames) that can be decoded without reference to other frames.

Underlying FFmpeg Command

In extract_keyframes (lines 45-90 of frames.py), the code constructs an ffmpeg command using -skip_frame nokey:

ffmpeg -skip_frame nokey -i input.mp4 -vf "showinfo,scale=512:-1" ...

This flag instructs ffmpeg to skip every non-keyframe during decoding. The decoder only touches I-frames, dramatically reducing CPU work.

What Keyframes Actually Capture

Keyframes are inserted by the encoder for two reasons:

• Scene cuts — when visual content changes significantly
• Bitrate management — to reset the prediction chain and control file size

This means keyframe extraction often captures scene boundaries, but misses subtle visual changes that didn't trigger a keyframe insertion. Videos with long GOP (Group of Pictures) structures may have sparse keyframes despite frequent visual changes.

Fallback Behavior

If the keyframe count falls below KEYFRAME_MIN (4 frames), the engine abandons its results and calls extract() for uniform sampling:

# From frames.py lines 78-85

if len(frames) < KEYFRAME_MIN:
    logger.warning(f"Only {len(frames)} keyframes found, falling back to uniform sampling")
    return extract(
        video_path=video_path,
        out_dir=out_dir,
        fps=1.0,  # Default uniform rate

        # ... additional parameters

    )

This ensures you always receive usable frames even from static videos.

How Scene Detection Works

The scene detection engine prioritizes recall over speed. It analyzes every frame to detect perceptual changes, regardless of encoder keyframe placement.

Underlying FFmpeg Command

In extract_scene_candidates (lines 115-165), the engine uses ffmpeg's select filter with the scene detection expression:

ffmpeg -i input.mp4 -vf "select='eq(n\,0)+gt(scene\,0.20)',showinfo,scale=512:-1" ...

The filter gt(scene\,THRESH) outputs frames where the scene-change metric exceeds the threshold (default 0.20). The eq(n\,0) term always includes the first frame.

Full Decode Requirement

Unlike keyframe extraction, this approach decode every frame to compute inter-frame differences. The scene filter analyzes pixel-level changes between consecutive frames, which requires complete video decoding.

This makes scene detection significantly more expensive for long or high-resolution videos, but it catches:

• Slide transitions in presentations
• Screen recording changes without keyframes
• Subtle visual shifts that encoders optimize away

Fallback and Post-Processing

The extract_scene_or_uniform wrapper (lines 168-210) validates results against SCENE_MIN_FRAMES (8 cuts minimum):

# From frames.py lines 185-195

if len(scene_frames) < SCENE_MIN_FRAMES:
    logger.info(f"Scene detection found {len(scene_frames)} frames, using uniform sampling")
    return extract(
        video_path=video_path,
        out_dir=out_dir,
        fps=fps,
        # ... additional parameters

    )

When sufficient scenes exist, the pipeline applies dedupe_perceptual() to remove near-identical frames, then evenly samples down to max_frames.

Shared Infrastructure: Deduplication and Uniform Fallback

Both engines converge on common utilities for final processing.

Perceptual Deduplication

After any extraction path, frames pass through dedupe_perceptual() (lines 220-275). This function:

1. Resizes each JPEG to 16×16 pixels
2. Computes mean absolute pixel difference between consecutive frames
3. Drops frames where difference ≤ DEDUP_THRESHOLD (2.0)

from skills.watch.scripts import frames

frames, meta = frames.extract_keyframes(
    video_path="talk.mp4",
    out_dir="./frames",
    dedup=True,  # Enable perceptual deduplication (default)

)

Uniform Sampling Fallback

The extract() function (lines 35-70) serves as the reliable baseline. It requests frames at a fixed FPS without scene analysis:

frames, meta = frames.extract(
    video_path="static.mp4",
    out_dir="./frames",
    fps=1.0,        # One frame per second

    max_frames=30,
    resolution=512,
)

Both keyframe and scene engines delegate to this function when their primary method yields insufficient frames.

Configuration Constants

The behavior of both engines is controlled by constants defined at the top of frames.py (lines 19-30):

MAX_FPS = 2.0
SCENE_THRESHOLD = 0.20      # scene-change detection sensitivity (0.0-1.0)

SCENE_MIN_FRAMES = 8        # minimum scene cuts before trusting scene mode

KEYFRAME_MIN = 4            # minimum keyframes before falling back

DEDUP_THRESHOLD = 2.0       # mean pixel difference for deduplication

Practical Code Examples

Fast Keyframe Extraction

Use this for typical web videos with reasonable keyframe density:

from pathlib import Path
from skills.watch.scripts import frames

frames, meta = frames.extract_keyframes(
    video_path="youtube_video.mp4",
    out_dir=Path("keyframes"),
    resolution=512,
    max_frames=30,
    dedup=True,
)
print(f"Engine: {meta['engine']}")  # "keyframe"

print(f"Frames: {len(frames)}")

Precision Scene Detection

Use this for presentations, screen recordings, or videos with long GOPs:

frames, meta = frames.extract_scene_or_uniform(
    video_path="webinar_recording.mp4",
    out_dir=Path("scenes"),
    fps=1.5,                    # Ignored if scene detection succeeds

    target_frames=30,
    resolution=512,
    max_frames=30,
    dedup=True,
)
print(f"Engine: {meta['engine']}")  # "scene" or "uniform"

Direct Uniform Sampling

Force consistent frame spacing without content analysis:

frames, meta = frames.extract(
    video_path="security_footage.mp4",
    out_dir=Path("uniform"),
    fps=0.5,                    # One frame every 2 seconds

    resolution=512,
    max_frames=30,
)
print(f"Engine: {meta['engine']}")  # "uniform"

Key Differences Summary

Aspect	Keyframe Extraction	Scene Detection
Speed	Very fast (decoder skips non-keyframes)	Slower (full decode required)
Accuracy	Encoder-dependent; may miss changes	Catches all perceptual changes
Best for	Standard web videos, fast previews	Presentations, screen recordings, analysis
Fallback trigger	< 4 keyframes found	< 8 scene cuts detected
Unique parameter	None (simple on/off)	threshold (0.0-1.0 sensitivity)

Summary

• Keyframe extraction uses ffmpeg -skip_frame nokey to decode only I-frames, making it the fastest option for well-keyframed videos.
• Scene detection employs ffmpeg select='gt(scene,THRESH)' with full decoding to catch every visual change at higher CPU cost.
• Both engines fall back to uniform fps-based sampling when their primary method yields insufficient frames (KEYFRAME_MIN = 4, SCENE_MIN_FRAMES = 8).
• All paths apply dedupe_perceptual() to remove redundant frames using 16×16 thumbnail comparison.
• Choose keyframes for speed on typical content; choose scene detection when visual completeness matters more than processing time.

Frequently Asked Questions

Can I use both keyframe extraction and scene detection on the same video?

No — these are mutually exclusive engines in Claude Video's current architecture. You call either extract_keyframes() or extract_scene_or_uniform() for a given extraction task. If you need both approaches for comparison, run them as separate operations and merge the results manually.

How do I adjust the sensitivity of scene detection?

Pass the threshold parameter to extract_scene_or_uniform(). The default is 0.20 (defined by SCENE_THRESHOLD in frames.py). Lower values (0.10-0.15) detect more subtle changes; higher values (0.30-0.40) require more dramatic scene cuts. Values outside 0.0-1.0 may produce unpredictable results.

Why does keyframe extraction sometimes fall back to uniform sampling despite visible scene changes?

The encoder may have placed keyframes sparsely for bitrate efficiency, or the video may genuinely lack visual variety. The KEYFRAME_MIN = 4 safeguard ensures you receive usable frames rather than a sparse, potentially unrepresentative set. Uniform sampling guarantees coverage across the video timeline.

Does deduplication run before or after max_frames is applied?

Deduplication (dedupe_perceptual) runs before the final frame count reduction in the scene detection pipeline, but the timing varies by path. For keyframe extraction, deduplication filters the raw keyframes, then results are returned directly. For scene detection, deduplication occurs before max_frames caps the result set through even sampling.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}