Scene Detection vs Keyframe Extraction in Claude Video: How They Differ and When to Use Each

Keyframe extraction decodes only encoder-flagged I-frames for speed, while scene detection performs a full video decode to catch perceptual changes using ffmpeg’s scene filter, trading performance for higher accuracy.

The bradautomates/claude-video repository offers two independent engines for converting video into representative frames before transcription. While both methods populate the frames.py pipeline, they differ in underlying ffmpeg commands, computational cost, and their ability to detect subtle visual changes. Choosing between scene detection and keyframe extraction depends on whether your priority is processing speed or capturing every visual transition.

Core Technical Differences

What Keyframe Extraction Captures

Keyframe extraction leverages the encoder’s existing I-frame markers by passing -skip_frame nokey to ffmpeg. This instructs the decoder to skip all non-keyframes, processing only frames that the video encoder already flagged as complete reference frames. According to the source in skills/watch/scripts/frames.py, these I-frames usually correspond to scene cuts but can also be inserted for bitrate management reasons, potentially missing subtle transitions in videos with long GOP structures.

What Scene Detection Captures

Scene detection uses ffmpeg’s select='eq(n\\,0)+gt(scene\\,THRESH)' filter to evaluate every frame’s visual difference from its predecessor. When the calculated scene score exceeds SCENE_THRESHOLD (default 0.20), the frame is retained. This method catches any perceptible change—including slide transitions in screen recordings or subtle lighting shifts—even when the encoder did not insert a keyframe at that boundary.

Performance and Cost Comparison

Keyframe extraction is computationally cheap because ffmpeg skips the majority of frames during decode. Scene detection requires a full decode of the entire video to evaluate the scene-change filter, consuming significantly more CPU resources. Both methods enforce minimum thresholds: if extract_keyframes finds fewer than KEYFRAME_MIN (4) frames, or if scene detection finds fewer than SCENE_MIN_FRAMES (8) cuts, the pipeline automatically falls back to uniform sampling via the extract function.

Implementation Details in frames.py

Keyframe Extraction with extract_keyframes

Located in skills/watch/scripts/frames.py, the extract_keyframes function builds an ffmpeg command that applies the -skip_frame nokey flag. The pipeline parses ffmpeg’s showinfo filter output to collect timestamps of the decoded I-frames. If the resulting list contains fewer than 4 frames, the function discards the candidates and invokes the uniform extractor (extract) as a fallback, ensuring the transcription pipeline always receives sufficient visual context.

Scene Detection with extract_scene_candidates

The extract_scene_candidates function implements the perceptual detection logic. It always includes the first frame (eq(n\\,0)) and uses the scene filter to identify cuts where visual difference exceeds the threshold. The caller, extract_scene_or_uniform, validates the count against SCENE_MIN_FRAMES (8). When sufficient cuts exist, the pipeline proceeds to dedupe_perceptual before down-sampling to the requested max_frames.

Shared Deduplication and Fallback Logic

Both engines share common post-processing steps defined in the same module. After extraction, frames pass through dedupe_perceptual, which thumbnails each JPEG to 16×16 pixels and drops frames whose mean absolute pixel difference is ≤ 2.0 (DEDUP_THRESHOLD). This prevents near-identical visual content from over-representing static scenes. The uniform fallback (extract) simply requests a fixed FPS and writes JPEGs, serving as the safety net when both specialized engines encounter insufficient variation.

Practical Usage Examples

Fast Keyframe Extraction

Use this when processing well-keyframed content like typical YouTube uploads where speed matters more than catching every visual nuance.

from pathlib import Path
from skills.watch.scripts import frames

video = "sample.mp4"
out_dir = Path("keyframes")

frames_key, meta_key = frames.extract_keyframes(
    video_path=video,
    out_dir=out_dir,
    resolution=512,
    max_frames=30,
    start_seconds=None,
    end_seconds=None,
    dedup=True,
)
print("Engine:", meta_key["engine"])  # "keyframe"

print("Frames:", len(frames_key))

Accurate Scene Detection

Use this for screen captures, presentations, or videos with long GOPs where visual changes might not align with encoder keyframes.

out_dir = Path("scenes")

frames_scene, meta_scene = frames.extract_scene_or_uniform(
    video_path=video,
    out_dir=out_dir,
    fps=1.5,                # Ignored when scene detection succeeds

    target_frames=30,
    resolution=512,
    max_frames=30,
    start_seconds=None,
    end_seconds=None,
    dedup=True,
)
print("Engine:", meta_scene["engine"])  # "scene" or "uniform"

print("Frames:", len(frames_scene))

Uniform Fallback (Automatic)

This runs automatically when the primary engines fail to meet minimum frame thresholds, ensuring robustness across all video types.

out_dir = Path("uniform")

frames_uni, meta_uni = frames.extract(
    video_path=video,
    out_dir=out_dir,
    fps=1.0,
    resolution=512,
    max_frames=30,
    start_seconds=None,
    end_seconds=None,
)
print("Engine:", meta_uni["engine"])  # "uniform"

Summary

• Keyframe extraction uses -skip_frame nokey to cheaply decode only I-frames, falling back to uniform sampling if fewer than 4 keyframes are found.
• Scene detection uses ffmpeg’s scene filter with a default threshold of 0.20 to catch perceptual changes during a full decode, requiring SCENE_MIN_FRAMES (8) cuts to avoid fallback.
• Both engines apply dedupe_perceptual to remove near-duplicate frames using a 16×16 pixel thumbnail comparison with a threshold of 2.0.
• Keyframe extraction is preferred for speed and typical web video; scene detection is preferred for screen recordings and content with sparse keyframes.

Frequently Asked Questions

When should I use keyframe extraction over scene detection?

Use keyframe extraction when processing speed is critical and the video source uses standard encoding with frequent I-frames, such as most YouTube uploads or streaming content. It is significantly cheaper because it skips non-keyframes during decode. Use scene detection when working with screen recordings, presentation slides, or videos using long GOPs where important visual changes might occur between encoded keyframes.

Why does scene detection take longer to process?

Scene detection requires a full decode of every video frame to calculate the scene filter value in ffmpeg, comparing each frame against its predecessor to detect perceptual differences. In contrast, keyframe extraction uses -skip_frame nokey to decode only the pre-flagged I-frames, drastically reducing CPU cycles. The trade-off is accuracy: scene detection catches subtle visual transitions that keyframe extraction might miss.

What happens if a video has no scene cuts or keyframes?

If extract_keyframes finds fewer than KEYFRAME_MIN (4) frames, or if scene detection identifies fewer than SCENE_MIN_FRAMES (8) cuts, Claude Video automatically invokes the uniform sampling engine (extract). This fallback extracts frames at a fixed FPS (typically 1.0–2.0) to ensure the transcription pipeline receives the minimum required visual context regardless of the video’s encoding characteristics.

How does Claude Video handle duplicate frames?

After both extraction engines, frames pass through dedupe_perceptual in frames.py. This function creates 16×16 pixel thumbnails of each JPEG and calculates the mean absolute pixel difference between consecutive frames. If the difference is ≤ 2.0 (DEDUP_THRESHOLD), the frame is discarded as a near-duplicate. This deduplication step runs by default but can be disabled by setting dedup=False in the extraction calls.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}