extract_keyframes vs extract_scene_or_uniform in Claude Video: Efficient vs Balanced Frame Extraction

In the bradautomates/claude-video repository, extract_keyframes provides near-instant extraction by sampling only I-frames using ffmpeg's -skip_frame nokey, while extract_scene_or_uniform performs computationally expensive scene-cut detection across the entire video stream to identify meaningful transitions, falling back to uniform sampling for static content.

The watch skill in bradautomates/claude-video provides two primary video frame extraction strategies that trade speed against semantic coverage. Understanding the architectural differences between these functions allows you to optimize the --detail flag for your specific latency and accuracy requirements.

How the Two Functions Differ Internally

Both functions reside in skills/watch/scripts/frames.py and serve different operational tiers of the video processing pipeline. Their implementation reveals a fundamental trade-off between decoder efficiency and visual comprehension.

extract_keyframes: Fast Keyframe-Only Extraction

The extract_keyframes function (defined at line 576 in skills/watch/scripts/frames.py) optimizes for speed by leveraging existing encoder metadata rather than analyzing visual content.

The implementation follows this sequence:

1. Invokes ffmpeg with the -skip_frame nokey flag, instructing the decoder to reconstruct only I-frames (keyframes) that the encoder previously inserted at scene boundaries.
2. Parses ffmpeg's showinfo output to collect precise timestamps for each keyframe.
3. Checks against KEYFRAME_MIN: if the clip contains fewer keyframes than this threshold, it falls back to a uniform sampler that performs full decoding.
4. De-duplicates near-identical frames when dedup=True, then evenly samples the remaining candidates to meet max_frames (defaulting to 50 for efficient mode).

This path never decodes every frame, resulting in processing times of approximately 0.5 seconds per clip, making it the fastest extraction tier.

extract_scene_or_uniform: Scene-Aware Detection with Fallback

The extract_scene_or_uniform function (defined at line 510 in skills/watch/scripts/frames.py) prioritizes semantic coverage by analyzing actual visual changes across the complete video stream.

The algorithm executes as follows:

1. Runs extract_scene_candidates (a scene-cut detector) over the entire decoded clip, identifying genuine visual transitions rather than relying on encoder keyframes.
2. Validates against SCENE_MIN_FRAMES: if detected cuts exceed this threshold, de-duplicates frames and evenly samples to the configured max_frames (defaulting to 100 for balanced mode).
3. Falls back to extract (uniform sampling at calculated fps) when the video lacks sufficient scene changes, ensuring coverage for static content.
4. Returns metadata indicating "scene" when cuts are detected or "uniform" when falling back.

Because this function must decode every frame to detect cuts, it incurs significantly higher computational costs than the keyframe-only approach, but provides semantically meaningful frame selection for most video content.

Performance and Coverage Comparison

The watch skill maps these functions to CLI detail flags in skills/watch/scripts/watch.py (lines 198–208):

Detail Flag	Function Called	Default Frame Cap	Decode Strategy	Typical Use Case
efficient	extract_keyframes	50	Keyframe-only (I-frames)	High-volume processing, speed-critical
balanced	extract_scene_or_uniform	100	Full decode with scene detection	General analysis, semantic accuracy

Key distinction: The efficient path relies on encoder-inserted keyframes, which may not align perfectly with actual scene changes if the encoder placed I-frames for structural rather than visual reasons. The balanced path performs content-aware analysis, ensuring frames capture genuine visual transitions even when keyframes are sparse.

Practical Code Examples

Efficient Mode Implementation

from pathlib import Path
from skills.watch.scripts.frames import extract_keyframes

# Fast extraction using keyframe-only sampling (≤50 frames)

frames, meta = extract_keyframes(
    video_path="interview_clip.mp4",
    out_dir=Path("/tmp/frames"),
    max_frames=50,      # Default cap for efficient mode

    dedup=True,
)

print(meta["engine"])   # Outputs: "keyframe"

print(len(frames))      # ≤ 50 frames

Balanced Mode Implementation

from pathlib import Path
from skills.watch.scripts.frames import extract_scene_or_uniform

# Scene-aware extraction with uniform fallback (≤100 frames)

frames, meta = extract_scene_or_uniform(
    video_path="interview_clip.mp4",
    out_dir=Path("/tmp/frames"),
    target_frames=100,  # Requested frame count

    max_frames=100,     # Hard cap for balanced mode

    dedup=True,
)

print(meta["engine"])   # Outputs: "scene" or "uniform" if fallback triggered

print(len(frames))      # ≤ 100 frames

Summary

• extract_keyframes provides speed by extracting only existing I-frames via ffmpeg's -skip_frame nokey, processing clips in ~0.5 seconds with a default cap of 50 frames.
• extract_scene_or_uniform provides accuracy by decoding the full stream to detect genuine scene cuts, supporting up to 100 frames with intelligent fallback to uniform sampling for static videos.
• Both functions support de-duplication and even-sampling to meet frame caps while avoiding redundant visual information.
• The choice between efficient and balanced modes in skills/watch/scripts/watch.py determines whether your pipeline prioritizes latency or semantic coverage.

Frequently Asked Questions

When should I use efficient mode versus balanced mode in Claude Video?

Use efficient mode (extract_keyframes) when processing large video archives where latency matters more than precise scene alignment, as it completes in under a second by reading only pre-encoded I-frames. Use balanced mode (extract_scene_or_uniform) when analyzing content where missing a scene transition would reduce comprehension, such as educational videos or dialogue-heavy content, accepting the full-decode performance cost for better semantic coverage.

Why does balanced mode require full video decoding while efficient mode does not?

The extract_scene_or_uniform function must decode every frame to calculate visual differences between consecutive images and identify genuine scene cuts, whereas extract_keyframes relies on the video encoder's existing I-frame metadata (keyframes) which are accessible without full reconstruction of intermediate frames. This architectural difference means balanced mode performs content analysis while efficient mode performs metadata reading.

What happens if a video has no scene cuts or very few keyframes?

Both functions implement fallback safeguards: extract_keyframes falls back to uniform sampling if fewer than KEYFRAME_MIN keyframes are detected, while extract_scene_or_uniform falls back to uniform fps-based extraction when fewer than SCENE_MIN_FRAMES cuts are found. These fallbacks ensure you receive the requested number of frames even for static content, though the fallback extraction method varies by function.

How do I adjust the frame extraction limits for custom requirements?

Both functions accept a max_frames parameter (defaulting to 50 for efficient and 100 for balanced) and dedup boolean for near-duplicate removal. When using the CLI in watch.py, these defaults are hardcoded based on the --detail flag selected, but you can modify the Python API calls directly to set custom caps such as max_frames=200 for high-detail analysis or reduce to max_frames=10 for thumbnail generation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}