Keyframe Extraction Strategy in Claude-Video: How It Works and When to Use Uniform Sampling Fallback

Claude-Video extracts visual keyframes by decoding only I-frames with FFmpeg, then falls back to uniform sampling when fewer than 4 keyframes are detected.

The bradautomates/claude-video repository implements a sophisticated three-stage pipeline for extracting representative frames from video files. This keyframe extraction strategy prioritizes efficiency by targeting scene-change frames (I-frames) before optionally deduplicating and capping results. When source material contains too few natural keyframes, the system automatically switches to a uniform sampling approach to guarantee coverage.

Three-Stage Keyframe Extraction Pipeline

The core logic resides in skills/watch/scripts/frames.py, where the extract_keyframes() function orchestrates a multi-phase extraction process designed to minimize processing while maximizing visual diversity.

Stage 1: I-Frame Decoding with FFmpeg

The engine begins with fast keyframe decoding, using FFmpeg’s ‑skip_frame nokey flag to extract only I-frames (keyframes) that encoders insert at scene boundaries.

cmd += [
    "-skip_frame", "nokey",               # Decode I-frames only

    "-i", str(Path(video_path).resolve()),
    "-vf", f"{_scale_filter(resolution)},showinfo",
    "-vsync", "vfr",
    "-q:v", "4",
    output_pattern,
]

This approach avoids decoding full video streams, significantly reducing CPU and I/O overhead while capturing natural scene transitions. (Source: lines 602‑613).

Stage 2: Perceptual Deduplication

After extraction, candidate frames pass through dedupe_perceptual() to eliminate near-identical shots. The function generates tiny grayscale thumbnails and greedily removes frames where the mean-pixel difference falls below DEDUP_THRESHOLD (2.0).

This preserves the first and last frames of each distinct shot while removing redundant visual data. (Source: lines 664‑677).

Stage 3: Even-Sampling and Frame Capping

When max_frames limits the output budget, the pipeline applies _even_sample() to distribute selections uniformly across the timeline. This guarantees retention of the first and last frames while spacing intermediate selections evenly. (Source: lines 740‑756).

When Does Claude-Video Fall Back to Uniform Sampling?

The keyframe extraction strategy includes a critical fallback mechanism triggered when the initial I-frame harvest yields insufficient coverage.

The KEYFRAME_MIN Threshold

The system defines a minimum viable threshold of 4 keyframes (KEYFRAME_MIN = 4). If FFmpeg’s I-frame extraction returns fewer than 4 candidates, the engine considers the keyframe set "too sparse" for meaningful analysis.

This commonly occurs with:

• Very short video clips
• Content encoded with low keyframe intervals
• Static scenes with minimal scene changes

The Fallback Code Path

When the threshold triggers, the system discards partial keyframe results and invokes the generic uniform-sampling path:

if len(candidates) < KEYFRAME_MIN:          # Fallback trigger

    # Compute fps for the full range

    frames_out = extract(
        video_path, out_dir,
        fps=fps, resolution=resolution,
        max_frames=budget,
        start_seconds=start_seconds, end_seconds=end_seconds,
    )

The fallback operates across the same time window but uses an auto-calculated FPS (auto_fps) to generate a uniform frame grid respecting the max_frames budget. (Source: lines 636‑648).

Implementation Details and Configuration

Several configuration parameters in skills/watch/scripts/config.py control extraction behavior:

• KEYFRAME_MIN: Set to 4, determines the minimum I-frame count before triggering uniform fallback
• DEDUP_THRESHOLD: Set to 2.0, controls perceptual similarity threshold for duplicate removal
• resolution: Target scaling applied during extraction to balance quality and storage

The extract_keyframes() function returns a tuple containing the frame paths and metadata dictionary, including an "engine" key indicating whether "keyframe" or "uniform" processing was used.

Practical Code Examples

Basic Keyframe Extraction

Extract up to 50 keyframes with automatic deduplication:

from pathlib import Path
from skills.watch.scripts.frames import extract_keyframes

frames, meta = extract_keyframes(
    video_path="example.mp4",
    out_dir=Path("out/keyframes"),
    resolution=512,
    max_frames=50,
)

print(meta["engine"])          # → "keyframe" (or "uniform" on fallback)

print(len(frames))             # Number of frames actually saved

Raw Keyframes Without Deduplication

Force the engine to skip perceptual deduplication:

frames, meta = extract_keyframes(
    video_path="example.mp4",
    out_dir=Path("out/raw_keyframes"),
    dedup=False,
)

Triggering the Uniform Fallback

Short clips automatically trigger the fallback mechanism:

frames, meta = extract_keyframes(
    video_path="short_clip.mp4",
    out_dir=Path("out/fallback"),
    max_frames=30,
)
print(meta.get("fallback"))    # → True if uniform sampling was used

Summary

• Primary strategy: Decode only I-frames using ffmpeg -skip_frame nokey for efficient scene-change detection
• Deduplication: Perceptual thumbnail comparison removes near-duplicates using a threshold of 2.0 mean-pixel difference
• Capping: Even-sampling ensures first and last frames are preserved when reducing to max_frames budget
• Fallback trigger: When fewer than 4 keyframes are detected (KEYFRAME_MIN), the system switches to uniform FPS-based sampling
• Source location: All logic implemented in skills/watch/scripts/frames.py with configuration in config.py

Frequently Asked Questions

What is the difference between keyframe extraction and uniform sampling in Claude-Video?

Keyframe extraction targets I-frames (keyframes) that represent scene changes, making it efficient for videos with distinct shots. Uniform sampling extracts frames at regular time intervals regardless of content, ensuring coverage when videos lack natural scene breaks or contain fewer than 4 keyframes.

Why does Claude-Video fall back to uniform sampling instead of using more keyframes?

The fallback activates when the initial I-frame scan detects fewer than KEYFRAME_MIN (4) frames, indicating the video either lacks scene diversity or uses encoding settings with infrequent keyframes. Uniform sampling guarantees a minimum viable visual representation across the timeline rather than returning an insufficient sparse set.

How can I force Claude-Video to use uniform sampling instead of keyframe extraction?

While the API prioritizes keyframe extraction automatically, you can indirectly force uniform sampling by providing videos with very short durations or by using the lower-level extract() function directly from skills/watch/scripts/frames.py with an explicit fps parameter, bypassing the extract_keyframes() wrapper entirely.

What file contains the deduplication logic for extracted frames?

The perceptual deduplication algorithm resides in skills/watch/scripts/frames.py within the dedupe_perceptual() function (lines 664‑677). It compares grayscale thumbnails and removes frames with mean differences below the DEDUP_THRESHOLD of 2.0.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}