Claude-Video Keyframe Extraction Method: How It Works and When Uniform Sampling Kicks In

Claude-Video extracts keyframes using FFmpeg's -skip_frame nokey filter to decode only I-frames, but falls back to uniform sampling when fewer than 4 keyframes are found in a video segment.

The bradautomates/claude-video repository implements a hybrid frame extraction strategy that prioritizes keyframes for efficiency, with an automatic fallback to uniform temporal sampling when keyframe density is insufficient. This article breaks down the implementation in skills/watch/scripts/frames.py and explains the exact conditions triggering each mode.

Keyframe Extraction Method in Claude-Video

The core keyframe extraction logic lives in the extract_keyframes function found at skills/watch/scripts/frames.py (lines 776-814). This function implements a sophisticated two-phase approach optimized for video analysis workloads.

Phase 1: Keyframe-Only Decode with FFmpeg

Claude-Video leverages FFmpeg's frame-skipping capabilities to avoid decoding non-keyframes entirely. The command construction uses -skip_frame nokey, which instructs the decoder to discard all frames except I-frames (keyframes):

cmd = [
    "ffmpeg", "-hide_banner", "-loglevel", "info", "-y",
    "-skip_frame", "nokey",
    "-i", str(Path(video_path).resolve()),
    "-vf", f"{_scale_filter(resolution)},showinfo",
    "-vsync", "vfr", "-q:v", "4", output_pattern,
]

This approach provides significant performance advantages:

• Reduced decode overhead: Only keyframes are processed, skipping P-frames and B-frames
• Lower memory footprint: Fewer frames enter the processing pipeline
• Deterministic output: Keyframe positions are encoding-dependent but reproducible

The showinfo filter in the video filter (-vf) chain outputs per-frame metadata to stderr, which the Python code parses to extract frame timestamps (lines 825-834).

Timestamp Collection and Candidate Building

After FFmpeg execution, the function parses the showinfo output to build a list of candidate frames. Each candidate receives metadata marking its extraction reason:

# Conceptual representation of candidate structure

candidate = {
    "path": frame_path,
    "timestamp": parsed_timestamp,
    "reason": "keyframe"  # Always "keyframe" in this phase

}

When Claude-Video Uses Uniform Sampling

The keyframe extraction method includes a mandatory minimum threshold that triggers automatic fallback behavior. This is the critical decision point in the extraction pipeline.

The KEYFRAME_MIN Threshold

At lines 836-868, the function evaluates whether the keyframe-only extraction yielded sufficient results:

Condition	Action
len(candidates) >= KEYFRAME_MIN (4)	Proceed with keyframe set
len(candidates) < KEYFRAME_MIN	Discard candidates, trigger uniform fallback

The constant KEYFRAME_MIN = 4 represents a hardcoded minimum viable keyframe count. When this threshold is unmet, the function assumes the video segment lacks adequate keyframe density for meaningful analysis.

Uniform Sampling Fallback Implementation

If fallback triggers, extract_keyframes performs these steps (lines 836-868):

1. Cleanup: Deletes all partially-extracted keyframe candidates
2. Duration calculation: Computes the time range from start_seconds to end_seconds
3. Automatic FPS calculation: Calls auto_fps() to determine an even sampling rate based on duration and max_frames
4. Uniform extraction: Invokes the standard extract() function with calculated parameters

The auto_fps helper calculates sampling frequency to distribute frames evenly across the video duration while respecting the max_frames cap.

Complete Extraction Workflow

Understanding the full pipeline clarifies how keyframe and uniform methods integrate:

┌─────────────────┐     ┌─────────────────────┐     ┌─────────────────┐
│  extract_       │────▶│  FFmpeg -skip_frame │────▶│  Parse showinfo │
│  keyframes()    │     │  nokey              │     │  timestamps     │
└─────────────────┘     └─────────────────────┘     └────────┬────────┘
                                                             │
                              ┌────────────────────────────┘
                              ▼
                    ┌─────────────────┐
                    │  Candidates >=4? │
                    └────────┬────────┘
                             │
              ┌──────────────┼──────────────┐
              ▼ YES                         ▼ NO (uniform sampling)
    ┌─────────────────┐            ┌─────────────────┐
    │  Deduplication  │            │  Delete partial │
    │  (optional)     │            │  keyframes      │
    └────────┬────────┘            └────────┬────────┘
             │                              │
             │                    ┌─────────┴─────────┐
             │                    │  Calculate        │
             │                    │  auto_fps from    │
             │                    │  duration         │
             │                    └─────────┬─────────┘
             │                              │
             │                    ┌─────────▼─────────┐
             │                    │  extract() with   │
             │                    │  uniform FPS      │
             │                    └─────────┬─────────┘
             │                              │
             └──────────────┬───────────────┘
                            ▼
                   ┌─────────────────┐
                   │  Even-sampling  │
                   │  to max_frames  │
                   └─────────────────┘

Practical Code Examples

Standard Keyframe Extraction with Fallback

This call attempts keyframe extraction and automatically falls back to uniform sampling if needed:

from pathlib import Path
from skills.watch.scripts import frames

video_path = "example.mp4"
out_dir = Path("frames_out")

frames_out, meta = frames.extract_keyframes(
    video_path,
    out_dir,
    resolution=512,      # Max dimension for scaling

    max_frames=50,       # Hard cap on returned frames

    start_seconds=10.0,  # Optional: segment start

    end_seconds=40.0,    # Optional: segment end

    dedup=True,          # Enable perceptual deduplication

)

print(meta)

# Output indicates which engine succeeded: "keyframe" or "uniform"

Manual Uniform Sampling

To bypass keyframe extraction entirely, use the extract function directly with calculated FPS:

# For a 30-second segment, sample 30 frames (1 fps)

fps, _ = frames.auto_fps(duration_seconds=30, max_frames=30)

uniform_frames = frames.extract(
    video_path,
    out_dir,
    fps=fps,
    resolution=512,
    max_frames=30,
)

Performance and Quality Considerations

Aspect	Keyframe Mode	Uniform Mode
Speed	Faster (decodes ~1-5% of frames)	Slower (decodes all sampled frames)
Temporal coverage	Irregular, encoding-dependent	Perfectly even
Best for	Long videos, standard encodings	Short clips, synthetic/cg content
Memory	Lower peak usage	Higher proportional to FPS

The fallback to uniform sampling addresses edge cases where video encoding uses:

• All-intra encoding (already uniform, but may have few keyframes)
• Very short segments (< few seconds)
• Non-standard GOP structures (extremely long keyframe intervals)

Key Source Files in Claude-Video

• skills/watch/scripts/frames.py: Core implementation of extract_keyframes, uniform fallback, auto_fps, and deduplication
• skills/watch/scripts/watch.py: Orchestrates the watch skill, calls extract_keyframes with user parameters
• tests/test_frames.py: Validates keyframe extraction and fallback behavior
• tests/test_dedup.py: Tests perceptual duplicate removal applied post-extraction

Summary

• Claude-Video's keyframe extraction method uses FFmpeg's -skip_frame nokey to decode only I-frames, minimizing processing overhead
• Uniform sampling triggers automatically when fewer than 4 keyframes (KEYFRAME_MIN) are found in the target video segment
• The fallback occurs at lines 836-868 in skills/watch/scripts/frames.py, where candidates are discarded and extract() is called with calculated FPS
• Both paths ultimately respect max_frames through even sampling and support optional perceptual deduplication
• This hybrid approach balances efficiency for typical content with reliability for edge-case encodings

Frequently Asked Questions

What is the minimum keyframe threshold in Claude-Video?

The KEYFRAME_MIN constant is set to 4 keyframes. If a video segment yields 3 or fewer keyframes, the extraction automatically switches to uniform sampling mode.

How does Claude-Video calculate the FPS for uniform fallback?

The auto_fps function computes sampling frequency by dividing the video segment duration by the max_frames parameter, ensuring even temporal distribution while respecting the frame cap.

Can I force uniform sampling even if enough keyframes exist?

Yes. Call frames.extract() directly with an explicit fps parameter instead of using extract_keyframes(). This bypasses the keyframe engine entirely.

Why would a video have fewer than 4 keyframes?

Common causes include: very short clips (under 1-2 seconds), all-intra codecs where every frame is technically a keyframe (confusing the detection), or videos with intentionally long GOP intervals (e.g., 10+ seconds between I-frames).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}