How the Keyframe Engine Works for Frame Extraction in Claude-Video

The keyframe engine in claude-video uses ffmpeg's -skip_frame nokey flag to extract only I-frames (keyframes), providing a fast, low-cost sampling method that falls back to uniform extraction if fewer than 4 keyframes are found.

The keyframe engine is the fastest tier for frame extraction in the bradautomates/claude-video repository. Located in skills/watch/scripts/frames.py, this engine leverages the fact that video encoders insert keyframes (I-frames) at scene changes, allowing near-instant sampling without decoding every picture. By extracting only these distinct moments, the system minimizes computational overhead while preserving visual diversity.

Core Architecture of the Keyframe Engine

Fast I-Frame Extraction with ffmpeg

The main entry point is extract_keyframes, which constructs an ffmpeg command targeting only keyframes. The command includes the -skip_frame nokey flag, instructing ffmpeg to skip all P/B frames and output only I-frames.

The implementation also applies the showinfo video filter (lines 100-118 in skills/watch/scripts/frames.py), causing ffmpeg to print timestamps for each decoded frame to stderr. This output capture enables precise temporal indexing without additional file I/O operations.

Timestamp Parsing and Candidate Generation

After ffmpeg execution completes, the function parses the captured stderr output using the compiled regex pattern SHOWINFO_TS_RE (defined at line 39). The parsing loop extracts timestamps and assembles a list of candidate frames, where each entry contains:

• Frame index
• Timestamp in seconds
• File path to the extracted JPEG
• Extraction reason (set to "keyframe")

This candidate list serves as the foundation for subsequent deduplication and sampling stages.

Fallback and Coverage Guarantees

If the video yields fewer than KEYFRAME_MIN (4) keyframes, the engine automatically falls back to the uniform extractor (extract). This threshold ensures that short clips or poorly encoded videos still provide adequate visual coverage.

The fallback sequence (lines 136-161) executes three steps:

1. Metadata retrieval – Calls get_metadata to determine total video duration
2. Frame rate calculation – Uses auto_fps (lines 22-38) to compute an appropriate sampling rate that respects the user-provided max_frames budget
3. Uniform extraction – Runs the uniform extractor, followed by optional perceptual deduplication via dedupe_perceptual

This graceful degradation guarantees that callers always receive a useful frame set regardless of video encoding characteristics.

Post-Processing Pipeline

When sufficient keyframes are present, the engine proceeds through deduplication and capping stages to refine the candidate set.

Perceptual Deduplication

The dedupe_perceptual function (line 64) eliminates near-identical frames that may occur during static scenes. The process generates 16×16 grayscale thumbnails for each candidate using _thumb_frames, then calculates the mean-absolute-per-pixel difference between consecutive frames.

Frames with a difference ≤ DEDUP_THRESHOLD (2.0) are considered duplicates. The function deletes the corresponding JPEG files on-the-fly (lines 63-78) and removes them from the candidate list, ensuring only visually distinct frames survive.

Even Sampling and Frame Capping

Even after deduplication, the candidate count may exceed the user's max_frames limit. The _even_sample helper (lines 83-92) selects n evenly-spaced frames while always preserving the first and last frames to maintain temporal boundaries.

This function deletes unused JPEG files and re-indexes the survivors, producing a final list that respects the frame budget without clustering selections in specific video segments.

Implementation Details and Code Examples

The following Python example demonstrates direct usage of the keyframe engine:

from pathlib import Path
from skills.watch.scripts.frames import extract_keyframes

video = "example.mp4"
out_dir = Path("./frames")
frames, meta = extract_keyframes(
    video_path=video,
    out_dir=out_dir,
    resolution=512,
    max_frames=30,      # cap to 30 frames

    start_seconds=None,
    end_seconds=None,
    dedup=True,
)

print("Engine:", meta["engine"])
print("Selected frames:", len(frames))
print(frames[0])   # first frame dict

The engine also exposes a command-line interface:

python -m skills.watch.scripts.frames \
    example.mp4 ./frames \
    --max-frames 30 \
    --resolution 512

Both interfaces return a tuple (selected_frames, meta) where meta contains:

• "engine": "keyframe" (or "uniform" if fallback occurred)
• Candidate count, deduplication count, and final selection count
• Boolean flag indicating whether fallback extraction was triggered

Summary

• The keyframe engine resides in skills/watch/scripts/frames.py and uses extract_keyframes as its primary entry point.
• It leverages ffmpeg's -skip_frame nokey flag to decode only I-frames, parsing timestamps via SHOWINFO_TS_RE.
• If fewer than 4 keyframes are detected, the system automatically falls back to uniform extraction using auto_fps for frame rate calculation.
• Perceptual deduplication compares 16×16 thumbnails and removes frames below a difference threshold of 2.0.
• _even_sample ensures the final output respects max_frames while preserving temporal coverage across the video duration.

Frequently Asked Questions

What makes the keyframe engine faster than uniform frame extraction?

The keyframe engine avoids decoding intermediate P-frames and B-frames by passing -skip_frame nokey to ffmpeg. Because I-frames represent complete images stored independently in the video stream, ffmpeg can locate and extract them without processing intervening delta frames, reducing CPU usage and I/O operations significantly compared to decoding the entire video.

How does the engine handle videos with few scene changes?

When a video contains fewer than KEYFRAME_MIN (4) keyframes—common in static recordings or slideshows—the engine triggers a fallback to uniform extraction. It calculates an appropriate frames-per-second rate using auto_fps to distribute samples evenly across the duration, ensuring the user still receives the requested number of frames up to max_frames.

What is the deduplication threshold and how does it work?

The DEDUP_THRESHOLD is set to 2.0, representing the maximum mean-absolute-per-pixel difference between 16×16 grayscale thumbnails. If two consecutive frames differ by 2.0 or less, they are considered visually identical, and the duplicate is deleted immediately. This threshold balances removal of static frames against preservation of subtle motion.

Can I disable deduplication when using the keyframe engine?

Yes. Both the Python API and CLI support disabling deduplication by setting dedup=False (Python) or omitting the deduplication flag. When disabled, the engine skips the dedupe_perceptual call and proceeds directly to even sampling, preserving all extracted keyframes regardless of visual similarity.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}