How Keyframe Extraction Works with ffmpeg `-skip_frame nokey`

The -skip_frame nokey option instructs ffmpeg to decode only I-frames (keyframes), allowing the extract_keyframes function to rapidly capture scene-cut moments while skipping redundant P- and B-frames.

The bradautomates/claude-video repository implements intelligent video analysis through its watch skill, leveraging ffmpeg's -skip_frame nokey flag for efficient keyframe extraction with -skip_frame nokey. This approach minimizes processing overhead by extracting only the frames that represent distinct scene changes rather than processing every frame in the video.

Core Implementation in extract_keyframes

The heart of the system resides in skills/watch/scripts/frames.py, specifically within the extract_keyframes function (lines 776-822). This function orchestrates a multi-stage pipeline that uses ffmpeg's selective decoding capabilities to isolate keyframes.

Command Construction with -skip_frame nokey

The ffmpeg command explicitly targets I-frames through the -skip_frame nokey parameter, which instructs the decoder to skip all predictive (P) and bidirectional (B) frames. The command array constructed in lines 1006-1014 appears as follows:

cmd = [
    "ffmpeg",
    "-hide_banner",
    "-loglevel", "info",
    "-y",
    # optional seek arguments …

    "-skip_frame", "nokey",                     # <‑‑ only decode I-frames

    "-i", str(Path(video_path).resolve()),
    "-vf", f"{_scale_filter(resolution)},showinfo",  # scale + showinfo filter

    "-vsync", "vfr",
    "-q:v", "4",
    output_pattern,
]

The -skip_frame nokey flag ensures that ffmpeg processes only frames that are complete images without reference to other frames, naturally corresponding to scene cuts and significant visual transitions.

Timestamp Extraction via showinfo

To associate extracted images with their temporal positions, the pipeline utilizes ffmpeg's showinfo video filter. This filter outputs diagnostic lines containing pts_time values for each decoded frame. The function parses these timestamps using the compiled regex SHOWINFO_TS_RE (defined at lines 39-40 as r"pts_time:([0-9.]+)"), enabling precise mapping of each JPEG to its exact position in the video timeline.

Frame Processing and Fallback Logic

Once keyframes are extracted, the system applies several refinement steps to ensure quality and coverage.

Candidate Generation

For each extracted frame, the function constructs a candidate dictionary (lines 1028-1034) containing metadata:

{
    "index": i,
    "timestamp_seconds": ts,
    "path": str(path),
    "reason": "keyframe",
}

Robust Fallback Mechanism

If the video contains fewer than KEYFRAME_MIN (4) keyframes—a common occurrence in static or low-motion content—the system discards the partial results and automatically falls back to uniform-FPS extraction (lines 1036-1050). This fallback uses the standard extract function with an automatically calculated frame rate based on the video's effective duration, ensuring adequate frame coverage regardless of the original encoding structure.

Deduplication and Sampling

When sufficient keyframes exist, the pipeline optionally applies perceptual deduplication via dedupe_perceptual (lines 86-91), which compares thumbnails using mean pixel difference to eliminate near-identical frames. The system then evenly samples the remaining frames to respect the max_frames cap, always preserving the first and last frames to maintain temporal boundaries (lines 74-78).

Practical Usage Examples

Python API Integration

Extract keyframes programmatically using the extract_keyframes function:

from pathlib import Path
from skills.watch.scripts.frames import extract_keyframes

video_path = "example.mp4"
out_dir = Path("tmp/keyframes")
frames, meta = extract_keyframes(
    video_path,
    out_dir,
    resolution=640,
    max_frames=30,          # limit to 30 frames (None = uncapped)

    start_seconds=10,       # optional start offset

    end_seconds=70,         # optional end offset

    dedup=True,             # drop near-duplicate frames

)

print("Engine:", meta["engine"])
print("Selected frames:", len(frames))
for f in frames:
    print(f["index"], f["timestamp_seconds"], f["path"])

Command-Line Interface

Invoke the extraction directly via the module CLI:

python -m skills.watch.scripts.frames \
    example.mp4 tmp/keyframes \
    --resolution 640 \
    --max-frames 30 \
    --start 10 \
    --end 70

The script outputs a JSON object containing metadata (including the engine type keyframe or uniform) and the list of extracted frame paths.

Summary

• -skip_frame nokey instructs ffmpeg to decode only I-frames, dramatically reducing processing time by skipping inter-frame dependencies.
• The implementation in skills/watch/scripts/frames.py combines this flag with the showinfo filter to capture precise timestamps for each keyframe.
• A robust fallback mechanism ensures that videos with sparse keyframes (< 4) automatically switch to uniform frame extraction.
• Optional perceptual deduplication and even-sampling refine the final frame selection while respecting the max_frames constraint.
• The system returns comprehensive metadata indicating whether keyframe or uniform extraction was used, along with candidate and deduplication counts.

Frequently Asked Questions

What is the difference between -skip_frame nokey and standard frame extraction?

Standard frame extraction processes every frame (I, P, and B) at a specified FPS, creating uniform temporal samples. The -skip_frame nokey option processes only keyframes (I-frames), which are typically placed at scene cuts or significant visual changes. This results in fewer frames that better represent distinct moments in the video, while significantly reducing CPU and I/O overhead.

Why does the code fall back to uniform extraction if fewer than 4 keyframes are found?

The KEYFRAME_MIN threshold of 4 ensures that videos with minimal motion or static content—which may contain very few natural keyframes—still provide adequate visual coverage for analysis. Without this fallback, a 10-minute static lecture video might yield only 1-2 frames, insufficient for meaningful video understanding tasks.

How does the showinfo filter capture timestamps without parsing the video twice?

The showinfo video filter runs during the same ffmpeg invocation that extracts the frames. It prints frame metadata to stderr/stdout including pts_time values, which the Python function captures and parses in real-time using the SHOWINFO_TS_RE regex. This single-pass approach ensures timestamp accuracy without requiring a separate probe operation.

Can I use this keyframe extraction method with video formats other than MP4?

Yes. The -skip_frame nokey option works with any video codec that supports keyframe indexing, including MKV, AVI, MOV, and WebM. The extract_keyframes function in bradautomates/claude-video handles format detection automatically through ffmpeg's universal input interface, making the technique applicable across modern video container formats.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}