Claude-Video FFmpeg Command Flow for Scene Cut and Keyframe Extraction

Claude-Video utilizes two distinct FFmpeg pipelines—one leveraging the select='gt(scene,THRESH)' video filter for scene-change detection and another employing the -skip_frame nokey flag for I-frame extraction—to decode only salient frames while harvesting timestamps from showinfo stderr output.

The claude-video repository by bradautomates implements intelligent frame sampling through targeted FFmpeg commands defined in skills/watch/scripts/frames.py. Rather than uniformly extracting frames at fixed intervals, the codebase selectively decodes visually significant moments using FFmpeg’s built-in scene detection and keyframe indexing capabilities.

Scene-Cut Frame Extraction Pipeline

The scene-cut extraction pipeline identifies abrupt visual transitions to capture representative frames from distinct shots.

FFmpeg Command Construction

In skills/watch/scripts/frames.py, the extract_scene_candidates function (lines 17-41, 44-63) constructs an FFmpeg command utilizing the scene detection filter:

ffmpeg -hide_banner -loglevel info -y \
  -ss START -to END \
  -i INPUT_VIDEO \
  -vf "select='eq(n\,0)+gt(scene\,THRESH)',scale=WIDTH:HEIGHT:force_original_aspect_ratio=decrease:force_divisible_by=2,showinfo" \
  -vsync vfr \
  -q:v 4 \
  OUTPUT_PATTERN

Key parameters:

• select='eq(n\,0)+gt(scene\,THRESH)' – Retains the first frame (eq(n,0)) and any frame where the scene-change confidence exceeds the threshold (default 0.20).
• showinfo – Emits frame metadata to stderr, including presentation timestamps.
• -vsync vfr – Outputs only the selected frames as individual JPEGs without duplicating frames to maintain constant frame rate.

Timestamp Harvesting

After execution, the code parses the stderr output using the regex pattern pts_time:([0-9.]+) to extract precise timestamps for each saved frame. This allows downstream processing to associate specific times with extracted images without re-scanning the video.

Key-Frame (I-Frame) Extraction Pipeline

For videos where scene detection may fail or for alternative sampling strategies, Claude-Video provides a keyframe-only extraction method that decodes exclusively I-frames produced by the video encoder.

FFmpeg Command Structure

The extract_keyframes function (lines 75-85, 100-124) implements this pipeline:

ffmpeg -hide_banner -loglevel info -y \
  -skip_frame nokey \
  -ss START -to END \
  -i INPUT_VIDEO \
  -vf "scale=WIDTH:HEIGHT:force_original_aspect_ratio=decrease:force_divisible_by=2,showinfo" \
  -vsync vfr \
  -q:v 4 \
  OUTPUT_PATTERN

Critical flags:

• -skip_frame nokey – Instructs the decoder to process only keyframes (I-frames), skipping all predictive frames. This significantly reduces CPU load while capturing structurally complete images that typically align with scene boundaries.
• Identical scaling and timestamp extraction logic as the scene-cut pipeline.

Fallback Mechanism

If the encoder generated fewer than 4 keyframes (configurable via SCENE_MIN_FRAMES at line 22-24), the function automatically falls back to uniform FPS-based extraction to ensure sufficient frame coverage.

Implementation Architecture

The frame extraction workflow follows a structured pipeline defined across several functions in skills/watch/scripts/frames.py:

1. Metadata Gathering – get_metadata runs ffprobe (lines 86-20) to determine duration, dimensions, and codec information before selecting extraction parameters.

2. Budget Calculation – auto_fps or auto_fps_focus (lines 22-39) computes an appropriate frame rate or budget based on video length and user-specified max_frames.

3. Extraction Dispatch – extract_scene_or_uniform decides whether to return scene-cut candidates or fall back to uniform extraction based on the minimum frame threshold.

4. Deduplication – dedupe_perceptual (lines 63-78) removes near-identical frames using tiny grayscale thumbnails to prevent redundant processing of visually similar images.

5. Even Sampling – _even_sample (lines 93-103) evenly distributes the final frame selection across the video duration while preserving the first and last frames, respecting the max_frames constraint.

Usage Examples

Extract Scene-Cut Frames (Python API)

from pathlib import Path
from skills.watch.scripts.frames import extract_scene_candidates

candidates = extract_scene_candidates(
    video_path="example.mp4",
    out_dir=Path("frames"),
    resolution=512,
    max_frames=100,
    threshold=0.20,
    start_seconds=10,
    end_seconds=60
)

# Returns list of dicts with keys: index, timestamp_seconds, path, reason

Extract Keyframes (Python API)

from pathlib import Path
from skills.watch.scripts.frames import extract_keyframes

frames, meta = extract_keyframes(
    video_path="example.mp4",
    out_dir=Path("keyframes"),
    resolution=512,
    max_frames=50,
    dedup=True
)

print(meta)  # engine='keyframe', candidate_count=..., selected_count=...

Command-Line Interface

python3 skills/watch/scripts/frames.py example.mp4 frames_out \
    --resolution 640 \
    --max-frames 80

This CLI entry point automatically selects the appropriate extraction strategy based on video characteristics and performs post-processing deduplication.

Summary

• Scene-cut extraction uses FFmpeg’s select='gt(scene,0.20)' filter combined with showinfo to capture frames at visual discontinuities, implemented in extract_scene_candidates.
• Keyframe extraction leverages -skip_frame nokey to decode only I-frames, falling back to uniform sampling if fewer than 4 keyframes exist, implemented in extract_keyframes.
• Both pipelines parse pts_time values from showinfo stderr output to maintain precise temporal metadata.
• Post-processing includes perceptual deduplication (dedupe_perceptual) and even sampling (_even_sample) to respect frame budgets while maximizing visual coverage.
• All logic resides in skills/watch/scripts/frames.py, orchestrated by skills/watch/scripts/watch.py for the /watch skill interface.

Frequently Asked Questions

What is the default scene detection threshold in Claude-Video?

The default threshold is 0.20, defined as the threshold parameter in extract_scene_candidates. Values closer to 0.0 increase sensitivity (detecting subtle changes), while values approaching 1.0 require drastic scene changes to trigger frame selection.

How does the pipeline handle videos with insufficient keyframes?

When -skip_frame nokey yields fewer than 4 frames (configurable via SCENE_MIN_FRAMES), the extract_keyframes function automatically falls back to uniform FPS-based extraction. This ensures the system always returns a usable set of frames even for videos with aggressive keyframe optimization or very short durations.

Why does Claude-Video use showinfo instead of ffprobe for timestamps?

The showinfo video filter emits timestamps to stderr during the frame extraction process, eliminating the need for a separate metadata scan. This single-pass approach improves performance by decoding the video once while simultaneously capturing both the JPEG output and temporal data via regex parsing of pts_time values.

Where is the frame extraction logic implemented in the repository?

All FFmpeg command construction and execution logic resides in skills/watch/scripts/frames.py. High-level orchestration—including the decision between scene-cut, keyframe, or uniform extraction—occurs in skills/watch/scripts/watch.py, while test coverage exists in tests/test_frames.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}