How Scene Detection Works for Frame Extraction in the watch Skill

Scene detection in the watch skill uses FFmpeg's built-in scene filter to identify visual cuts in video content, extracting representative frames at scene boundaries while automatically falling back to uniform sampling for static videos.

The watch skill in the bradautomates/claude-video repository intelligently extracts video frames by analyzing visual discontinuities rather than blindly sampling at fixed intervals. This content-aware approach lives primarily in skills/watch/scripts/frames.py and leverages FFmpeg's scene detection capabilities to produce meaningful thumbnails that represent actual editorial cuts in the source material.

Scene Detection Pipeline Overview

The scene detection workflow follows a six-stage pipeline that balances accuracy with efficiency. First, FFmpeg analyzes the video stream to detect frames where the scene metric exceeds a threshold. Then the system parses timestamps, collects JPEG outputs, annotates metadata, and applies post-processing to remove duplicates and cap the final frame count. If the video lacks sufficient scene changes, the pipeline automatically switches to uniform temporal sampling to guarantee coverage.

Core Implementation in frames.py

The heart of the scene detection logic resides in skills/watch/scripts/frames.py, specifically within the extract_scene_candidates function and its supporting constants.

FFmpeg Scene Filter Configuration

The detection relies on an FFmpeg select filter string constructed at lines 52-55:

select='eq(n\,0)+gt(scene\,{threshold})'

This filter expression emits the first frame (eq(n,0)) and every subsequent frame where the scene difference metric exceeds the configured threshold. The SCENE_THRESHOLD constant defaults to 0.20 (defined at lines 20-24), where higher values require larger visual changes to register as cuts. The command includes showinfo to print frame metadata to stderr:

ffmpeg -i input.mp4 -vf "select='eq(n\,0)+gt(scene\,0.20)',scale=512:-1,showinfo" -q:v 2 frame_%04d.jpg

Timestamp Extraction and Parsing

FFmpeg writes timestamp information to stderr during processing. The script captures this output and parses it using the SHOWINFO_TS_RE regex (lines 39-40) to extract absolute seconds for each candidate frame. This parsing converts FFmpeg's verbose logging into a clean list of timestamps that correspond one-to-one with the extracted JPEG files.

Frame Annotation and Metadata

After FFmpeg writes the numbered JPEGs (frame_%04d.jpg), the script gathers them from the output directory using frames = sorted(out_dir.glob("frame_*.jpg")) (lines 70-71). For each frame, it constructs a metadata dictionary containing:

• index – sequential order in the extraction sequence
• timestamp_seconds – the parsed absolute timestamp
• path – absolute file path to the JPEG
• reason – either "first-frame" for the initial frame or "scene-change" for detected cuts (lines 73-80)

Fallback and Post-Processing Logic

The higher-level extract_scene_or_uniform function orchestrates intelligent fallback behavior and content optimization.

Uniform Sampling Fallback

When extract_scene_or_uniform calls extract_scene_candidates, it evaluates the results against SCENE_MIN_FRAMES (default 8). If the detected scene count falls below this threshold (defined at lines 21-27 and referenced at lines 10-16), the video is considered visually static. In this case, the function abandons scene detection and invokes extract to perform uniform temporal sampling instead, ensuring the user receives a representative set of frames even when no cuts exist.

Deduplication and Frame Capping

For videos with sufficient scene changes, the pipeline applies two additional optimizations. First, dedupe_perceptual removes near-identical frames using perceptual hashing of thumbnails. Then _even_sample reduces the remaining frames to the user-specified max_frames limit (lines 43-48). This combination ensures the final output contains diverse, evenly distributed representative images without redundancy.

Practical Usage Examples

You can invoke the scene detection logic directly through the Python API or via the command-line interface.

Direct API Usage

Call extract_scene_or_uniform from skills/watch/scripts/frames.py to process a video with automatic fallback:

from pathlib import Path
from skills.watch.scripts.frames import extract_scene_or_uniform

video = "my_video.mp4"
out_dir = Path("./frames")
fps, target = 1.5, 100  # fallback uniform params (used only if scene fails)

frames, meta = extract_scene_or_uniform(
    video_path=video,
    out_dir=out_dir,
    fps=fps,
    target_frames=target,
    resolution=512,
    max_frames=80,          # cap after dedup

    start_seconds=None,
    end_seconds=None,
    dedup=True,
)

print("Engine used:", meta["engine"])
for f in frames[:5]:
    print(f["index"], f["timestamp_seconds"], f["reason"], f["path"])

Command-Line Interface

Use the high-level watch.py entry point for automatic engine selection:

python -m skills.watch.scripts.watch /path/to/video.mp4 ./out \
    --max-frames 120 --resolution 640

The script internally invokes extract_scene_or_uniform when sufficient scene cuts are detected, otherwise falling back to uniform extraction via extract.

Summary

• Scene detection in the watch skill leverages FFmpeg's scene filter to measure per-frame luma differences and identify visual cuts.
• The SCENE_THRESHOLD constant (0.20) and SCENE_MIN_FRAMES constant (8) control sensitivity and fallback behavior in skills/watch/scripts/frames.py.
• FFmpeg's showinfo flag provides timestamps that the script parses using SHOWINFO_TS_RE to create accurate frame metadata.
• The extract_scene_or_uniform function automatically falls back to uniform sampling when videos lack sufficient scene changes.
• Post-processing includes perceptual deduplication (dedupe_perceptual) and even sampling (_even_sample) to respect max_frames limits.

Frequently Asked Questions

What FFmpeg filter enables scene detection in the watch skill?

The implementation uses the select filter with the expression eq(n,0)+gt(scene,THRESHOLD) as defined in extract_scene_candidates at lines 52-55 of skills/watch/scripts/frames.py. This filter compares consecutive frames and emits those where the scene change metric exceeds the SCENE_THRESHOLD value of 0.20.

How does the system handle videos without scene cuts?

When fewer than 8 scene-change frames are detected (controlled by SCENE_MIN_FRAMES), the extract_scene_or_uniform function automatically falls back to uniform temporal sampling. This ensures static videos or single-shot content still yield a representative set of frames rather than returning empty results.

Where are the extracted frame timestamps sourced from?

Timestamps originate from FFmpeg's showinfo filter, which prints frame metadata to stderr during processing. The script captures this output and parses it using the SHOWINFO_TS_RE regex (lines 39-40) to extract absolute seconds for each candidate frame, stored as timestamp_seconds in the frame metadata.

Can the scene detection sensitivity be adjusted?

Yes. The SCENE_THRESHOLD constant at lines 20-24 of skills/watch/scripts/frames.py defaults to 0.20, but you can modify this value to make detection more or less sensitive. Lower values detect subtle changes, while higher values require dramatic visual differences to trigger scene boundaries.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}