How Scene-Change Detection Works with FFmpeg in Claude-Video

Claude-Video leverages FFmpeg's built-in scene filter to detect cuts by comparing per-frame luminance differences against a configurable threshold, parsing timestamps from showinfo stderr output to build frame metadata.

Scene-change detection is the backbone of intelligent video analysis in the bradautomates/claude-video repository. By utilizing FFmpeg's native filtering capabilities rather than external computer vision libraries, the tool achieves cross-platform compatibility with minimal dependencies. This article breaks down the exact implementation found in the source code, from threshold configuration to timestamp parsing.

Configuring the Detection Threshold

The sensitivity of scene detection is controlled by a single constant defined in skills/watch/scripts/frames.py.

SCENE_THRESHOLD = 0.20 (line 20) determines how much the average per-pixel luminance must differ between consecutive frames before FFmpeg flags a cut. This value represents a 20% luminance change. Lower values increase sensitivity, catching subtle transitions, while higher values create a more conservative detector that only captures hard cuts.

Building the FFmpeg Select Filter

The function extract_scene_candidates() constructs a complex FFmpeg command that combines scene detection with frame selection logic.

The critical filter expression appears on lines 52-56:

vf = f"select='eq(n\\,0)+gt(scene\\,{threshold})',{_scale_filter(resolution)},showinfo"

This expression performs two simultaneous operations:

• select='eq(n\,0)+gt(scene\,{threshold})' - Outputs the first frame (eq(n,0)) unconditionally, plus any frame where the scene metric exceeds the threshold (gt(scene,0.20))
• showinfo - Appends pts_time entries to FFmpeg's stderr for each selected frame, enabling precise timestamp extraction

The command also applies a scaling filter to constrain output resolution while preserving aspect ratio.

Parsing Timestamps from FFmpeg Output

After FFmpeg execution completes, the Python code extracts precise timing data using regular expression matching.

SHOWINFO_TS_RE = re.compile(r"pts_time:([0-9.]+)") (line 39) scans the showinfo log entries emitted to stderr. The matching timestamps (extracted on line 69) are converted into structured metadata dictionaries:

{
    "index": i,
    "timestamp_seconds": ts,
    "path": str(path),
    "reason": "first-frame" if i == 0 else "scene-change",
}

This metadata generation occurs on lines 73-78, creating a machine-readable record of exactly why each frame was selected.

Fallback to Uniform Sampling

Static or short videos may not produce enough natural scene changes for meaningful analysis. To handle this edge case, the system implements a fallback strategy.

If the video yields fewer than SCENE_MIN_FRAMES = 8 (line 26) distinct cuts, the engine automatically switches to extract()—a uniform-sampling strategy that distributes frames evenly across the duration. This ensures that even content with minimal motion still generates a representative set of frames for downstream processing.

Practical Implementation Examples

The high-level entry point skills/watch/scripts/watch.py orchestrates the detection flow, but you can invoke the scene engine directly from your own scripts.

Python: Extract Scene Candidates

from pathlib import Path
from skills.watch.scripts.frames import extract_scene_or_uniform

video_path = "example.mp4"
out_dir = Path("frames")

# Request 120 frames; FFmpeg detects natural cuts first

frames, meta = extract_scene_or_uniform(
    video_path,
    out_dir,
    fps=2.0,              # fallback fps (used only if uniform engine activates)

    target_frames=120,
    resolution=512,
    max_frames=120,
    start_seconds=None,
    end_seconds=None,
    dedup=True,
)

print(meta)   # → {'engine': 'scene', 'candidate_count': 34, ...}

Bash: Underlying FFmpeg Command

The extract_scene_candidates() function generates commands equivalent to:

ffmpeg -hide_banner -loglevel info -y \
  -i example.mp4 \
  -vf "select='eq(n\,0)+gt(scene\,0.20)',scale=w='min(512,iw)':h='min(1998,ih)'" \
  -vsync vfr -q:v 4 frame_%04d.jpg

The select filter emits the first frame and any frame exceeding the 0.20 scene threshold, while the scale filter ensures dimensions remain even and width stays under 512 pixels.

Summary

• Scene detection relies on FFmpeg's native scene filter comparing per-pixel luminance between frames.
• SCENE_THRESHOLD = 0.20 in skills/watch/scripts/frames.py controls detection sensitivity.
• extract_scene_candidates() builds a filter chain combining select, scale, and showinfo filters.
• Timestamp parsing uses the regex SHOWINFO_TS_RE to extract pts_time values from FFmpeg's stderr.
• Automatic fallback to uniform sampling occurs when fewer than 8 scene changes are detected, ensuring adequate frame coverage.

Frequently Asked Questions

What FFmpeg filter enables scene detection in claude-video?

Claude-Video uses FFmpeg's scene filter within a select expression. The specific syntax gt(scene,0.20) compares each frame's scene score against the threshold, outputting only frames that exceed this luminance difference or the first frame of the video.

How does claude-video handle videos with few scene changes?

When natural cuts produce fewer than 8 frames (SCENE_MIN_FRAMES), the system automatically falls back to uniform sampling via the extract() function. This distributes frames evenly across the timeline to guarantee a minimum viable dataset for analysis.

Where is the scene detection logic implemented?

The core implementation resides in skills/watch/scripts/frames.py, specifically within the extract_scene_candidates() function (lines 52-56) and the timestamp parsing logic using SHOWINFO_TS_RE (line 39). The high-level orchestration occurs in skills/watch/scripts/watch.py, which selects between scene, keyframe, and uniform engines based on user parameters.

Can I adjust the scene detection sensitivity?

Yes. Modify the SCENE_THRESHOLD constant (default 0.20) in skills/watch/scripts/frames.py. Lower values (e.g., 0.15) detect more subtle transitions, while higher values (e.g., 0.30) restrict detection to only major cuts. Note that extremely low values may increase processing time and produce redundant frames.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}