Scene-Change Detection Engine in Claude-Video: FFmpeg-Based Shot Detection

The scene-change detection engine in Claude-Video leverages FFmpeg's built-in scene filter with a configurable 0.20 threshold to identify meaningful shot boundaries, automatically falling back to uniform sampling when fewer than 8 scenes are detected.

The bradautomates/claude-video repository implements an intelligent frame extraction pipeline that prioritizes content-rich moments over blind temporal sampling. At the heart of this system lies the scene-change detection engine, which analyzes video streams to identify meaningful transitions and extract representative frames for downstream AI processing.

FFmpeg Scene Filter Architecture

The engine is implemented in skills/watch/scripts/frames.py, where it orchestrates FFmpeg-based detection through the extract_scene_candidates function.

Scene Metric and Threshold Configuration

Sensitivity is controlled by the constant SCENE_THRESHOLD = 0.20 defined at line 20. This threshold determines how drastic a visual change must be to register as a new scene boundary.

At lines 52-53, the engine constructs an FFmpeg select filter string:

select='eq(n\,0)+gt(scene\,{threshold})'

This filter instructs FFmpeg to keep the first frame (eq(n\,0)) and any subsequent frame where the computed scene metric exceeds the threshold (gt(scene\,{threshold})).

Timestamp Extraction

After invoking FFmpeg with the showinfo flag, the engine parses the resulting log output to extract precise timestamps for each detected shot. This parsing logic resides at lines 68-70 of frames.py, converting FFmpeg's verbose output into structured frame metadata.

Fallback Logic for Static Content

Not all videos contain meaningful scene transitions. The engine implements a safeguard through SCENE_MIN_FRAMES, set to 8 scenes by default.

If extract_scene_or_uniform (lines 22-23) detects fewer than 8 scene changes, it classifies the video as "effectively static" and switches to uniform frame extraction at the user-specified FPS rate. This ensures that single-shot or low-motion content still yields a usable frame set for analysis.

Perceptual Deduplication Pipeline

After collecting scene candidates, the engine performs post-processing to eliminate redundancy. At lines 43-45, extract_scene_or_uniform calls dedupe_perceptual to remove near-identical frames caused by slight encoding variations or minimal motion.

The deduplication step occurs before the final even-sampling phase, which adjusts the frame count to match the target budget specified by the max_frames parameter.

Integration with the Watch Pipeline

While frames.py contains the core engine, skills/watch/scripts/watch.py serves as the top-level orchestrator. This entry point selects between scene-based detection, keyframe extraction, or uniform sampling based on video characteristics and user preferences.

The test suite in tests/test_frames.py verifies the correctness of the scene-change detection logic across various video formats and content types.

Using the Engine in Your Code

You can interact with the scene-change detection engine directly through Python or the command-line interface.

Python API

Import extract_scene_or_uniform to leverage the engine with automatic fallback handling:

from pathlib import Path
from skills.watch.scripts.frames import extract_scene_or_uniform

video_path = "example.mp4"
out_dir = Path("out/frames")

# Ask the engine to prefer scene cuts, falling back to uniform sampling.

frames, meta = extract_scene_or_uniform(
    video_path=video_path,
    out_dir=out_dir,
    fps=2.0,               # fallback fps if uniform sampling is needed

    target_frames=100,     # desired number of frames

    resolution=512,
    max_frames=100,        # hard cap for the final output

    start_seconds=None,
    end_seconds=None,
    dedup=True,
)

print("Engine used:", meta["engine"])
print("Number of frames returned:", len(frames))

Command-Line Interface

Run the full extraction pipeline directly from the terminal:

python -m skills.watch.scripts.frames example.mp4 out/frames

Summary

• The scene-change detection engine uses FFmpeg's scene filter with a default SCENE_THRESHOLD of 0.20 defined in skills/watch/scripts/frames.py.
• The filter string select='eq(n\,0)+gt(scene\,{threshold})' captures the first frame and all subsequent shot boundaries.
• If fewer than 8 scenes are detected, the engine falls back to uniform sampling via extract_scene_or_uniform.
• Perceptual deduplication via dedupe_perceptual removes redundant frames before final sampling.
• The watch.py module orchestrates the engine alongside alternative extraction strategies.

Frequently Asked Questions

What threshold does Claude-Video use for scene detection?

The engine uses SCENE_THRESHOLD = 0.20 as defined at line 20 of skills/watch/scripts/frames.py. This value balances sensitivity to capture genuine shot changes while filtering out minor lighting fluctuations and compression artifacts.

How does the engine handle videos without scene changes?

If the detector finds fewer than SCENE_MIN_FRAMES (8) scenes, the extract_scene_or_uniform function (lines 22-23) automatically switches to uniform frame extraction at the specified FPS rate. This ensures static or single-take videos still produce the target number of frames for analysis.

Can I adjust the scene detection sensitivity?

While the default threshold is hardcoded at 0.20, you can modify the SCENE_THRESHOLD constant in frames.py or extend the extract_scene_candidates function to accept a custom threshold parameter for the FFmpeg scene filter.

What file formats does the scene-change detection engine support?

The engine supports any video format compatible with FFmpeg, including MP4, MOV, AVI, and MKV. Because the underlying detection relies entirely on FFmpeg's decoding and scene filtering capabilities, format support matches your local FFmpeg installation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}