How the Scene-Detection Algorithm Works in frames.py: A Technical Deep Dive

The scene-detection algorithm in frames.py uses ffmpeg's scene filter to identify candidate frames, parses timestamps from stderr output, applies perceptual deduplication to remove near-duplicates, and automatically falls back to uniform sampling if fewer than 8 scene changes are detected.

The claude-video repository by bradautomates provides intelligent video processing capabilities for Claude hosts. At the heart of its frame extraction system lies skills/watch/scripts/frames.py, which implements a robust scene-detection algorithm that balances content-aware accuracy with reliable fallback mechanisms for static or screen-recording content.

The Three-Step Scene-Detection Pipeline

The algorithm coordinates three distinct phases to extract representative frames from video files.

Step 1: Detecting Raw Scene Changes with ffmpeg

The process begins in extract_scene_candidates() where the code constructs an ffmpeg command using the select filter. The filter expression at lines 52-57 combines two conditions:

vf = f"select='eq(n\\,0)+gt(scene\\,{threshold})',{_scale_filter(resolution)},showinfo"

• eq(n,0) ensures the first frame is always captured, guaranteeing a start-point for the sequence.
• gt(scene,threshold) activates ffmpeg's built-in scene-change detector, comparing consecutive frames against the SCENE_THRESHOLD constant of 0.20 defined at line 20.
• showinfo emits metadata to stderr, enabling timestamp extraction for each candidate frame.

The _scale_filter() helper (referenced at lines 42-46) limits the extraction resolution to optimize performance before the scene detection runs.

Step 2: Parsing Timestamps and Building Candidates

After ffmpeg executes, the algorithm processes the stderr stream to extract temporal data. The regular expression SHOWINFO_TS_RE defined at lines 39-40 parses the pts_time values from lines like ... pts_time:12.345 ....

The function pairs these timestamps with the saved JPEG files (named frame_*.jpg) to construct the initial candidate list at lines 72-80. This step preserves the first frame unconditionally while maintaining the chronological order of detected scene changes.

Step 3: Perceptual Deduplication and Final Selection

Raw scene detection often produces near-duplicate frames from subtle lighting changes or minor camera movements. The dedupe_perceptual() function (implemented at lines 64-70) removes redundancies through the following process:

1. Downscaling: Each frame is converted to a 16×16 grayscale thumbnail using the DEDUP_THUMB constant of 16 defined at line 37.
2. Difference calculation: The algorithm computes the mean absolute per-pixel difference (_frame_delta) between successive thumbnails.
3. Threshold filtering: Any frame with a difference ≤ DEDUP_THRESHOLD (2.0) is considered a duplicate and discarded.

If the requested max_frames is fewer than the deduplicated candidates, the _even_indices() helper at lines 84-92 performs even-sampling, always retaining the first and last frames while distributing selections evenly across the timeline.

Fallback to Uniform Sampling

The algorithm implements a reliability guard through extract_scene_or_uniform(). At lines 41-53, the code checks if the raw detection produces at least SCENE_MIN_FRAMES (8) distinct shots, defined at lines 26-27.

If the scene engine detects fewer than 8 candidates, the function triggers the fallback mechanism at lines 54-73, switching to the simpler extract() function for uniform fps-based extraction. This ensures that static videos, screen recordings, or content with minimal visual changes still produce a useful frame set.

Implementation Details and Code Examples

To extract frames using the scene-detection pipeline with automatic fallback:

from pathlib import Path
from skills.watch.scripts.frames import extract_scene_or_uniform

video_path = "example.mp4"
out_dir = Path("out_frames")
out_dir.mkdir(parents=True, exist_ok=True)

# Request maximum 50 frames at 512px resolution

frames, meta = extract_scene_or_uniform(
    video_path=video_path,
    out_dir=out_dir,
    fps=2.0,                     # fallback fps (used only if scene detection fails)

    target_frames=50,
    resolution=512,
    max_frames=50,
    start_seconds=None,
    end_seconds=None,
    dedup=True,
)

print("Engine used:", meta["engine"])          # → "scene" or "uniform"

print("Selected frames:", len(frames))
for f in frames[:5]:
    print(f["index"], f["timestamp_seconds"], f["reason"])

For direct scene-only extraction without fallback or caps:

from skills.watch.scripts.frames import extract_scene_candidates

candidates = extract_scene_candidates(
    video_path="example.mp4",
    out_dir=Path("scene_frames"),
    resolution=512,
    max_frames=None,               # detect every cut, no early stop

    start_seconds=None,
    end_seconds=None,
    threshold=0.20,                # match repository default

)

print(f"Detected {len(candidates)} raw scene changes")

Summary

• skills/watch/scripts/frames.py implements a hybrid scene-detection algorithm using ffmpeg's scene filter with a default 0.20 threshold.
• The pipeline extracts timestamps via regex parsing of ffmpeg's showinfo stderr output.
• Perceptual deduplication uses 16×16 grayscale thumbnails and a mean absolute difference threshold of 2.0 to remove near-duplicates.
• The algorithm requires at least 8 scene changes (SCENE_MIN_FRAMES) to use scene-based extraction; otherwise, it falls back to uniform fps-based sampling.
• Even-sampling logic ensures representative frame distribution when capping the output count.

Frequently Asked Questions

What ffmpeg filter does the scene detection use?

The algorithm uses ffmpeg's select filter with the expression eq(n,0)+gt(scene,threshold) combined with showinfo. The eq(n,0) component ensures the first frame is always captured, while gt(scene,threshold) compares consecutive frames using ffmpeg's built-in scene-change detection score. The default threshold is 0.20 as defined by SCENE_THRESHOLD at line 20.

How does the algorithm handle videos with few scene changes?

If the detection finds fewer than SCENE_MIN_FRAMES (8) candidates, the extract_scene_or_uniform() function automatically falls back to uniform sampling. This fallback uses fps-based extraction to guarantee usable output for static content, screen recordings, or videos with minimal visual variation.

What is perceptual deduplication and why is it needed?

Perceptual deduplication removes near-identical frames that pass the initial scene detection but represent the same visual content. The dedupe_perceptual() function downscales frames to 16×16 grayscale thumbnails and calculates the mean absolute difference between successive frames. Frames with a difference ≤ 2.0 are discarded, ensuring only visually distinct shots remain in the final set.

How are timestamps extracted from ffmpeg output?

The algorithm captures ffmpeg's stderr stream, which contains showinfo metadata lines including pts_time values. The regular expression SHOWINFO_TS_RE defined at lines 39-40 parses these timestamps, which are then paired with the extracted JPEG files to create temporally accurate frame metadata.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}