Scene Detection Algorithm for Frame Extraction in claude-video: FFmpeg Implementation Explained

The claude-video repository uses FFmpeg's built-in scene filter as its scene detection algorithm, measuring histogram differences between consecutive frames with a default threshold of 0.20 to extract key frames without deploying custom computer vision models.

The bradautomates/claude-video repository provides video processing capabilities through its "watch" skill, which relies on this specific scene detection algorithm for intelligent frame extraction. By delegating to FFmpeg's native capabilities rather than implementing custom vision models, the system efficiently identifies meaningful visual transitions while maintaining cross-platform compatibility.

How FFmpeg Scene Detection Works in claude-video

The scene detection algorithm leverages FFmpeg's native scene filter through the select video filter. In skills/watch/scripts/frames.py, the implementation constructs a filter expression that captures both the initial frame and any subsequent frame where the scene change metric exceeds the configured threshold.

Frame Selection Logic

At line 252 in skills/watch/scripts/frames.py, the video filter string is dynamically assembled:

vf = f"select='eq(n\\,0)+gt(scene\\,{threshold})',{_scale_filter(resolution)},showinfo"

This filter expression operates through two conditions:

• eq(n,0) ensures the first frame is always selected
• gt(scene,{threshold}) selects additional frames where the scene score exceeds the threshold value

FFmpeg computes the scene metric by calculating the difference between consecutive frames, typically using histogram comparison or pixel-wise change detection. When this difference surpasses the configured threshold, FFmpeg emits the frame as a scene-change candidate for further processing.

Scene Detection Threshold Configuration

The sensitivity of the algorithm is controlled by the SCENE_THRESHOLD constant defined at line 20 in skills/watch/scripts/frames.py:

SCENE_THRESHOLD = 0.20

This default value of 0.20 represents the cutoff for detecting significant visual changes. Lower values increase sensitivity (capturing subtle transitions), while higher values require more dramatic scene changes before triggering extraction.

Extracting Scene-Aware Frames

The primary entry point for frame extraction is the extract_scene_or_uniform() function. When the detail parameter is set to "balanced", the function activates scene detection rather than uniform sampling.

from skills.watch.scripts import frames
from pathlib import Path

video_path = "example.mp4"
out_dir = Path("/tmp/frames")

# Returns list of frame dicts and metadata

frames_list, meta = frames.extract_scene_or_uniform(
    video_path,
    out_dir,
    max_frames=100,
    detail="balanced",  # Activates scene detection algorithm

)

print(f"Engine used: {meta['engine']}")  # → "scene"

print(f"Extracted {len(frames_list)} frames")

Under the hood, the script executes an FFmpeg command similar to:

ffmpeg -i example.mp4 -vf \
"select='eq(n\,0)+gt(scene\,0.20)',scale='min(iw,512)':'min(ih,1998)',showinfo" \
-frames:v 100 frame_%04d.jpg

Fallback and Deduplication Strategy

The implementation includes robust fallback mechanisms. If the scene detection algorithm identifies too few scene changes (indicating insufficient visual diversity in the source video), the system automatically falls back to uniform frame sampling to ensure adequate coverage. Additionally, the skill optionally deduplicates candidate frames to prevent processing visually similar content, as coordinated by skills/watch/scripts/watch.py.

Summary

• FFmpeg scene filter: The claude-video repository delegates scene detection to FFmpeg's proven algorithm rather than implementing custom vision models.
• Default threshold: Visual changes trigger at SCENE_THRESHOLD = 0.20, defined in skills/watch/scripts/frames.py.
• Filter construction: Line 252 builds the select video filter combining first-frame capture with threshold-based scene detection.
• Balanced extraction mode: Setting detail="balanced" activates scene-aware extraction through the extract_scene_or_uniform() function.
• Automatic fallback: The system switches to uniform sampling when scene detection yields insufficient candidate frames.

Frequently Asked Questions

What specific algorithm does claude-video use for scene detection?

The repository uses FFmpeg's native scene filter algorithm, which computes the difference between consecutive frames using histogram or pixel-wise comparison techniques. This implementation in skills/watch/scripts/frames.py requires no custom computer vision models or external ML dependencies.

How is the scene change threshold configured in the claude-video watch skill?

The threshold is defined by the SCENE_THRESHOLD constant set to 0.20 in skills/watch/scripts/frames.py at line 20. This scalar value determines the minimum visual difference required between consecutive frames to register as a scene change, with FFmpeg outputting frames that exceed this cutoff.

Does claude-video provide fallback options if scene detection finds too few frames?

Yes, the extract_scene_or_uniform() function automatically implements a fallback strategy. If the scene detection algorithm produces fewer frames than required, the system switches to uniform sampling across the video duration, ensuring the max_frames requirement is always met regardless of visual content complexity.

Which files contain the core frame extraction logic for scene detection?

The primary implementation resides in skills/watch/scripts/frames.py, which constructs the FFmpeg command-line arguments and manages the scene filter workflow. The orchestration layer in skills/watch/scripts/watch.py coordinates overall video processing, while skills/watch/SKILL.md documents the detail modes that trigger scene-aware extraction versus uniform sampling.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}