# Scene Detection Algorithm for Frame Extraction in claude-video: FFmpeg Implementation Explained

> Discover how FFmpeg's scene detection algorithm extracts key frames in claude-video. Learn about histogram differences and efficient frame extraction without custom CV models.

- Repository: [bradautomates/claude-video](https://github.com/bradautomates/claude-video)
- Tags: deep-dive
- Published: 2026-08-07

---

**The claude-video repository uses FFmpeg's built-in `scene` filter as its scene detection algorithm, measuring histogram differences between consecutive frames with a default threshold of 0.20 to extract key frames without deploying custom computer vision models.**

The `bradautomates/claude-video` repository provides video processing capabilities through its "watch" skill, which relies on this specific scene detection algorithm for intelligent frame extraction. By delegating to FFmpeg's native capabilities rather than implementing custom vision models, the system efficiently identifies meaningful visual transitions while maintaining cross-platform compatibility.

## How FFmpeg Scene Detection Works in claude-video

The scene detection algorithm leverages FFmpeg's native `scene` filter through the `select` video filter. In [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py), the implementation constructs a filter expression that captures both the initial frame and any subsequent frame where the scene change metric exceeds the configured threshold.

### Frame Selection Logic

At line 252 in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py), the video filter string is dynamically assembled:

```python
vf = f"select='eq(n\\,0)+gt(scene\\,{threshold})',{_scale_filter(resolution)},showinfo"

```

This filter expression operates through two conditions:
- `eq(n,0)` ensures the first frame is always selected
- `gt(scene,{threshold})` selects additional frames where the scene score exceeds the threshold value

FFmpeg computes the `scene` metric by calculating the difference between consecutive frames, typically using histogram comparison or pixel-wise change detection. When this difference surpasses the configured threshold, FFmpeg emits the frame as a scene-change candidate for further processing.

## Scene Detection Threshold Configuration

The sensitivity of the algorithm is controlled by the `SCENE_THRESHOLD` constant defined at line 20 in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py):

```python
SCENE_THRESHOLD = 0.20

```

This default value of 0.20 represents the cutoff for detecting significant visual changes. Lower values increase sensitivity (capturing subtle transitions), while higher values require more dramatic scene changes before triggering extraction.

## Extracting Scene-Aware Frames

The primary entry point for frame extraction is the `extract_scene_or_uniform()` function. When the `detail` parameter is set to `"balanced"`, the function activates scene detection rather than uniform sampling.

```python
from skills.watch.scripts import frames
from pathlib import Path

video_path = "example.mp4"
out_dir = Path("/tmp/frames")

# Returns list of frame dicts and metadata

frames_list, meta = frames.extract_scene_or_uniform(
    video_path,
    out_dir,
    max_frames=100,
    detail="balanced",  # Activates scene detection algorithm

)

print(f"Engine used: {meta['engine']}")  # → "scene"

print(f"Extracted {len(frames_list)} frames")

```

Under the hood, the script executes an FFmpeg command similar to:

```bash
ffmpeg -i example.mp4 -vf \
"select='eq(n\,0)+gt(scene\,0.20)',scale='min(iw,512)':'min(ih,1998)',showinfo" \
-frames:v 100 frame_%04d.jpg

```

## Fallback and Deduplication Strategy

The implementation includes robust fallback mechanisms. If the scene detection algorithm identifies too few scene changes (indicating insufficient visual diversity in the source video), the system automatically falls back to uniform frame sampling to ensure adequate coverage. Additionally, the skill optionally deduplicates candidate frames to prevent processing visually similar content, as coordinated by [`skills/watch/scripts/watch.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/watch.py).

## Summary

- **FFmpeg `scene` filter**: The claude-video repository delegates scene detection to FFmpeg's proven algorithm rather than implementing custom vision models.
- **Default threshold**: Visual changes trigger at `SCENE_THRESHOLD = 0.20`, defined in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py).
- **Filter construction**: Line 252 builds the `select` video filter combining first-frame capture with threshold-based scene detection.
- **Balanced extraction mode**: Setting `detail="balanced"` activates scene-aware extraction through the `extract_scene_or_uniform()` function.
- **Automatic fallback**: The system switches to uniform sampling when scene detection yields insufficient candidate frames.

## Frequently Asked Questions

### What specific algorithm does claude-video use for scene detection?

The repository uses FFmpeg's native `scene` filter algorithm, which computes the difference between consecutive frames using histogram or pixel-wise comparison techniques. This implementation in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) requires no custom computer vision models or external ML dependencies.

### How is the scene change threshold configured in the claude-video watch skill?

The threshold is defined by the `SCENE_THRESHOLD` constant set to 0.20 in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) at line 20. This scalar value determines the minimum visual difference required between consecutive frames to register as a scene change, with FFmpeg outputting frames that exceed this cutoff.

### Does claude-video provide fallback options if scene detection finds too few frames?

Yes, the `extract_scene_or_uniform()` function automatically implements a fallback strategy. If the scene detection algorithm produces fewer frames than required, the system switches to uniform sampling across the video duration, ensuring the `max_frames` requirement is always met regardless of visual content complexity.

### Which files contain the core frame extraction logic for scene detection?

The primary implementation resides in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py), which constructs the FFmpeg command-line arguments and manages the `scene` filter workflow. The orchestration layer in [`skills/watch/scripts/watch.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/watch.py) coordinates overall video processing, while [`skills/watch/SKILL.md`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/SKILL.md) documents the `detail` modes that trigger scene-aware extraction versus uniform sampling.