# How Claude-Video Uses Scene-Change Detection to Select Representative Frames

> Discover how Claude-Video uses FFmpeg scene-change detection to select representative frames. Learn about shot boundary detection and frame extraction with a configurable threshold.

- Repository: [bradautomates/claude-video](https://github.com/bradautomates/claude-video)
- Tags: deep-dive
- Published: 2026-07-27

---

**Claude-Video relies on FFmpeg's built-in `scene` filter to detect shot boundaries, extracting the first frame of each detected scene using a configurable threshold of 0.20.**

The `bradautomates/claude-video` repository implements intelligent frame extraction by leveraging FFmpeg's scene-change detection capabilities. This approach identifies distinct visual shots within video content, ensuring that extracted frames represent meaningful transitions rather than arbitrary intervals. Understanding this scene-change detection method reveals how the system balances computational efficiency with content-aware sampling.

## FFmpeg Scene Filter as the Detection Engine

The core detection mechanism utilizes FFmpeg's `scene` video filter, which computes the per-pixel difference between consecutive frames and normalizes the result to a value between 0 and 1. When this **scene score** exceeds the defined threshold, FFmpeg classifies the frame as the start of a new scene or shot boundary.

## Implementation Details in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py)

### Scene Threshold Configuration

The detection sensitivity is controlled by the `SCENE_THRESHOLD` constant, defined as `0.20` in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py). This value determines how much visual change must occur between frames to trigger a scene detection event. A lower threshold increases sensitivity, capturing subtle transitions, while a higher threshold only registers significant visual changes.

### Filter Chain Construction

When processing videos, the script constructs an FFmpeg video-filter chain that combines two selection criteria:

1. The very first frame of the video (`eq(n\,0)`)
2. Any frame where the scene score exceeds the threshold (`gt(scene, threshold)`)

The complete filter expression appears as:

```python
select='eq(n\,0)+gt(scene\,{threshold})'

```

This ensures capture of the opening frame plus every detected scene cut.

### Frame Processing Pipeline

After scene detection, the pipeline applies additional processing:

- **Scaling**: The `_scale_filter` function scales output to a configurable resolution (default 512px)
- **Timestamp extraction**: FFmpeg's `showinfo` output is parsed using the `SHOWINFO_TS_RE` regex pattern to associate each extracted JPEG with its precise source time in seconds
- **Post-processing**: The resulting candidate list undergoes deduplication and even-sampling adjustments within the same module

## Practical Usage Example

You can programmatically extract scene-based frames using the `extract_scene_candidates` function:

```python
from pathlib import Path
from skills.watch.scripts.frames import extract_scene_candidates

video = "example.mp4"
out_dir = Path("frames")
candidates = extract_scene_candidates(
    video_path=video,
    out_dir=out_dir,
    resolution=512,
    max_frames=100,           # optional cap; None = uncapped

    start_seconds=None,
    end_seconds=None,
    threshold=0.20,           # default scene-change sensitivity

)

for f in candidates:
    print(f["index"], f["timestamp_seconds"], f["reason"])

```

When too few scene changes are detected, the system automatically falls back to uniform sampling to ensure adequate frame coverage.

## Summary

- **FFmpeg scene filter** drives the detection logic by calculating normalized per-pixel differences between consecutive frames
- **Threshold-based selection** uses a default value of `0.20` defined in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) to identify shot boundaries
- **Filter syntax** combines initial frame capture with scene-cut detection via `select='eq(n\,0)+gt(scene\,{threshold})'`
- **Fallback mechanism** ensures uniform sampling when scene changes are insufficient for the requested frame count
- **Timestamp tracking** maintains precise temporal metadata for each extracted frame through FFmpeg's `showinfo` output

## Frequently Asked Questions

### What specific algorithm does Claude-Video use for scene-change detection?

Claude-Video uses FFmpeg's built-in `scene` filter, which calculates the per-pixel difference between the current frame and the previous one, normalizing the result to a value between 0 and 1. Frames exceeding the threshold are treated as scene cuts.

### How can I adjust the sensitivity of scene-change detection?

Modify the `threshold` parameter when calling `extract_scene_candidates()` or adjust the `SCENE_THRESHOLD` constant in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py). The default value of `0.20` provides balanced detection; lower values capture subtle transitions while higher values require significant visual changes.

### What happens when a video contains no scene changes?

If the scene-change detection yields too few frames, the system automatically falls back to uniform sampling. This behavior is implemented in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) to ensure the extraction process always returns a usable set of representative frames.

### Does the method capture the first frame of every scene or all frames within a scene?

The method captures only the **first frame** of each detected scene (plus the video's initial frame). The FFmpeg `select` filter triggers once when the scene score exceeds the threshold, marking the boundary transition, rather than capturing multiple frames within a single continuous shot.