# How Scene-Change Detection Works with FFmpeg in Claude-Video

> Learn how Claude-Video uses FFmpeg's scene filter and showinfo output for accurate scene-change detection. Understand frame luminance comparison and timestamp parsing.

- Repository: [bradautomates/claude-video](https://github.com/bradautomates/claude-video)
- Tags: internals
- Published: 2026-08-13

---

**Claude-Video leverages FFmpeg's built-in `scene` filter to detect cuts by comparing per-frame luminance differences against a configurable threshold, parsing timestamps from `showinfo` stderr output to build frame metadata.**

Scene-change detection is the backbone of intelligent video analysis in the `bradautomates/claude-video` repository. By utilizing FFmpeg's native filtering capabilities rather than external computer vision libraries, the tool achieves cross-platform compatibility with minimal dependencies. This article breaks down the exact implementation found in the source code, from threshold configuration to timestamp parsing.

## Configuring the Detection Threshold

The sensitivity of scene detection is controlled by a single constant defined in **[`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py)**.

**`SCENE_THRESHOLD = 0.20`** (line 20) determines how much the average per-pixel luminance must differ between consecutive frames before FFmpeg flags a cut. This value represents a 20% luminance change. Lower values increase sensitivity, catching subtle transitions, while higher values create a more conservative detector that only captures hard cuts.

## Building the FFmpeg Select Filter

The function `extract_scene_candidates()` constructs a complex FFmpeg command that combines scene detection with frame selection logic.

The critical filter expression appears on lines 52-56:

```python
vf = f"select='eq(n\\,0)+gt(scene\\,{threshold})',{_scale_filter(resolution)},showinfo"

```

This expression performs two simultaneous operations:

- **`select='eq(n\,0)+gt(scene\,{threshold})'`** - Outputs the first frame (`eq(n,0)`) unconditionally, plus any frame where the scene metric exceeds the threshold (`gt(scene,0.20)`)
- **`showinfo`** - Appends `pts_time` entries to FFmpeg's stderr for each selected frame, enabling precise timestamp extraction

The command also applies a scaling filter to constrain output resolution while preserving aspect ratio.

## Parsing Timestamps from FFmpeg Output

After FFmpeg execution completes, the Python code extracts precise timing data using regular expression matching.

**`SHOWINFO_TS_RE = re.compile(r"pts_time:([0-9.]+)")`** (line 39) scans the `showinfo` log entries emitted to stderr. The matching timestamps (extracted on line 69) are converted into structured metadata dictionaries:

```python
{
    "index": i,
    "timestamp_seconds": ts,
    "path": str(path),
    "reason": "first-frame" if i == 0 else "scene-change",
}

```

This metadata generation occurs on lines 73-78, creating a machine-readable record of exactly why each frame was selected.

## Fallback to Uniform Sampling

Static or short videos may not produce enough natural scene changes for meaningful analysis. To handle this edge case, the system implements a fallback strategy.

If the video yields fewer than **`SCENE_MIN_FRAMES = 8`** (line 26) distinct cuts, the engine automatically switches to `extract()`—a uniform-sampling strategy that distributes frames evenly across the duration. This ensures that even content with minimal motion still generates a representative set of frames for downstream processing.

## Practical Implementation Examples

The high-level entry point **[`skills/watch/scripts/watch.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/watch.py)** orchestrates the detection flow, but you can invoke the scene engine directly from your own scripts.

### Python: Extract Scene Candidates

```python
from pathlib import Path
from skills.watch.scripts.frames import extract_scene_or_uniform

video_path = "example.mp4"
out_dir = Path("frames")

# Request 120 frames; FFmpeg detects natural cuts first

frames, meta = extract_scene_or_uniform(
    video_path,
    out_dir,
    fps=2.0,              # fallback fps (used only if uniform engine activates)

    target_frames=120,
    resolution=512,
    max_frames=120,
    start_seconds=None,
    end_seconds=None,
    dedup=True,
)

print(meta)   # → {'engine': 'scene', 'candidate_count': 34, ...}

```

### Bash: Underlying FFmpeg Command

The `extract_scene_candidates()` function generates commands equivalent to:

```bash
ffmpeg -hide_banner -loglevel info -y \
  -i example.mp4 \
  -vf "select='eq(n\,0)+gt(scene\,0.20)',scale=w='min(512,iw)':h='min(1998,ih)'" \
  -vsync vfr -q:v 4 frame_%04d.jpg

```

The `select` filter emits the first frame and any frame exceeding the 0.20 scene threshold, while the scale filter ensures dimensions remain even and width stays under 512 pixels.

## Summary

- **Scene detection** relies on FFmpeg's native `scene` filter comparing per-pixel luminance between frames.
- **`SCENE_THRESHOLD = 0.20`** in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) controls detection sensitivity.
- **`extract_scene_candidates()`** builds a filter chain combining `select`, `scale`, and `showinfo` filters.
- **Timestamp parsing** uses the regex `SHOWINFO_TS_RE` to extract `pts_time` values from FFmpeg's stderr.
- **Automatic fallback** to uniform sampling occurs when fewer than 8 scene changes are detected, ensuring adequate frame coverage.

## Frequently Asked Questions

### What FFmpeg filter enables scene detection in claude-video?

Claude-Video uses FFmpeg's **`scene`** filter within a **`select`** expression. The specific syntax `gt(scene,0.20)` compares each frame's scene score against the threshold, outputting only frames that exceed this luminance difference or the first frame of the video.

### How does claude-video handle videos with few scene changes?

When natural cuts produce fewer than **8 frames** (`SCENE_MIN_FRAMES`), the system automatically falls back to uniform sampling via the `extract()` function. This distributes frames evenly across the timeline to guarantee a minimum viable dataset for analysis.

### Where is the scene detection logic implemented?

The core implementation resides in **[`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py)**, specifically within the `extract_scene_candidates()` function (lines 52-56) and the timestamp parsing logic using `SHOWINFO_TS_RE` (line 39). The high-level orchestration occurs in **[`skills/watch/scripts/watch.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/watch.py)**, which selects between scene, keyframe, and uniform engines based on user parameters.

### Can I adjust the scene detection sensitivity?

Yes. Modify the **`SCENE_THRESHOLD`** constant (default 0.20) in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py). Lower values (e.g., 0.15) detect more subtle transitions, while higher values (e.g., 0.30) restrict detection to only major cuts. Note that extremely low values may increase processing time and produce redundant frames.