# Scene-Change Detection Engine in Claude-Video: FFmpeg-Based Shot Detection

> Explore Claude-Video's scene-change detection engine. Discover how it uses FFmpeg for efficient shot detection and automatic fallback to uniform sampling.

- Repository: [bradautomates/claude-video](https://github.com/bradautomates/claude-video)
- Tags: internals
- Published: 2026-08-09

---

**The scene-change detection engine in Claude-Video leverages FFmpeg's built-in scene filter with a configurable 0.20 threshold to identify meaningful shot boundaries, automatically falling back to uniform sampling when fewer than 8 scenes are detected.**

The `bradautomates/claude-video` repository implements an intelligent frame extraction pipeline that prioritizes content-rich moments over blind temporal sampling. At the heart of this system lies the **scene-change detection engine**, which analyzes video streams to identify meaningful transitions and extract representative frames for downstream AI processing.

## FFmpeg Scene Filter Architecture

The engine is implemented in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py), where it orchestrates FFmpeg-based detection through the `extract_scene_candidates` function.

### Scene Metric and Threshold Configuration

Sensitivity is controlled by the constant `SCENE_THRESHOLD = 0.20` defined at line 20. This threshold determines how drastic a visual change must be to register as a new scene boundary.

At lines 52-53, the engine constructs an FFmpeg `select` filter string:

```python
select='eq(n\,0)+gt(scene\,{threshold})'

```

This filter instructs FFmpeg to keep the first frame (`eq(n\,0)`) and any subsequent frame where the computed `scene` metric exceeds the threshold (`gt(scene\,{threshold})`).

### Timestamp Extraction

After invoking FFmpeg with the `showinfo` flag, the engine parses the resulting log output to extract precise timestamps for each detected shot. This parsing logic resides at lines 68-70 of [`frames.py`](https://github.com/bradautomates/claude-video/blob/main/frames.py), converting FFmpeg's verbose output into structured frame metadata.

## Fallback Logic for Static Content

Not all videos contain meaningful scene transitions. The engine implements a safeguard through `SCENE_MIN_FRAMES`, set to 8 scenes by default.

If `extract_scene_or_uniform` (lines 22-23) detects fewer than 8 scene changes, it classifies the video as "effectively static" and switches to uniform frame extraction at the user-specified FPS rate. This ensures that single-shot or low-motion content still yields a usable frame set for analysis.

## Perceptual Deduplication Pipeline

After collecting scene candidates, the engine performs post-processing to eliminate redundancy. At lines 43-45, `extract_scene_or_uniform` calls `dedupe_perceptual` to remove near-identical frames caused by slight encoding variations or minimal motion.

The deduplication step occurs before the final even-sampling phase, which adjusts the frame count to match the target budget specified by the `max_frames` parameter.

## Integration with the Watch Pipeline

While [`frames.py`](https://github.com/bradautomates/claude-video/blob/main/frames.py) contains the core engine, [`skills/watch/scripts/watch.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/watch.py) serves as the top-level orchestrator. This entry point selects between scene-based detection, keyframe extraction, or uniform sampling based on video characteristics and user preferences.

The test suite in [`tests/test_frames.py`](https://github.com/bradautomates/claude-video/blob/main/tests/test_frames.py) verifies the correctness of the scene-change detection logic across various video formats and content types.

## Using the Engine in Your Code

You can interact with the scene-change detection engine directly through Python or the command-line interface.

### Python API

Import `extract_scene_or_uniform` to leverage the engine with automatic fallback handling:

```python
from pathlib import Path
from skills.watch.scripts.frames import extract_scene_or_uniform

video_path = "example.mp4"
out_dir = Path("out/frames")

# Ask the engine to prefer scene cuts, falling back to uniform sampling.

frames, meta = extract_scene_or_uniform(
    video_path=video_path,
    out_dir=out_dir,
    fps=2.0,               # fallback fps if uniform sampling is needed

    target_frames=100,     # desired number of frames

    resolution=512,
    max_frames=100,        # hard cap for the final output

    start_seconds=None,
    end_seconds=None,
    dedup=True,
)

print("Engine used:", meta["engine"])
print("Number of frames returned:", len(frames))

```

### Command-Line Interface

Run the full extraction pipeline directly from the terminal:

```bash
python -m skills.watch.scripts.frames example.mp4 out/frames

```

## Summary

- The **scene-change detection engine** uses FFmpeg's `scene` filter with a default `SCENE_THRESHOLD` of 0.20 defined in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py).
- The filter string `select='eq(n\,0)+gt(scene\,{threshold})'` captures the first frame and all subsequent shot boundaries.
- If fewer than 8 scenes are detected, the engine falls back to uniform sampling via `extract_scene_or_uniform`.
- Perceptual deduplication via `dedupe_perceptual` removes redundant frames before final sampling.
- The [`watch.py`](https://github.com/bradautomates/claude-video/blob/main/watch.py) module orchestrates the engine alongside alternative extraction strategies.

## Frequently Asked Questions

### What threshold does Claude-Video use for scene detection?

The engine uses `SCENE_THRESHOLD = 0.20` as defined at line 20 of [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py). This value balances sensitivity to capture genuine shot changes while filtering out minor lighting fluctuations and compression artifacts.

### How does the engine handle videos without scene changes?

If the detector finds fewer than `SCENE_MIN_FRAMES` (8) scenes, the `extract_scene_or_uniform` function (lines 22-23) automatically switches to uniform frame extraction at the specified FPS rate. This ensures static or single-take videos still produce the target number of frames for analysis.

### Can I adjust the scene detection sensitivity?

While the default threshold is hardcoded at 0.20, you can modify the `SCENE_THRESHOLD` constant in [`frames.py`](https://github.com/bradautomates/claude-video/blob/main/frames.py) or extend the `extract_scene_candidates` function to accept a custom threshold parameter for the FFmpeg scene filter.

### What file formats does the scene-change detection engine support?

The engine supports any video format compatible with FFmpeg, including MP4, MOV, AVI, and MKV. Because the underlying detection relies entirely on FFmpeg's decoding and scene filtering capabilities, format support matches your local FFmpeg installation.