# How Scene Detection Works for Frame Selection in Claude Video

> Learn how Claude Video uses ffmpeg scene detection for frame selection. Discover deduplication and uniform sampling techniques for efficient video processing.

- Repository: [bradautomates/claude-video](https://github.com/bradautomates/claude-video)
- Tags: internals
- Published: 2026-07-26

---

**Claude Video detects scene changes using ffmpeg's scene metric filter, deduplicates perceptually similar frames, and falls back to uniform sampling when videos lack enough visual cuts.**

The **bradautomates/claude-video** repository implements an intelligent frame selection pipeline that prioritizes scene boundaries over fixed intervals. By analyzing video content with ffmpeg and applying post-processing filters, the system ensures that only the most visually informative frames are extracted for downstream analysis.

## Scene-Change Detection with ffmpeg

The detection logic resides in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) within the `extract_scene_candidates` function (lines 217-281). Claude Video invokes ffmpeg with the `select` filter using the expression:

```text
eq(n,0)+gt(scene,<threshold>)

```

This expression serves two purposes:

- **`eq(n,0)`** forces retention of the first frame regardless of content.
- **`gt(scene,threshold)`** selects any frame where the scene change metric exceeds the default **0.20** threshold defined by `SCENE_THRESHOLD` on line 20.

The filter simultaneously scales frames to the requested resolution and injects the `showinfo` filter so ffmpeg prints presentation timestamps (PTS) to `stderr`. These timestamps are later captured using the `SHOWINFO_TS_RE` regular expression defined on line 39.

## Timestamp Extraction and Candidate Generation

After ffmpeg execution completes, the pipeline parses the `stderr` output to extract PTS timestamps for each selected frame (lines 68-70). These values populate the `timestamp_seconds` field in the returned frame dictionaries.

The raw output from ffmpeg represents candidate frames—potential scene boundaries that have not yet been validated or deduplicated. Each candidate includes its index, extracted timestamp, and file path, forming the input for the post-processing stages.

## Post-Processing Pipeline

### Minimum Shots Guard

If the initial scan produces fewer than **8** scene cuts (`SCENE_MIN_FRAMES`), the engine assumes the video is static content (such as a screen recording) and triggers a fallback mechanism. The `extract_scene_or_uniform` function (starting at line 511) detects this condition and switches from scene-based selection to uniform temporal sampling.

### Perceptual Deduplication

Raw scene candidates often contain near-duplicate frames. The `dedupe_perceptual` function (lines 63-66) removes these by generating 16×16 grayscale thumbnails for each JPEG via `_thumb_frames`. It then calculates the mean absolute pixel difference between consecutive thumbnails. Frames with a difference ≤ **2.0** (`DEDUP_THRESHOLD` on line 38) are considered duplicates, removed from the candidate list, and deleted from disk.

### Even Sampling to Frame Budget

After deduplication, the remaining frames may still exceed the user-defined budget. The `_even_indices` helper (lines 83-92) implements `_even_sample` to select evenly-spaced indices across the candidate list. This method guarantees that the first and last frames are always retained while distributing the remaining selections uniformly throughout the video duration.

## Implementation Details and Code Example

The orchestration function `extract_scene_or_uniform` returns both the selected frames and a metadata dictionary indicating which engine was used (`scene` vs `uniform`), the number of candidates found, how many were deduplicated, and whether a fallback occurred.

```python
from pathlib import Path
from skills.watch.scripts.frames import extract_scene_or_uniform

video = Path("example.mp4")
out_dir = Path("frames_out")
out_dir.mkdir(parents=True, exist_ok=True)

# Ask for a maximum of 80 frames; let the function decide fps automatically.

frames, info = extract_scene_or_uniform(
    str(video), out_dir,
    fps=0,                     # 0 → auto-fps is computed internally

    target_frames=80,
    resolution=512,
    max_frames=80,
    start_seconds=None,
    end_seconds=None,
    dedup=True,
)

print("Engine used:", info["engine"])
for f in frames:
    print(f"{f['index']:03d} – {f['timestamp_seconds']}s – {f['reason']}")

```

Executing this snippet creates JPEGs named `frame_0000.jpg`, `frame_0001.jpg`, and so on in `frames_out/`, while printing whether the scene-detection engine succeeded or fell back to uniform sampling.

## Summary

- **Scene detection** in Claude Video relies on ffmpeg's `select` filter with a default threshold of **0.20** to identify visual cuts.
- The pipeline extracts timestamps from ffmpeg's `stderr` using regex pattern `SHOWINFO_TS_RE`, then validates that at least **8** scene cuts exist before accepting scene-based results.
- **Perceptual deduplication** uses 16×16 grayscale thumbnails and a mean-absolute-difference threshold of **2.0** to eliminate redundant frames.
- When scene cuts are insufficient, the system automatically falls back to uniform sampling via `extract_scene_or_uniform`.
- Final frame selection uses even-spaced indexing to respect the `max_frames` budget while preserving first and last frames.

## Frequently Asked Questions

### What ffmpeg expression does Claude Video use for scene detection?

Claude Video uses the expression `eq(n,0)+gt(scene,<threshold>)` within ffmpeg's `select` filter. The `eq(n,0)` component ensures the first frame is always selected, while `gt(scene,0.20)` selects frames where the scene change metric exceeds the default threshold defined by `SCENE_THRESHOLD` on line 20 of [`frames.py`](https://github.com/bradautomates/claude-video/blob/main/frames.py).

### How does Claude Video handle videos without scene changes?

When fewer than **8** scene cuts are detected (`SCENE_MIN_FRAMES`), the `extract_scene_or_uniform` function treats the video as static content and automatically falls back to uniform temporal sampling based on frames-per-second calculations rather than scene boundaries.

### What is the perceptual deduplication threshold?

The `dedupe_perceptual` function uses a `DEDUP_THRESHOLD` of **2.0** (defined on line 38), representing the maximum mean absolute pixel difference allowed between 16×16 grayscale thumbnails before consecutive frames are considered duplicates and removed from the selection.

### Which function orchestrates the entire frame extraction pipeline?

The `extract_scene_or_uniform` function starting at line 511 of [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) coordinates the complete workflow, calling `extract_scene_candidates` for detection, `dedupe_perceptual` for deduplication, and managing the fallback to uniform extraction when scene-based detection yields insufficient results.