# How the Scene-Detection Algorithm Works in frames.py: A Technical Deep Dive

> Explore the scene-detection algorithm in frames.py. Learn how it uses ffmpeg, parses timestamps, deduplicates frames, and handles fallback sampling. Understand video scene analysis.

- Repository: [bradautomates/claude-video](https://github.com/bradautomates/claude-video)
- Tags: deep-dive
- Published: 2026-07-12

---

**The scene-detection algorithm in [`frames.py`](https://github.com/bradautomates/claude-video/blob/main/frames.py) uses ffmpeg's `scene` filter to identify candidate frames, parses timestamps from stderr output, applies perceptual deduplication to remove near-duplicates, and automatically falls back to uniform sampling if fewer than 8 scene changes are detected.**

The `claude-video` repository by bradautomates provides intelligent video processing capabilities for Claude hosts. At the heart of its frame extraction system lies [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py), which implements a robust **scene-detection algorithm** that balances content-aware accuracy with reliable fallback mechanisms for static or screen-recording content.

## The Three-Step Scene-Detection Pipeline

The algorithm coordinates three distinct phases to extract representative frames from video files.

### Step 1: Detecting Raw Scene Changes with ffmpeg

The process begins in `extract_scene_candidates()` where the code constructs an ffmpeg command using the `select` filter. The filter expression at **[lines 52-57](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py#L52-L57)** combines two conditions:

```python
vf = f"select='eq(n\\,0)+gt(scene\\,{threshold})',{_scale_filter(resolution)},showinfo"

```

- **`eq(n,0)`** ensures the first frame is always captured, guaranteeing a start-point for the sequence.
- **`gt(scene,threshold)`** activates ffmpeg's built-in scene-change detector, comparing consecutive frames against the **SCENE_THRESHOLD** constant of **0.20** defined at **[line 20](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py#L20)**.
- **`showinfo`** emits metadata to stderr, enabling timestamp extraction for each candidate frame.

The `_scale_filter()` helper (referenced at **[lines 42-46](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py#L42-L46)**) limits the extraction resolution to optimize performance before the scene detection runs.

### Step 2: Parsing Timestamps and Building Candidates

After ffmpeg executes, the algorithm processes the stderr stream to extract temporal data. The regular expression **`SHOWINFO_TS_RE`** defined at **[lines 39-40](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py#L39-L40)** parses the `pts_time` values from lines like `... pts_time:12.345 ...`.

The function pairs these timestamps with the saved JPEG files (named `frame_*.jpg`) to construct the initial candidate list at **[lines 72-80](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py#L72-L80)**. This step preserves the first frame unconditionally while maintaining the chronological order of detected scene changes.

### Step 3: Perceptual Deduplication and Final Selection

Raw scene detection often produces near-duplicate frames from subtle lighting changes or minor camera movements. The **`dedupe_perceptual()`** function (implemented at **[lines 64-70](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py#L64-L70)**) removes redundancies through the following process:

1. **Downscaling**: Each frame is converted to a 16×16 grayscale thumbnail using the **DEDUP_THUMB** constant of **16** defined at **[line 37](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py#L37)**.
2. **Difference calculation**: The algorithm computes the mean absolute per-pixel difference (`_frame_delta`) between successive thumbnails.
3. **Threshold filtering**: Any frame with a difference ≤ **DEDUP_THRESHOLD** (2.0) is considered a duplicate and discarded.

If the requested `max_frames` is fewer than the deduplicated candidates, the **`_even_indices()`** helper at **[lines 84-92](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py#L84-L92)** performs even-sampling, always retaining the first and last frames while distributing selections evenly across the timeline.

## Fallback to Uniform Sampling

The algorithm implements a reliability guard through `extract_scene_or_uniform()`. At **[lines 41-53](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py#L41-L53)**, the code checks if the raw detection produces at least **SCENE_MIN_FRAMES** (8) distinct shots, defined at **[lines 26-27](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py#L26-L27)**.

If the scene engine detects fewer than 8 candidates, the function triggers the fallback mechanism at **[lines 54-73](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py#L54-L73)**, switching to the simpler `extract()` function for uniform fps-based extraction. This ensures that static videos, screen recordings, or content with minimal visual changes still produce a useful frame set.

## Implementation Details and Code Examples

To extract frames using the scene-detection pipeline with automatic fallback:

```python
from pathlib import Path
from skills.watch.scripts.frames import extract_scene_or_uniform

video_path = "example.mp4"
out_dir = Path("out_frames")
out_dir.mkdir(parents=True, exist_ok=True)

# Request maximum 50 frames at 512px resolution

frames, meta = extract_scene_or_uniform(
    video_path=video_path,
    out_dir=out_dir,
    fps=2.0,                     # fallback fps (used only if scene detection fails)

    target_frames=50,
    resolution=512,
    max_frames=50,
    start_seconds=None,
    end_seconds=None,
    dedup=True,
)

print("Engine used:", meta["engine"])          # → "scene" or "uniform"

print("Selected frames:", len(frames))
for f in frames[:5]:
    print(f["index"], f["timestamp_seconds"], f["reason"])

```

For direct scene-only extraction without fallback or caps:

```python
from skills.watch.scripts.frames import extract_scene_candidates

candidates = extract_scene_candidates(
    video_path="example.mp4",
    out_dir=Path("scene_frames"),
    resolution=512,
    max_frames=None,               # detect every cut, no early stop

    start_seconds=None,
    end_seconds=None,
    threshold=0.20,                # match repository default

)

print(f"Detected {len(candidates)} raw scene changes")

```

## Summary

- **[`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py)** implements a hybrid scene-detection algorithm using ffmpeg's `scene` filter with a default **0.20** threshold.
- The pipeline extracts timestamps via regex parsing of ffmpeg's `showinfo` stderr output.
- **Perceptual deduplication** uses 16×16 grayscale thumbnails and a mean absolute difference threshold of **2.0** to remove near-duplicates.
- The algorithm requires at least **8** scene changes (**SCENE_MIN_FRAMES**) to use scene-based extraction; otherwise, it falls back to uniform fps-based sampling.
- Even-sampling logic ensures representative frame distribution when capping the output count.

## Frequently Asked Questions

### What ffmpeg filter does the scene detection use?

The algorithm uses ffmpeg's `select` filter with the expression `eq(n,0)+gt(scene,threshold)` combined with `showinfo`. The `eq(n,0)` component ensures the first frame is always captured, while `gt(scene,threshold)` compares consecutive frames using ffmpeg's built-in scene-change detection score. The default threshold is **0.20** as defined by **SCENE_THRESHOLD** at line 20.

### How does the algorithm handle videos with few scene changes?

If the detection finds fewer than **SCENE_MIN_FRAMES** (8) candidates, the `extract_scene_or_uniform()` function automatically falls back to uniform sampling. This fallback uses fps-based extraction to guarantee usable output for static content, screen recordings, or videos with minimal visual variation.

### What is perceptual deduplication and why is it needed?

Perceptual deduplication removes near-identical frames that pass the initial scene detection but represent the same visual content. The `dedupe_perceptual()` function downscales frames to 16×16 grayscale thumbnails and calculates the mean absolute difference between successive frames. Frames with a difference ≤ **2.0** are discarded, ensuring only visually distinct shots remain in the final set.

### How are timestamps extracted from ffmpeg output?

The algorithm captures ffmpeg's stderr stream, which contains `showinfo` metadata lines including `pts_time` values. The regular expression **SHOWINFO_TS_RE** defined at lines 39-40 parses these timestamps, which are then paired with the extracted JPEG files to create temporally accurate frame metadata.