# How Scene Detection Works for Frame Extraction in the watch Skill

> Discover how scene detection works for frame extraction in the watch skill. Learn how FFmpeg's scene filter identifies cuts and extracts key frames automatically.

- Repository: [bradautomates/claude-video](https://github.com/bradautomates/claude-video)
- Tags: deep-dive
- Published: 2026-08-06

---

**Scene detection in the `watch` skill uses FFmpeg's built-in scene filter to identify visual cuts in video content, extracting representative frames at scene boundaries while automatically falling back to uniform sampling for static videos.**

The `watch` skill in the `bradautomates/claude-video` repository intelligently extracts video frames by analyzing visual discontinuities rather than blindly sampling at fixed intervals. This content-aware approach lives primarily in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) and leverages FFmpeg's scene detection capabilities to produce meaningful thumbnails that represent actual editorial cuts in the source material.

## Scene Detection Pipeline Overview

The scene detection workflow follows a six-stage pipeline that balances accuracy with efficiency. First, FFmpeg analyzes the video stream to detect frames where the scene metric exceeds a threshold. Then the system parses timestamps, collects JPEG outputs, annotates metadata, and applies post-processing to remove duplicates and cap the final frame count. If the video lacks sufficient scene changes, the pipeline automatically switches to uniform temporal sampling to guarantee coverage.

## Core Implementation in frames.py

The heart of the scene detection logic resides in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py), specifically within the `extract_scene_candidates` function and its supporting constants.

### FFmpeg Scene Filter Configuration

The detection relies on an FFmpeg select filter string constructed at lines 52-55:

```python
select='eq(n\,0)+gt(scene\,{threshold})'

```

This filter expression emits the first frame (`eq(n,0)`) and every subsequent frame where the scene difference metric exceeds the configured threshold. The `SCENE_THRESHOLD` constant defaults to **0.20** (defined at lines 20-24), where higher values require larger visual changes to register as cuts. The command includes `showinfo` to print frame metadata to stderr:

```python
ffmpeg -i input.mp4 -vf "select='eq(n\,0)+gt(scene\,0.20)',scale=512:-1,showinfo" -q:v 2 frame_%04d.jpg

```

### Timestamp Extraction and Parsing

FFmpeg writes timestamp information to stderr during processing. The script captures this output and parses it using the `SHOWINFO_TS_RE` regex (lines 39-40) to extract absolute seconds for each candidate frame. This parsing converts FFmpeg's verbose logging into a clean list of `timestamps` that correspond one-to-one with the extracted JPEG files.

### Frame Annotation and Metadata

After FFmpeg writes the numbered JPEGs (`frame_%04d.jpg`), the script gathers them from the output directory using `frames = sorted(out_dir.glob("frame_*.jpg"))` (lines 70-71). For each frame, it constructs a metadata dictionary containing:

- `index` – sequential order in the extraction sequence
- `timestamp_seconds` – the parsed absolute timestamp
- `path` – absolute file path to the JPEG
- `reason` – either `"first-frame"` for the initial frame or `"scene-change"` for detected cuts (lines 73-80)

## Fallback and Post-Processing Logic

The higher-level `extract_scene_or_uniform` function orchestrates intelligent fallback behavior and content optimization.

### Uniform Sampling Fallback

When `extract_scene_or_uniform` calls `extract_scene_candidates`, it evaluates the results against `SCENE_MIN_FRAMES` (default **8**). If the detected scene count falls below this threshold (defined at lines 21-27 and referenced at lines 10-16), the video is considered visually static. In this case, the function abandons scene detection and invokes `extract` to perform uniform temporal sampling instead, ensuring the user receives a representative set of frames even when no cuts exist.

### Deduplication and Frame Capping

For videos with sufficient scene changes, the pipeline applies two additional optimizations. First, `dedupe_perceptual` removes near-identical frames using perceptual hashing of thumbnails. Then `_even_sample` reduces the remaining frames to the user-specified `max_frames` limit (lines 43-48). This combination ensures the final output contains diverse, evenly distributed representative images without redundancy.

## Practical Usage Examples

You can invoke the scene detection logic directly through the Python API or via the command-line interface.

### Direct API Usage

Call `extract_scene_or_uniform` from [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) to process a video with automatic fallback:

```python
from pathlib import Path
from skills.watch.scripts.frames import extract_scene_or_uniform

video = "my_video.mp4"
out_dir = Path("./frames")
fps, target = 1.5, 100  # fallback uniform params (used only if scene fails)

frames, meta = extract_scene_or_uniform(
    video_path=video,
    out_dir=out_dir,
    fps=fps,
    target_frames=target,
    resolution=512,
    max_frames=80,          # cap after dedup

    start_seconds=None,
    end_seconds=None,
    dedup=True,
)

print("Engine used:", meta["engine"])
for f in frames[:5]:
    print(f["index"], f["timestamp_seconds"], f["reason"], f["path"])

```

### Command-Line Interface

Use the high-level [`watch.py`](https://github.com/bradautomates/claude-video/blob/main/watch.py) entry point for automatic engine selection:

```bash
python -m skills.watch.scripts.watch /path/to/video.mp4 ./out \
    --max-frames 120 --resolution 640

```

The script internally invokes `extract_scene_or_uniform` when sufficient scene cuts are detected, otherwise falling back to uniform extraction via `extract`.

## Summary

- **Scene detection** in the `watch` skill leverages FFmpeg's `scene` filter to measure per-frame luma differences and identify visual cuts.
- The `SCENE_THRESHOLD` constant (0.20) and `SCENE_MIN_FRAMES` constant (8) control sensitivity and fallback behavior in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py).
- FFmpeg's `showinfo` flag provides timestamps that the script parses using `SHOWINFO_TS_RE` to create accurate frame metadata.
- The `extract_scene_or_uniform` function automatically falls back to uniform sampling when videos lack sufficient scene changes.
- Post-processing includes perceptual deduplication (`dedupe_perceptual`) and even sampling (`_even_sample`) to respect `max_frames` limits.

## Frequently Asked Questions

### What FFmpeg filter enables scene detection in the watch skill?

The implementation uses the **select filter** with the expression `eq(n,0)+gt(scene,THRESHOLD)` as defined in `extract_scene_candidates` at lines 52-55 of [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py). This filter compares consecutive frames and emits those where the scene change metric exceeds the `SCENE_THRESHOLD` value of 0.20.

### How does the system handle videos without scene cuts?

When fewer than **8** scene-change frames are detected (controlled by `SCENE_MIN_FRAMES`), the `extract_scene_or_uniform` function automatically falls back to uniform temporal sampling. This ensures static videos or single-shot content still yield a representative set of frames rather than returning empty results.

### Where are the extracted frame timestamps sourced from?

Timestamps originate from FFmpeg's `showinfo` filter, which prints frame metadata to stderr during processing. The script captures this output and parses it using the `SHOWINFO_TS_RE` regex (lines 39-40) to extract absolute seconds for each candidate frame, stored as `timestamp_seconds` in the frame metadata.

### Can the scene detection sensitivity be adjusted?

Yes. The `SCENE_THRESHOLD` constant at lines 20-24 of [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) defaults to 0.20, but you can modify this value to make detection more or less sensitive. Lower values detect subtle changes, while higher values require dramatic visual differences to trigger scene boundaries.