# Claude Video Scene-Change Detection Algorithm: Technical Deep Dive

> Explore the Claude Video scene-change detection algorithm. Learn how it uses ffmpeg, perceptual deduplication, and sampling for efficient video analysis. Read the technical deep dive.

- Repository: [bradautomates/claude-video](https://github.com/bradautomates/claude-video)
- Tags: deep-dive
- Published: 2026-07-29

---

**Claude Video detects scene changes by using ffmpeg's built-in scene filter with a configurable 0.20 threshold, followed by perceptual deduplication of 16×16 grayscale thumbnails and even sampling to respect token budgets.**

The scene-change detection algorithm powers the `/watch` slash command in the `bradautomates/claude-video` repository, enabling efficient video analysis by extracting only visually significant frames. Implemented primarily in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py), this system balances computational cost against coverage through three distinct detail modes.

## Core Algorithm Components

The algorithm relies on ffmpeg's native scene detection capabilities combined with post-processing filters to eliminate redundancy and enforce frame caps.

### Scene Threshold and Constants

At the heart of the implementation are several hardcoded constants defined in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py):

- **`SCENE_THRESHOLD = 0.20`** – The minimum scene-change score required for ffmpeg to flag a frame as a cut point
- **`SCENE_MIN_FRAMES = 8`** – The fallback threshold; if fewer than 8 scene changes are detected, the system switches to uniform sampling
- **`MAX_READ_DIMENSION = 1998`** – Maximum JPEG height constraint for processing

These values determine the sensitivity of detection and the minimum viable granularity for scene-based extraction.

### Frame Extraction Pipeline

The primary entry point `extract_scene_candidates` invokes ffmpeg with a complex filtergraph:

```bash
ffmpeg -i input.mp4 -vf "select='eq(n\,0)+gt(scene,0.20)',scale=...,showinfo" ...

```

This filter expression performs two critical functions:
1. **`eq(n\,0)`** – Always selects the first frame to ensure coverage from the start
2. **`gt(scene,0.20)`** – Selects any frame where the scene-change score exceeds the threshold

The `showinfo` filter emits timestamp data, which the algorithm parses using the regex pattern `SHOWINFO_TS_RE` to map frame indices to precise temporal positions.

### Perceptual Deduplication

After candidate extraction, the `dedupe_perceptual` function eliminates near-identical frames that may result from minor encoding variations or static scenes with subtle noise.

The deduplication process:
1. Resizes each JPEG to a **16×16 grayscale thumbnail** via `_thumb_frames`
2. Computes the mean absolute difference between consecutive thumbnails
3. Discards frames where the difference is **≤ 2.0** (`DEDUP_THRESHOLD`)

This perceptual hashing approach ensures that only visually distinct shots are retained, preventing token waste on redundant imagery.

### Even Sampling Strategy

When the candidate count exceeds the requested budget, the `_even_indices` utility (used by `_even_sample`) distributes selections evenly across the timeline:

- Always preserves the **first and last frames** to maintain temporal boundaries
- Calculates step intervals to spread remaining selections uniformly
- Respects the `max_frames` cap specified by the detail mode

This guarantees comprehensive coverage regardless of video length or scene density.

## Implementation in frames.py

The orchestration logic resides in `extract_scene_or_uniform`, which implements the decision tree for extraction strategies:

```python

# From skills/watch/scripts/frames.py

def extract_scene_or_uniform(
    video_path,
    out_dir,
    fps=None,
    target_frames=None,
    resolution=512,
    max_frames=None,
    start_seconds=None,
    end_seconds=None,
    dedup=True,
):
    # 1. Attempt scene-based extraction (uncapped initially)

    candidates = extract_scene_candidates(
        video_path, out_dir, resolution, start_seconds, end_seconds
    )
    
    # 2. Fallback check

    if len(candidates) < SCENE_MIN_FRAMES:
        # Switch to uniform sampling for static content

        return extract(video_path, out_dir, fps, target_frames, resolution)
    
    # 3. Optional deduplication

    if dedup:
        candidates = dedupe_perceptual(candidates)
    
    # 4. Apply frame cap via even sampling

    if max_frames and len(candidates) > max_frames:
        indices = _even_indices(len(candidates), max_frames)
        candidates = [candidates[i] for i in indices]
    
    return candidates

```

This function returns a list of dictionaries containing `index`, `timestamp_seconds`, `path`, and `reason` for each selected frame, along with metadata about the engine used and deduplication statistics.

## Detail Modes and Configuration

The repository exposes three extraction strategies via the `/watch` command, configured through the `detail` parameter:

| Mode | Frame Source | Cap | Engine | Use Case |
|------|--------------|-----|--------|----------|
| **efficient** | ffmpeg key-frames | 50 | `keyframe` | Quick previews, low token usage |
| **balanced** (default) | Scene-change frames | 100 | `scene` | General analysis with fallback |
| **token-burner** | Scene-change frames | Uncapped | `scene` | Maximum detail, highest coverage |

The **balanced** mode implements intelligent fallback logic: if `extract_scene_candidates` returns fewer than 8 frames (indicating a static video like a screen recording or talking-head), the system automatically switches to uniform sampling to ensure adequate representation.

## Usage Examples

### Python Integration

Clients invoke the scene detection through the high-level API in [`skills/watch/scripts/watch.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/watch.py):

```python
from skills.watch.scripts.frames import extract_scene_or_uniform, auto_fps

# Calculate target FPS based on duration and cap

fps, target = auto_fps(duration_seconds, max_frames=100)

frames, frame_meta = extract_scene_or_uniform(
    video_path="lecture.mp4",
    out_dir="./frames",
    fps=fps,
    target_frames=target,
    resolution=512,
    max_frames=100,  # Set to None for token-burner mode

    dedup=True,
)

print(f"Extracted {len(frames)} frames using {frame_meta['engine']}")

```

### Command-Line Execution

For debugging or standalone usage, run the module directly:

```bash
python -m skills.watch/scripts/frames.py \
    input.mp4 \
    ./output \
    --fps 2.0 \
    --max-frames 100 \
    --resolution 512

```

Add `--no-dedup` to disable perceptual deduplication and retain all scene-change candidates.

## Summary

- **Claude Video scene-change detection** leverages ffmpeg's native `scene` filter with a default threshold of 0.20 to identify significant visual transitions
- The algorithm in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) implements a three-stage pipeline: candidate extraction, 16×16 grayscale perceptual deduplication, and even sampling to enforce frame caps
- **Automatic fallback** to uniform sampling occurs when fewer than 8 scene changes are detected, preventing poor coverage on static content
- Three detail modes (efficient, balanced, token-burner) allow users to trade token consumption against analysis granularity
- All frames retain precise timestamp metadata extracted from ffmpeg's `showinfo` output, enabling accurate temporal referencing

## Frequently Asked Questions

### How does the scene-change detection handle videos with minimal motion?

If the algorithm detects fewer than `SCENE_MIN_FRAMES` (8) scene changes, it deems the video "effectively static" and automatically falls back to uniform sampling via the `extract` function. This prevents situations where screen recordings or single-shot interviews would yield only 1-2 frames while respecting the token budget through evenly spaced temporal sampling.

### What is the perceptual deduplication threshold and how does it work?

The `dedupe_perceptual` function resizes each candidate frame to a 16×16 grayscale thumbnail and calculates the mean absolute difference between consecutive frames. If the difference is ≤ 2.0 (`DEDUP_THRESHOLD`), the frames are considered visually identical and the latter is discarded. This removes redundant frames caused by encoding artifacts or subtle lighting changes without removing genuinely distinct content.

### Can I adjust the scene-change sensitivity threshold?

The `SCENE_THRESHOLD` constant is hardcoded to 0.20 in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py). While the current implementation does not expose this as a command-line parameter, you can modify the constant directly in the source or patch the `extract_scene_candidates` function to accept a custom threshold value for the `gt(scene,THRESH)` ffmpeg filter expression.

### How does the token-burner mode differ from balanced mode?

Both modes use the scene-change detection engine, but **balanced** mode applies a hard cap of 100 frames (`detail_cap=100`) and enables automatic fallback to uniform sampling. **Token-burner** mode sets `max_frames=None`, allowing unlimited scene-change candidates (subject only to deduplication), making it suitable for detailed analysis of complex, fast-cutting content where maximum visual coverage justifies higher token consumption.