# How Claude Video Detects Scene Changes in Balanced Detail Mode

> Discover Claude Video's balanced mode scene change detection. Learn how it uses FFmpeg filters and perceptual deduplication to identify scene shifts with optimal detail.

- Repository: [bradautomates/claude-video](https://github.com/bradautomates/claude-video)
- Tags: internals
- Published: 2026-07-28

---

**In balanced detail mode, Claude Video uses FFmpeg's `select='gt(scene,0.20)'` filter to detect luminance changes exceeding 20%, falls back to uniform sampling for static videos with fewer than 8 detected shots, deduplicates perceptually similar frames, and caps the output at 100 frames.**

Claude Video's `balanced` detail mode intelligently extracts **scene-aware frames** by analyzing video content rather than sampling at fixed intervals. According to the source code in the `bradautomates/claude-video` repository, this mode orchestrates a multi-stage pipeline involving FFmpeg scene detection, perceptual deduplication, and intelligent fallback mechanisms to ensure comprehensive video coverage.

## How Balanced Mode Selects the Scene Engine

The `watch` command initiates **scene-aware frame extraction** when the user specifies `detail=balanced` or sets the environment variable `WATCH_DETAIL=balanced`. In [`skills/watch/scripts/watch.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/watch.py), the engine selection logic routes balanced mode to the scene detection pipeline with a default frame cap of 100.

```python

# watch.py – engine selection (balanced → scene-aware frames)

# source: watch.py L214-L218

if detail == "balanced":
    frames, frame_meta = extract_scene_or_uniform(
        video_path=local_path,
        out_dir=frame_dir,
        fps=fps,
        target_frames=target_frames,
        resolution=resolution,
        max_frames=target_frames,
        dedup=True,
    )

```

This invocation triggers `extract_scene_or_uniform` in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py), which implements the complete detection workflow.

## FFmpeg Scene-Change Detection Pipeline

The core detection relies on FFmpeg's **scene change filter**. The `extract_scene_candidates` function constructs a video filter that selects the first frame plus every frame where the scene score exceeds the threshold.

```python

# frames.py – ffmpeg scene-change filter

# source: frames.py L52-L55

vf = f"select='eq(n\\,0)+gt(scene\\,{threshold})',{_scale_filter(resolution)},showinfo"

```

**`SCENE_THRESHOLD`** is defined as **0.20** (20% luminance change) in the configuration section of [`frames.py`](https://github.com/bradautomates/claude-video/blob/main/frames.py#L20-L21). This threshold determines the sensitivity of cut detection—higher values require more dramatic visual changes to register as new scenes.

The FFmpeg filter outputs frame metadata including timestamps, which the parser converts into structured candidate objects for downstream processing.

## Candidate Collection and Frame Metadata

After FFmpeg processing, `extract_scene_candidates` constructs a list of dictionaries containing frame metadata. Each entry includes the frame index, timestamp, file path, and a **reason** field indicating whether the frame represents the first frame or a detected scene change.

```python

# frames.py – candidate construction

# source: frames.py L72-L79

out.append({
    "index": i,
    "timestamp_seconds": ts,
    "path": str(path),
    "reason": "first-frame" if i == 0 else "scene-change",
})

```

This metadata persists through the pipeline and appears in the final extraction report, allowing users to trace which frames represent actual scene boundaries versus uniform samples.

## Fallback Logic for Static Content

The balanced mode implements **adaptive engine selection** to handle static or low-motion videos. After detecting scene candidates, the algorithm evaluates the total shot count against **`SCENE_MIN_FRAMES`** (defined as **8** in [`frames.py`](https://github.com/bradautomates/claude-video/blob/main/frames.py#L26-L27).

If `scene_count >= 8`, the system proceeds with the scene engine. Otherwise, it falls back to **uniform sampling** (evenly spaced frames) to guarantee coverage for videos lacking distinct cuts. This ensures that static presentations or single-shot recordings still yield useful frame extracts rather than empty results.

## Perceptual Deduplication and Frame Capping

When the scene engine remains active, the pipeline invokes **`dedupe_perceptual`** to remove near-identical frames. This function compares perceptual hashes and drops frames with a difference ≤ **`DEDUP_THRESHOLD`** (2.0), preventing multiple captures from the same scene transition.

```python

# frames.py – deduplication (when enabled)

# source: frames.py L43-L45

deduped, n_dropped = dedupe_perceptual(scene_frames) if dedup else (scene_frames, 0)

```

Following deduplication, **`_even_sample`** enforces the **100-frame cap** (configurable via `target_frames`) by evenly distributing selections across the remaining frames while preserving the first and last frames. This sampling ensures temporal coverage without exceeding API token budgets.

The function returns comprehensive metadata including the engine used (`"scene"`), candidate count, deduplication statistics, and fallback status, which [`watch.py`](https://github.com/bradautomates/claude-video/blob/main/watch.py) renders in the final report (lines L886-L898).

## Practical Usage Examples

**Running Claude Video in balanced mode from the command line:**

```bash
python -m skills.watch.scripts.watch \
    --detail balanced \
    https://www.youtube.com/watch?v=example

```

**Programmatic use of the scene engine:**

```python
from skills.watch.scripts.frames import extract_scene_or_uniform
from pathlib import Path

video = "sample.mp4"
out_dir = Path("./frames")
fps, _ = 2.0, None            # fps is ignored for the scene engine

target_frames = 100           # balanced default cap

frames, meta = extract_scene_or_uniform(
    video_path=video,
    out_dir=out_dir,
    fps=fps,
    target_frames=target_frames,
    resolution=512,
    max_frames=target_frames,
    dedup=True,
)

print("Engine:", meta["engine"])                # → "scene"

print("Detected shots:", meta["candidate_count"])
print("Frames kept:", len(frames))

```

**Inspecting the metadata:**

```python
print(meta)

# {

#   "engine": "scene",

#   "candidate_count": 27,

#   "deduped_count": 5,

#   "selected_count": 22,

#   "fallback": False,

# }

```

## Summary

- **Balanced mode** triggers FFmpeg scene detection with a **20% luminance threshold** (`SCENE_THRESHOLD=0.20`) via the `select='gt(scene,0.20)'` filter.
- The system requires a minimum of **8 detected shots** (`SCENE_MIN_FRAMES=8`) to use the scene engine; otherwise, it falls back to uniform sampling.
- **Perceptual deduplication** removes redundant frames with a hash difference threshold of 2.0 to avoid capturing multiple frames from the same cut.
- Output is capped at **100 frames** by default, with even sampling applied after deduplication to maximize temporal coverage.
- All logic resides in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) and is invoked from [`skills/watch/scripts/watch.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/watch.py) when `detail=balanced` is specified.

## Frequently Asked Questions

### What threshold does Claude Video use for scene detection?

Claude Video uses a **20% luminance change threshold** (0.20) for scene detection in balanced mode. This value is defined as `SCENE_THRESHOLD` in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py#L20-L21) and passed to FFmpeg's `select='gt(scene,0.20)'` filter.

### When does balanced mode fall back to uniform sampling?

Balanced mode falls back to uniform sampling when the video contains fewer than **8 distinct scene changes** (`SCENE_MIN_FRAMES=8`). This check occurs in `extract_scene_or_uniform` after candidate detection, ensuring static videos or single-shot content still yield usable frame extracts.

### How does the deduplication work in balanced mode?

After scene detection, the `dedupe_perceptual` function calculates perceptual hashes for each frame and removes duplicates where the hash difference is **≤ 2.0** (`DEDUP_THRESHOLD`). This eliminates redundant frames from the same scene transition while preserving visually distinct content.

### What is the maximum number of frames extracted in balanced mode?

Balanced mode caps extraction at **100 frames** by default, configurable via the `target_frames` parameter. After deduplication, the `_even_sample` function distributes selections evenly across the timeline to respect this cap while maintaining first and last frame coverage.