# How Claude-Video Performs Scene-Change Detection in Balanced Mode

> Discover how claude-video performs scene-change detection in balanced mode. Learn about luminance change, shot deduplication, and static content sampling for efficient video analysis.

- Repository: [bradautomates/claude-video](https://github.com/bradautomates/claude-video)
- Tags: deep-dive
- Published: 2026-07-07

---

**In balanced mode, Claude-Video uses ffmpeg's scene detection filter to identify frames with greater than 20% luminance change, deduplicates perceptually similar shots, and caps the output at 100 frames while falling back to uniform sampling for static content.**

The `bradautomates/claude-video` repository implements intelligent frame extraction through multiple detail modes. When you invoke the **balanced** detail mode, the system prioritizes **scene-change detection** to capture distinct visual moments without exceeding context window limits. This approach ensures comprehensive coverage of video content by focusing on editorial cuts and significant visual transitions.

## Engine Selection in Watch.py

The entry point for balanced mode resides in [`skills/watch/scripts/watch.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/watch.py), where the `detail` parameter determines the extraction strategy. When set to `"balanced"` (or via the `WATCH_DETAIL=balanced` environment variable), the code invokes `extract_scene_or_uniform` with a default frame budget of 100.

```python

# watch.py – engine selection (balanced → scene-aware frames)

# source: watch.py L214-L218

if detail == "balanced":
    frames, frame_meta = extract_scene_or_uniform(
        # ... parameters including target_frames=100

    )

```

This function call initiates the full scene-detection pipeline, which returns both the selected frames and metadata about the extraction process.

## FFmpeg Scene Detection Implementation

The core detection logic lives in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) within the `extract_scene_candidates` function. This implementation leverages ffmpeg's `scene` filter to identify meaningful visual boundaries.

### Scene Threshold Configuration

The system defines a **20% luminance change threshold** through the `SCENE_THRESHOLD` constant (0.20) at lines 20-21. The ffmpeg filter expression combines this threshold with scaling and showinfo filters to output relevant frames.

```python

# frames.py – ffmpeg scene-change filter

# source: frames.py L52-L55

vf = f"select='eq(n\\,0)+gt(scene\\,{threshold})',{_scale_filter(resolution)},showinfo"

```

This filter always captures the **first frame** (`eq(n,0)`) plus any frame where the scene score exceeds 0.20, ensuring you receive both the opening shot and all subsequent scene transitions.

### Candidate Frame Extraction

Each detected frame becomes a structured dictionary containing temporal and spatial metadata. The function returns a list of candidates with their capture reason marked as either `"first-frame"` or `"scene-change"`.

```python

# frames.py – candidate construction

# source: frames.py L72-L79

out.append({
    "index": i,
    "timestamp_seconds": ts,
    "path": str(path),
    "reason": "first-frame" if i == 0 else "scene-change",
})

```

## Fallback and Frame Optimization

After initial detection, the system validates whether the scene engine produced sufficient visual diversity to justify its use.

### Minimum Scene Requirements

The code checks if `scene_count` meets or exceeds `SCENE_MIN_FRAMES` (defined as 8 at lines 26-27). If the video contains fewer than 8 distinct shots—indicating static or single-scene content—the system abandons the scene engine in favor of uniform sampling. This fallback guarantees meaningful coverage even for footage without editorial cuts.

### Perceptual Deduplication

When the scene engine produces enough candidates, `dedupe_perceptual` removes near-duplicate frames using a hash difference threshold of 2.0 (`DEDUP_THRESHOLD`). This step eliminates redundant frames from gradual transitions or false positives while preserving distinct visual states.

```python

# frames.py – deduplication (when enabled)

# source: frames.py L43-L45

deduped, n_dropped = dedupe_perceptual(scene_frames) if dedup else (scene_frames, 0)

```

## Sampling and Output Constraints

Balanced mode enforces a hard cap of **100 frames** (`target_frames`) to manage token consumption. After deduplication, the `_even_sample` function distributes selections evenly across the timeline, always preserving the first and last frames to maintain temporal boundaries.

```python

# frames.py – even sampling of the deduped list

# source: frames.py L45-L48

cap = len(deduped) if max_frames is None else max_frames
selected = _even_sample(deduped, cap)

```

The final metadata object reports the engine used (`"scene"`), candidate counts, deduplication statistics, and fallback status, providing transparency into the extraction process.

## Practical Usage Examples

Run Claude-Video in balanced mode from the command line:

```bash
python -m skills.watch.scripts.watch \
    --detail balanced \
    https://www.youtube.com/watch?v=example

```

Programmatically invoke the scene engine with custom parameters:

```python
from skills.watch.scripts.frames import extract_scene_or_uniform
from pathlib import Path

video = "sample.mp4"
out_dir = Path("./frames")
target_frames = 100

frames, meta = extract_scene_or_uniform(
    video_path=video,
    out_dir=out_dir,
    fps=2.0,  # Ignored by scene engine but required

    target_frames=target_frames,
    resolution=512,
    max_frames=target_frames,
    dedup=True,
)

print("Engine:", meta["engine"])                # → "scene"

print("Detected shots:", meta["candidate_count"])
print("Frames kept:", len(frames))

```

Inspect the metadata to verify extraction behavior:

```python
print(meta)

# {

#   "engine": "scene",

#   "candidate_count": 27,

#   "deduped_count": 5,

#   "selected_count": 22,

#   "fallback": False,

# }

```

## Summary

- **Balanced mode** activates the scene engine via `extract_scene_or_uniform` in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py).
- Scene detection relies on ffmpeg's `select='gt(scene,0.20)'` filter to identify 20% luminance changes.
- The system requires a minimum of **8 distinct scenes**; otherwise, it falls back to uniform sampling.
- **Perceptual deduplication** removes similar frames using a hash threshold of 2.0 before final selection.
- Output is capped at **100 frames** by default, with even sampling applied to distribute selections temporally.

## Frequently Asked Questions

### What luminance threshold triggers scene detection in balanced mode?

Claude-Video uses a **20% luminance change threshold** (`SCENE_THRESHOLD = 0.20`) to identify distinct shots. When ffmpeg calculates a frame-to-frame difference exceeding this value via the `scene` filter, it marks that frame as a scene boundary. This threshold effectively catches editorial cuts while filtering out minor lighting fluctuations and camera noise.

### How does the system handle videos with minimal scene changes?

If the detector identifies fewer than **8 distinct scenes** (`SCENE_MIN_FRAMES`), the engine automatically falls back to uniform sampling. This ensures static videos, screen recordings, or single-shot content still provide useful frame coverage across the entire timeline. The fallback mechanism guarantees the 100-frame budget is always populated with temporally distributed visual data.

### Can I disable perceptual deduplication when using balanced mode?

Yes, you can disable deduplication by passing `dedup=False` to the `extract_scene_or_uniform` function. When enabled (the default), the system uses perceptual hashing to remove frames with hash differences of 2.0 or less. Disabling this feature preserves all scene-change candidates, which may be useful when analyzing videos with subtle visual variations that should be treated as distinct frames.

### Is the 100-frame maximum configurable in balanced mode?

Absolutely. While balanced mode defaults to 100 frames, you can override this by modifying the `target_frames` parameter in your Python code or adjusting the relevant CLI argument. However, the underlying **scene threshold** (0.20) and **minimum scene requirements** (8 frames) are constants defined in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) that require source modification to change.