# How Scene Detection Falls Back to Uniform Sampling in Claude-Video

> Discover how Claude-Video's scene detection fallback ensures frame coverage by switching to uniform sampling when fewer than 8 scene cuts are detected.

- Repository: [bradautomates/claude-video](https://github.com/bradautomates/claude-video)
- Tags: internals
- Published: 2026-08-08

---

**The scene detection fallback to uniform sampling triggers automatically when FFmpeg detects fewer than 8 distinct scene cuts, switching from scene-aware extraction to duration-based uniform sampling to guarantee adequate frame coverage.**

The `bradautomates/claude-video` repository implements an intelligent dual-engine frame extraction system in its watch skill. When processing visually static content—such as screen recordings or talking-head videos—the system automatically transitions from scene-detection to uniform sampling to ensure comprehensive frame coverage regardless of content dynamics.

## How the Fallback Mechanism Works

The frame extraction logic resides in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) and operates through two distinct engines:

- **Scene engine**: Uses FFmpeg-based scene-change detection via `extract_scene_candidates` to capture visually distinct shots
- **Uniform engine**: Applies straight-line time-based sampling via `extract` for static content

The decision point occurs in the `extract_scene_or_uniform` function (lines 52-74). First, the system extracts all scene candidates across the entire clip with `max_frames=None` to obtain an uncapped count of detected cuts. If the resulting `scene_count` meets or exceeds the `SCENE_MIN_FRAMES` threshold (set to 8), the system proceeds with scene-aware deduplication and sampling. When the count falls below this threshold, the function immediately triggers the uniform fallback.

## Inside the `extract_scene_or_uniform` Function

Located in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py), this function orchestrates the selection logic through a three-phase process.

### Detecting Scene Candidates

The function begins by extracting all potential scene frames without early limiting:

```python
scene_frames = extract_scene_candidates(
    video_path,
    out_dir,
    resolution=resolution,
    max_frames=None,          # do NOT limit early – we need the *full* cut list

    start_seconds=start_seconds,
    end_seconds=end_seconds,
)
scene_count = len(scene_frames)

```

This uncapped detection ensures the system evaluates the complete visual diversity of the video before making an engine selection.

### Threshold Evaluation Against `SCENE_MIN_FRAMES`

The code evaluates the candidate count against the constant threshold:

```python
if scene_count >= SCENE_MIN_FRAMES:          # SCENE_MIN_FRAMES == 8

    # …dedup and even‑sample → final scene result

```

When the threshold is satisfied, the pipeline applies perceptual deduplication (`dedupe_perceptual`) and even sampling (`_even_sample`) to distill the scene frames down to the user's `max_frames` limit.

### Uniform Sampling Execution

When `scene_count < SCENE_MIN_FRAMES`, the function calculates a fallback cap and invokes uniform sampling:

```python
fallback_cap = target_frames if max_frames is None else min(max_frames, target_frames)
frames = extract(
    video_path,
    out_dir,
    fps=fps,                 # auto‑calculated based on duration

    resolution=resolution,
    max_frames=fallback_cap,
    start_seconds=start_seconds,
    end_seconds=end_seconds,
)

```

The `extract` function generates a uniform grid of timestamps using the `fps` value derived from `auto_fps` or `auto_fps_focus`, ensuring consistent coverage across the video duration regardless of visual staticity.

## Metadata and Observability

The function returns a metadata dictionary alongside the extracted frames, enabling callers to track which engine was used:

```python
{
    "engine": "scene"   or "uniform",
    "candidate_count": scene_count,
    "deduped_count": n_dropped,
    "selected_count": len(selected_or_frames),
    "fallback": False   or True,
}

```

When the fallback activates, the `"engine"` field reports `"uniform"` and `"fallback"` sets to `True`, providing clear telemetry for debugging and optimization.

## Practical Implementation Examples

### Direct Function Invocation

To explicitly test the fallback behavior on static content:

```python
from pathlib import Path
from skills.watch.scripts.frames import extract_scene_or_uniform, auto_fps

video = "static_slide.mp4"          # a screen‑recording with almost no cuts

out_dir = Path("/tmp/frames")
fps, _ = auto_fps(120)              # 2 fps for a 2‑minute clip

frames, meta = extract_scene_or_uniform(
    video_path=video,
    out_dir=out_dir,
    fps=fps,
    target_frames=100,               # desired max frames

    resolution=512,
    max_frames=100,
    dedup=True,
)
print(meta["engine"])   # → "uniform"

print(len(frames))      # uniform‑sampled frames

```

### CLI Usage

When running the full watch skill, the system automatically handles the fallback:

```bash
python -m skills.watch.scripts.watch \
    --detail balanced \
    --url https://example.com/video.mp4 \
    --out /tmp/watch-output

```

For static videos, the console output indicates the fallback with a line similar to:

```

(uniform) 45 frames (fallback)

```

### Unit Test Verification

The repository's test suite validates this behavior in [`tests/test_frames.py`](https://github.com/bradautomates/claude-video/blob/main/tests/test_frames.py):

```python
def test_scene_fallback_on_static_clip(static_clip: Path, tmp_path: Path):
    fps, _ = auto_fps(30)                     # short clip

    out, meta = extract_scene_or_uniform(
        video_path=str(static_clip),
        out_dir=tmp_path / "frames",
        fps=fps,
        target_frames=20,
        resolution=512,
        max_frames=20,
        dedup=False,
    )
    assert meta["engine"] == "uniform"

```

## Summary

- **Threshold-driven**: The fallback activates when `extract_scene_candidates` returns fewer than 8 frames (`SCENE_MIN_FRAMES`)
- **Automatic transition**: The `extract_scene_or_uniform` function in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) handles the switch transparently
- **Duration-aware**: Uniform sampling uses `auto_fps`-calculated frame rates to ensure proportional coverage
- **Observable**: Return metadata includes `"engine"` and `"fallback"` fields for monitoring
- **Zero-configuration**: The system requires no manual intervention to select the appropriate extraction strategy

## Frequently Asked Questions

### What triggers the scene detection fallback to uniform sampling?

The fallback triggers when the FFmpeg-based scene detector identifies fewer than 8 distinct scene cuts in the video (the `SCENE_MIN_FRAMES` constant defined in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py)). This typically occurs with static content like screen recordings, slideshows, or talking-head videos where minimal visual changes occur between frames.

### How does the uniform sampling engine calculate frame intervals?

The uniform sampler uses the `auto_fps` function to calculate an appropriate frames-per-second value based on video duration and target frame count. The `extract` function then generates evenly spaced timestamps across the video timeline, ensuring consistent temporal coverage regardless of visual content changes.

### Can I force uniform sampling even if scene detection finds enough cuts?

While `extract_scene_or_uniform` automatically selects the engine based on scene count, you can bypass scene detection entirely by calling the `extract` function directly from [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py). However, the high-level [`watch.py`](https://github.com/bradautomates/claude-video/blob/main/watch.py) interface does not expose a manual override flag, as the automatic selection optimizes for both content variety and processing efficiency.

### What metadata indicates that a fallback occurred?

The metadata dictionary returned by `extract_scene_or_uniform` contains two key indicators: the `"engine"` field will report `"uniform"` instead of `"scene"`, and the `"fallback"` boolean field will set to `True`. Additionally, `"candidate_count"` will show the number of scene cuts detected (fewer than 8), while `"selected_count"` reports the final number of uniformly sampled frames extracted.