# Uniform Fallback Mechanism When Scene Detection Fails in claude-video

> Discover claude-video's uniform fallback mechanism. Learn how it ensures robust extraction via frame sampling when scene detection fails, even for static videos.

- Repository: [bradautomates/claude-video](https://github.com/bradautomates/claude-video)
- Tags: how-to-guide
- Published: 2026-07-19

---

**The claude-video watch skill automatically falls back to uniform frame sampling when the scene detection engine finds fewer than 8 distinct shots, ensuring robust extraction even for static or visually homogeneous videos.**

The `claude-video` repository implements a resilient frame extraction pipeline that prioritizes scene-based sampling but guarantees coverage through a **uniform fallback mechanism when scene detection fails**. This dual-strategy approach prevents downstream transcription and summarization pipelines from receiving insufficient frame data, particularly when processing static clips or videos with minimal visual changes. The implementation centers on the `extract_scene_or_uniform` function in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py), which seamlessly switches strategies without requiring user intervention.

## Scene Detection as the Primary Strategy

The watch skill first attempts to identify true scene cuts using FFmpeg’s `scene` filter through the `extract_scene_candidates` function. This engine analyzes the entire video to detect significant visual transitions, returning a list of candidate frames that represent distinct shots.

When the scene engine succeeds, the system optionally deduplicates perceptually similar frames via `dedupe_perceptual` and evenly samples the results down to the requested `max_frames` limit. This preserves the most visually significant moments while respecting output constraints.

## The SCENE_MIN_FRAMES Threshold

The uniform fallback mechanism activates when the initial scene detection proves ineffective. The code defines a hard threshold **`SCENE_MIN_FRAMES`** set to `8` in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) at line 26. 

If the detected scene count falls below this minimum, the system determines that scene-based sampling will not provide adequate coverage of the video content. This threshold acts as the gatekeeper between the two extraction strategies.

## Implementation in `extract_scene_or_uniform`

The core orchestration logic resides in the `extract_scene_or_uniform` function (lines 521-572 of [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py)). This function implements a decision tree that either proceeds with scene-based extraction or triggers the uniform fallback based on the scene count.

### Phase 1: Scene Detection and Validation

First, the function calls `extract_scene_candidates` to process the entire clip without capping the duration. It stores the resulting count in `scene_count`, then immediately evaluates it against `SCENE_MIN_FRAMES`:

- If `scene_count >= 8`: The function proceeds with scene-based sampling, marking metadata as `"engine": "scene"` and `"fallback": False`.
- If `scene_count < 8`: The function triggers the uniform fallback path.

### Phase 2: Uniform Sampling Execution

When the fallback activates, the function calculates a `fallback_cap` that respects both the user-requested `target_frames` and any `max_frames` constraints. It then invokes the generic **`extract`** routine, which decodes the video at a constant FPS to pull frames uniformly across the full temporal interval.

This approach guarantees complete coverage of the entire clip duration, avoiding the pitfall where a capped scene detector might stop early and miss the tail of a long video.

### Phase 3: Deduplication and Metadata

The fallback path optionally applies `dedupe_perceptual` to remove near-identical frames before returning the final set. The function returns the uniformly sampled frames with metadata explicitly marking `"engine": "uniform"` and `"fallback": True`, providing clear observability into which extraction strategy succeeded.

## Practical Example: Triggering the Fallback

The following example demonstrates how a static video automatically triggers the uniform fallback:

```python
from pathlib import Path
from skills.watch.scripts.frames import extract_scene_or_uniform

video_path = "example/static_clip.mp4"          # A video with little motion

out_dir = Path("tmp/frames")
fps = 1.0                                        # Desired uniform FPS

target_frames = 20                              # Desired number of frames

resolution = 512

# The call automatically falls back to uniform sampling if scene detection fails.

frames, meta = extract_scene_or_uniform(
    video_path,
    out_dir,
    fps=fps,
    target_frames=target_frames,
    resolution=resolution,
    max_frames=None,          # No hard cap; fallback will respect target_frames

    dedup=True,
)

print("Engine used:", meta["engine"])          # → "uniform"

print("Did we fall back?", meta["fallback"])   # → True

print("Number of frames returned:", len(frames))

```

For a scene-rich video, the same function would return `"engine": "scene"` and `"fallback": False`, demonstrating the seamless switch between strategies.

## Summary

- The uniform fallback mechanism triggers when fewer than **8 scenes** are detected, as defined by `SCENE_MIN_FRAMES` in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py).
- The `extract_scene_or_uniform` function orchestrates the decision logic at lines 521-572, automatically switching strategies based on scene count.
- **Uniform sampling** guarantees full video coverage by extracting frames at constant intervals across the entire duration, preventing data loss in static videos.
- Metadata flags (`"engine"` and `"fallback"`) provide clear observability into which extraction method succeeded.
- The orchestrating code in [`skills/watch/scripts/watch.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/watch.py) consumes these results to ensure downstream pipelines always receive sufficient frames regardless of visual complexity.

## Frequently Asked Questions

### What triggers the uniform fallback mechanism in claude-video?

The fallback activates when the scene detection engine identifies fewer than 8 distinct scenes, as defined by the `SCENE_MIN_FRAMES` constant. This threshold ensures that videos with minimal visual changes—such as static lectures or slideshows—still yield useful frame samples rather than empty or sparse results that would compromise downstream transcription quality.

### How does the uniform fallback ensure complete video coverage?

Unlike scene detection, which might concentrate samples in early portions if few cuts exist, the uniform sampler calculates intervals across the entire duration. The `extract` routine decodes the video at a constant FPS derived from the `target_frames` parameter, guaranteeing that returned frames span from the first second to the last, regardless of content complexity or length.

### Can I detect whether the fallback was used programmatically?

Yes. The `extract_scene_or_uniform` function returns a metadata dictionary alongside the frame list. When the fallback executes, `meta["engine"]` equals `"uniform"` and `meta["fallback"]` is `True`. For scene-based extraction, `meta["engine"]` equals `"scene"` and `meta["fallback"]` is `False`, allowing your application to log or adjust processing based on the extraction method.

### Where is the fallback logic implemented in the source code?

The primary implementation resides in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py), specifically within the `extract_scene_or_uniform` function (lines 521-572). The threshold constant `SCENE_MIN_FRAMES` is defined at line 26 of the same file. The test suite in [`tests/test_frames.py`](https://github.com/bradautomates/claude-video/blob/main/tests/test_frames.py) validates this behavior on static clips to ensure correct metadata reporting and frame coverage.