Uniform Fallback Mechanism When Scene Detection Fails in claude-video

The claude-video watch skill automatically falls back to uniform frame sampling when the scene detection engine finds fewer than 8 distinct shots, ensuring robust extraction even for static or visually homogeneous videos.

The claude-video repository implements a resilient frame extraction pipeline that prioritizes scene-based sampling but guarantees coverage through a uniform fallback mechanism when scene detection fails. This dual-strategy approach prevents downstream transcription and summarization pipelines from receiving insufficient frame data, particularly when processing static clips or videos with minimal visual changes. The implementation centers on the extract_scene_or_uniform function in skills/watch/scripts/frames.py, which seamlessly switches strategies without requiring user intervention.

Scene Detection as the Primary Strategy

The watch skill first attempts to identify true scene cuts using FFmpeg’s scene filter through the extract_scene_candidates function. This engine analyzes the entire video to detect significant visual transitions, returning a list of candidate frames that represent distinct shots.

When the scene engine succeeds, the system optionally deduplicates perceptually similar frames via dedupe_perceptual and evenly samples the results down to the requested max_frames limit. This preserves the most visually significant moments while respecting output constraints.

The SCENE_MIN_FRAMES Threshold

The uniform fallback mechanism activates when the initial scene detection proves ineffective. The code defines a hard threshold SCENE_MIN_FRAMES set to 8 in skills/watch/scripts/frames.py at line 26.

If the detected scene count falls below this minimum, the system determines that scene-based sampling will not provide adequate coverage of the video content. This threshold acts as the gatekeeper between the two extraction strategies.

Implementation in extract_scene_or_uniform

The core orchestration logic resides in the extract_scene_or_uniform function (lines 521-572 of skills/watch/scripts/frames.py). This function implements a decision tree that either proceeds with scene-based extraction or triggers the uniform fallback based on the scene count.

Phase 1: Scene Detection and Validation

First, the function calls extract_scene_candidates to process the entire clip without capping the duration. It stores the resulting count in scene_count, then immediately evaluates it against SCENE_MIN_FRAMES:

• If scene_count >= 8: The function proceeds with scene-based sampling, marking metadata as "engine": "scene" and "fallback": False.
• If scene_count < 8: The function triggers the uniform fallback path.

Phase 2: Uniform Sampling Execution

When the fallback activates, the function calculates a fallback_cap that respects both the user-requested target_frames and any max_frames constraints. It then invokes the generic extract routine, which decodes the video at a constant FPS to pull frames uniformly across the full temporal interval.

This approach guarantees complete coverage of the entire clip duration, avoiding the pitfall where a capped scene detector might stop early and miss the tail of a long video.

Phase 3: Deduplication and Metadata

The fallback path optionally applies dedupe_perceptual to remove near-identical frames before returning the final set. The function returns the uniformly sampled frames with metadata explicitly marking "engine": "uniform" and "fallback": True, providing clear observability into which extraction strategy succeeded.

Practical Example: Triggering the Fallback

The following example demonstrates how a static video automatically triggers the uniform fallback:

from pathlib import Path
from skills.watch.scripts.frames import extract_scene_or_uniform

video_path = "example/static_clip.mp4"          # A video with little motion

out_dir = Path("tmp/frames")
fps = 1.0                                        # Desired uniform FPS

target_frames = 20                              # Desired number of frames

resolution = 512

# The call automatically falls back to uniform sampling if scene detection fails.

frames, meta = extract_scene_or_uniform(
    video_path,
    out_dir,
    fps=fps,
    target_frames=target_frames,
    resolution=resolution,
    max_frames=None,          # No hard cap; fallback will respect target_frames

    dedup=True,
)

print("Engine used:", meta["engine"])          # → "uniform"

print("Did we fall back?", meta["fallback"])   # → True

print("Number of frames returned:", len(frames))

For a scene-rich video, the same function would return "engine": "scene" and "fallback": False, demonstrating the seamless switch between strategies.

Summary

• The uniform fallback mechanism triggers when fewer than 8 scenes are detected, as defined by SCENE_MIN_FRAMES in skills/watch/scripts/frames.py.
• The extract_scene_or_uniform function orchestrates the decision logic at lines 521-572, automatically switching strategies based on scene count.
• Uniform sampling guarantees full video coverage by extracting frames at constant intervals across the entire duration, preventing data loss in static videos.
• Metadata flags ("engine" and "fallback") provide clear observability into which extraction method succeeded.
• The orchestrating code in skills/watch/scripts/watch.py consumes these results to ensure downstream pipelines always receive sufficient frames regardless of visual complexity.

Frequently Asked Questions

What triggers the uniform fallback mechanism in claude-video?

The fallback activates when the scene detection engine identifies fewer than 8 distinct scenes, as defined by the SCENE_MIN_FRAMES constant. This threshold ensures that videos with minimal visual changes—such as static lectures or slideshows—still yield useful frame samples rather than empty or sparse results that would compromise downstream transcription quality.

How does the uniform fallback ensure complete video coverage?

Unlike scene detection, which might concentrate samples in early portions if few cuts exist, the uniform sampler calculates intervals across the entire duration. The extract routine decodes the video at a constant FPS derived from the target_frames parameter, guaranteeing that returned frames span from the first second to the last, regardless of content complexity or length.

Can I detect whether the fallback was used programmatically?

Yes. The extract_scene_or_uniform function returns a metadata dictionary alongside the frame list. When the fallback executes, meta["engine"] equals "uniform" and meta["fallback"] is True. For scene-based extraction, meta["engine"] equals "scene" and meta["fallback"] is False, allowing your application to log or adjust processing based on the extraction method.

Where is the fallback logic implemented in the source code?

The primary implementation resides in skills/watch/scripts/frames.py, specifically within the extract_scene_or_uniform function (lines 521-572). The threshold constant SCENE_MIN_FRAMES is defined at line 26 of the same file. The test suite in tests/test_frames.py validates this behavior on static clips to ensure correct metadata reporting and frame coverage.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}