Uniform Sampling Fallback in Claude-Video: How Scene Detection Gracefully Degrades

When scene detection finds fewer than 4 cuts, the extract_scene_or_uniform function automatically switches to uniform time-based sampling to guarantee a usable frame set.

The claude-video repository by bradautomates implements a robust frame extraction pipeline in its watch skill. The uniform sampling fallback ensures that videos always yield usable frames—even when scene detection fails, as with static screen recordings or short clips. This article explains the fallback mechanism implemented in skills/watch/scripts/frames.py, how it respects user frame budgets, and the metadata it returns for downstream processing.

How Scene Detection and Fallback Work Together

The entry point for frame extraction is extract_scene_or_uniform, which orchestrates a two-phase approach: attempt scene detection first, then fall back to uniform sampling if too few cuts are found.

Phase 1: Scene-Cut Detection

The function calls extract_scene_candidates to identify frames corresponding to visual scene changes. This produces a list of candidate timestamps where the video content shifts significantly.

Phase 2: The Minimum-Cut Check

The detected cut count is compared against SCENE_MIN_FRAMES, a constant set to 4 according to SKILL.md. The logic branches based on this threshold:

• ≥ 4 cuts: Frames are optionally deduplicated, then evenly sampled using the "scene" engine
• < 4 cuts: The uniform sampling fallback triggers automatically

The Fallback Implementation

When scene detection under-delivers, the code switches to time-based extraction. Here is the exact fallback logic from skills/watch/scripts/frames.py (lines 21–34):

fallback_cap = target_frames if max_frames is None else min(max_frames, target_frames)
frames = extract(
    video_path,
    out_dir,
    fps=fps,
    resolution=resolution,
    max_frames=fallback_cap,
    start_seconds=start_seconds,
    end_seconds=end_seconds,
)

Key Fallback Behaviors

• fallback_cap — Respects both the user-requested target_frames and any hard max_frames limit, taking the minimum of the two when both are specified
• extract — Performs simple time-based extraction at the supplied fps, producing evenly-spaced frames across the clip duration
• start_seconds / end_seconds — Honor trim boundaries even in fallback mode

Optional Perceptual Deduplication

When dedup=True, uniformly sampled frames pass through dedupe_perceptual before return. This removes near-identical frames based on perceptual thumbnail difference—useful when uniform sampling hits identical frames in static video regions.

Metadata Signaling the Fallback

The function returns a metadata dictionary that explicitly logs the fallback event:

{
  "engine": "uniform",
  "candidate_count": 2,
  "deduped_count": 0,
  "selected_count": 30,
  "fallback": true
}

Field	Meaning
engine	"uniform" or "scene" — indicates which engine produced frames
candidate_count	Scene cuts detected before fallback decision
deduped_count	Frames removed by perceptual deduplication
selected_count	Final frame count returned
fallback	Boolean flag, true only when uniform fallback activated

Practical Code Examples

Example 1: Fallback Triggered on Static Video

from pathlib import Path
from skills.watch.scripts.frames import extract_scene_or_uniform

# 30-second static screen recording with minimal visual changes

frames, meta = extract_scene_or_uniform(
    video_path="static_screen.mp4",
    out_dir=Path("frames"),
    fps=1.0,
    target_frames=100,
    resolution=512,
    max_frames=100,
    dedup=True,
)

print(meta)

# {

#   "engine": "uniform",

#   "candidate_count": 2,      # only 2 scene cuts detected

#   "deduped_count": 0,

#   "selected_count": 30,      # 30 seconds × 1 fps

#   "fallback": True,

# }

Example 2: Scene Engine Succeeds

frames, meta = extract_scene_or_uniform(
    video_path="movie_clip.mp4",
    out_dir=Path("frames"),
    fps=2.0,
    target_frames=120,
    resolution=512,
    max_frames=100,
    dedup=True,
)

assert meta["engine"] == "scene"
assert meta["fallback"] is False

Where This Logic Lives

File	Role
skills/watch/scripts/frames.py	Core implementation of extract_scene_or_uniform and the uniform fallback
skills/watch/SKILL.md	Documents the "fewer than four keyframes" fallback policy
tests/test_frames.py	Unit tests asserting meta["engine"] == "uniform" and meta["fallback"] is True
README.md	Mentions fallback behavior in detail-level descriptions

Summary

• The uniform sampling fallback activates automatically when extract_scene_candidates returns fewer than SCENE_MIN_FRAMES (4) cuts
• The fallback respects frame budgets through fallback_cap = min(max_frames, target_frames) and honors trim boundaries
• Uniform frames can be perceptually deduplicated via dedupe_perceptual when dedup=True
• Metadata returned includes engine, candidate_count, fallback flag, and final selection counts
• Tests in tests/test_frames.py verify fallback behavior for downstream consumers

Frequently Asked Questions

What triggers the uniform sampling fallback in claude-video?

The fallback triggers when scene detection finds fewer than 4 candidate cuts. This constant SCENE_MIN_FRAMES is checked in extract_scene_or_uniform within skills/watch/scripts/frames.py. Videos with minimal motion—static screen recordings, slideshows, or very short clips—commonly hit this threshold.

Does the fallback respect my frame budget and maximum limits?

Yes. The fallback_cap calculation takes min(max_frames, target_frames) when both are provided, ensuring the uniform sampler never exceeds your hard limit while attempting to reach your target. If only target_frames is specified, that becomes the cap.

How can I detect from code that the fallback occurred?

Check meta["fallback"] in the return value from extract_scene_or_uniform. This boolean is True only when uniform sampling was activated. The meta["engine"] field will also read "uniform" instead of "scene".

Is deduplication applied to uniformly sampled frames?

Yes, when dedup=True. The uniformly sampled frames are passed through dedupe_perceptual, which removes perceptually similar frames. The meta["deduped_count"] field reports how many duplicates were removed from the final set.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}