How Scene Detection Falls Back to Uniform Sampling in Claude-Video

The scene detection fallback to uniform sampling triggers automatically when FFmpeg detects fewer than 8 distinct scene cuts, switching from scene-aware extraction to duration-based uniform sampling to guarantee adequate frame coverage.

The bradautomates/claude-video repository implements an intelligent dual-engine frame extraction system in its watch skill. When processing visually static content—such as screen recordings or talking-head videos—the system automatically transitions from scene-detection to uniform sampling to ensure comprehensive frame coverage regardless of content dynamics.

How the Fallback Mechanism Works

The frame extraction logic resides in skills/watch/scripts/frames.py and operates through two distinct engines:

• Scene engine: Uses FFmpeg-based scene-change detection via extract_scene_candidates to capture visually distinct shots
• Uniform engine: Applies straight-line time-based sampling via extract for static content

The decision point occurs in the extract_scene_or_uniform function (lines 52-74). First, the system extracts all scene candidates across the entire clip with max_frames=None to obtain an uncapped count of detected cuts. If the resulting scene_count meets or exceeds the SCENE_MIN_FRAMES threshold (set to 8), the system proceeds with scene-aware deduplication and sampling. When the count falls below this threshold, the function immediately triggers the uniform fallback.

Inside the extract_scene_or_uniform Function

Located in skills/watch/scripts/frames.py, this function orchestrates the selection logic through a three-phase process.

Detecting Scene Candidates

The function begins by extracting all potential scene frames without early limiting:

scene_frames = extract_scene_candidates(
    video_path,
    out_dir,
    resolution=resolution,
    max_frames=None,          # do NOT limit early – we need the *full* cut list

    start_seconds=start_seconds,
    end_seconds=end_seconds,
)
scene_count = len(scene_frames)

This uncapped detection ensures the system evaluates the complete visual diversity of the video before making an engine selection.

Threshold Evaluation Against SCENE_MIN_FRAMES

The code evaluates the candidate count against the constant threshold:

if scene_count >= SCENE_MIN_FRAMES:          # SCENE_MIN_FRAMES == 8

    # …dedup and even‑sample → final scene result

When the threshold is satisfied, the pipeline applies perceptual deduplication (dedupe_perceptual) and even sampling (_even_sample) to distill the scene frames down to the user's max_frames limit.

Uniform Sampling Execution

When scene_count < SCENE_MIN_FRAMES, the function calculates a fallback cap and invokes uniform sampling:

fallback_cap = target_frames if max_frames is None else min(max_frames, target_frames)
frames = extract(
    video_path,
    out_dir,
    fps=fps,                 # auto‑calculated based on duration

    resolution=resolution,
    max_frames=fallback_cap,
    start_seconds=start_seconds,
    end_seconds=end_seconds,
)

The extract function generates a uniform grid of timestamps using the fps value derived from auto_fps or auto_fps_focus, ensuring consistent coverage across the video duration regardless of visual staticity.

Metadata and Observability

The function returns a metadata dictionary alongside the extracted frames, enabling callers to track which engine was used:

{
    "engine": "scene"   or "uniform",
    "candidate_count": scene_count,
    "deduped_count": n_dropped,
    "selected_count": len(selected_or_frames),
    "fallback": False   or True,
}

When the fallback activates, the "engine" field reports "uniform" and "fallback" sets to True, providing clear telemetry for debugging and optimization.

Practical Implementation Examples

Direct Function Invocation

To explicitly test the fallback behavior on static content:

from pathlib import Path
from skills.watch.scripts.frames import extract_scene_or_uniform, auto_fps

video = "static_slide.mp4"          # a screen‑recording with almost no cuts

out_dir = Path("/tmp/frames")
fps, _ = auto_fps(120)              # 2 fps for a 2‑minute clip

frames, meta = extract_scene_or_uniform(
    video_path=video,
    out_dir=out_dir,
    fps=fps,
    target_frames=100,               # desired max frames

    resolution=512,
    max_frames=100,
    dedup=True,
)
print(meta["engine"])   # → "uniform"

print(len(frames))      # uniform‑sampled frames

CLI Usage

When running the full watch skill, the system automatically handles the fallback:

python -m skills.watch.scripts.watch \
    --detail balanced \
    --url https://example.com/video.mp4 \
    --out /tmp/watch-output

For static videos, the console output indicates the fallback with a line similar to:

(uniform) 45 frames (fallback)

Unit Test Verification

The repository's test suite validates this behavior in tests/test_frames.py:

def test_scene_fallback_on_static_clip(static_clip: Path, tmp_path: Path):
    fps, _ = auto_fps(30)                     # short clip

    out, meta = extract_scene_or_uniform(
        video_path=str(static_clip),
        out_dir=tmp_path / "frames",
        fps=fps,
        target_frames=20,
        resolution=512,
        max_frames=20,
        dedup=False,
    )
    assert meta["engine"] == "uniform"

Summary

• Threshold-driven: The fallback activates when extract_scene_candidates returns fewer than 8 frames (SCENE_MIN_FRAMES)
• Automatic transition: The extract_scene_or_uniform function in skills/watch/scripts/frames.py handles the switch transparently
• Duration-aware: Uniform sampling uses auto_fps-calculated frame rates to ensure proportional coverage
• Observable: Return metadata includes "engine" and "fallback" fields for monitoring
• Zero-configuration: The system requires no manual intervention to select the appropriate extraction strategy

Frequently Asked Questions

What triggers the scene detection fallback to uniform sampling?

The fallback triggers when the FFmpeg-based scene detector identifies fewer than 8 distinct scene cuts in the video (the SCENE_MIN_FRAMES constant defined in skills/watch/scripts/frames.py). This typically occurs with static content like screen recordings, slideshows, or talking-head videos where minimal visual changes occur between frames.

How does the uniform sampling engine calculate frame intervals?

The uniform sampler uses the auto_fps function to calculate an appropriate frames-per-second value based on video duration and target frame count. The extract function then generates evenly spaced timestamps across the video timeline, ensuring consistent temporal coverage regardless of visual content changes.

Can I force uniform sampling even if scene detection finds enough cuts?

While extract_scene_or_uniform automatically selects the engine based on scene count, you can bypass scene detection entirely by calling the extract function directly from skills/watch/scripts/frames.py. However, the high-level watch.py interface does not expose a manual override flag, as the automatic selection optimizes for both content variety and processing efficiency.

What metadata indicates that a fallback occurred?

The metadata dictionary returned by extract_scene_or_uniform contains two key indicators: the "engine" field will report "uniform" instead of "scene", and the "fallback" boolean field will set to True. Additionally, "candidate_count" will show the number of scene cuts detected (fewer than 8), while "selected_count" reports the final number of uniformly sampled frames extracted.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}