Uniform Fallback for Frame Extraction in Claude Video: Implementation Guide

Claude Video automatically falls back to uniform frame sampling when scene detection identifies fewer than 8 distinct shots, ensuring static content like screen recordings still generate usable frame sequences for analysis.

The bradautomates/claude-video repository implements an intelligent two-stage extraction engine that balances quality with reliability. When processing videos with minimal visual changes, the uniform fallback for frame extraction guarantees consistent output by switching from scene-based to time-based sampling. This mechanism is critical for handling screen recordings, talking-head videos, and other low-motion content that would otherwise yield insufficient frames for analysis.

How the Uniform Fallback Works

The extraction logic in skills/watch/scripts/frames.py operates through a conditional pipeline that prioritizes scene detection before resorting to uniform sampling.

Scene Detection Stage

First, the system runs ffmpeg's scene-change detector via extract_scene_candidates to identify visual cuts exceeding SCENE_THRESHOLD. This captures the first frame and every subsequent frame where significant visual differences occur, creating a candidate list of scene boundaries.

The Fallback Threshold

The system compares the detected scene count against SCENE_MIN_FRAMES (default 8). If the video contains sufficient visual changes, the engine retains the scene-based frames, optionally applying perceptual deduplication and even-sampling to meet the user's max_frames limit.

Uniform Sampling Activation

When fewer than 8 scenes are detected, the code calculates a fallback_cap budget and invokes the extract function. This uses ffmpeg's fps= filter to grab frames at constant intervals, ensuring coverage across the entire video duration regardless of visual staticity. After extraction, dedupe_perceptual removes near-duplicate frames if the dedup parameter is enabled.

Implementation in frames.py

The core logic resides in the extract_scene_or_uniform function within skills/watch/scripts/frames.py:

def extract_scene_or_uniform(
    video_path: str,
    out_dir: Path,
    fps: float,
    target_frames: int,
    resolution: int = 512,
    max_frames: int | None = 100,
    start_seconds: float | None = None,
    end_seconds: float | None = None,
    dedup: bool = True,
) -> tuple[list[dict], dict]:
    """Prefer scene selection, falling back to uniform only when the video is
    effectively static (fewer than ``SCENE_MIN_FRAMES`` detected shots)."""
    # 1️⃣ Run full-range scene detection (uncapped)

    scene_frames = extract_scene_candidates(
        video_path,
        out_dir,
        resolution=resolution,
        max_frames=None,
        start_seconds=start_seconds,
        end_seconds=end_seconds,
    )
    scene_count = len(scene_frames)

    # 2️⃣ If enough cuts → keep them (optionally dedupe & even-sample)

    if scene_count >= SCENE_MIN_FRAMES:
        # ... dedup, cap, and return selected frames

        pass

    # 3️⃣ Otherwise fall back to uniform extraction

    fallback_cap = target_frames if max_frames is None else min(max_frames, target_frames)
    frames = extract(
        video_path,
        out_dir,
        fps=fps,
        resolution=resolution,
        max_frames=fallback_cap,
        start_seconds=start_seconds,
        end_seconds=end_seconds,
    )
    if dedup:
        frames, n_dropped = dedupe_perceptual(frames)
    return frames, {
        "engine": "uniform",
        "candidate_count": scene_count,
        "deduped_count": n_dropped,
        "selected_count": len(frames),
        "fallback": True,
    }

Using the Fallback via CLI and API

CLI Usage with watch.py

The top-level entry point skills/watch/scripts/watch.py automatically invokes the fallback when using --detail balanced or --detail token-burner modes:

watch "https://example.com/static-screen-recording.mp4" \
      --detail balanced \
      --resolution 640 \
      --max-frames 30

The generated markdown report indicates fallback activation with metadata showing "engine": "uniform" and "fallback": true, along with the final frame count and deduplication statistics.

Direct API Integration

To programmatically leverage the fallback mechanism:

from pathlib import Path
from skills.watch.scripts.frames import (
    get_metadata,
    auto_fps,
    extract_scene_or_uniform,
)

video = "lecture.mp4"
out_dir = Path("frames")
metadata = get_metadata(video)

# Compute FPS for 50 frames across 10 seconds

fps, _ = auto_fps(duration_seconds=10, max_frames=50)

frames, meta = extract_scene_or_uniform(
    video_path=video,
    out_dir=out_dir,
    fps=fps,
    target_frames=50,
    resolution=512,
    max_frames=50,
)

print("Engine:", meta["engine"])          # "scene" or "uniform"

print("Fallback:", meta["fallback"])      # True only if uniform path used

Configuration Parameters

The fallback behavior is controlled by constants defined in the codebase:

• SCENE_MIN_FRAMES: Default threshold of 8 scenes required to avoid fallback
• SCENE_THRESHOLD: Visual difference sensitivity for scene detection
• dedup: Boolean flag (default True) enabling perceptual deduplication via dedupe_perceptual after uniform extraction

Configuration defaults and detail-level mappings reside in skills/watch/scripts/config.py, which watch.py references when determining extraction parameters.

Summary

• Claude Video implements a uniform fallback for frame extraction that activates when scene detection finds fewer than SCENE_MIN_FRAMES (8) distinct shots.
• The logic is centralized in extract_scene_or_uniform within skills/watch/scripts/frames.py.
• Fallback extraction uses ffmpeg's constant FPS filtering via the extract function to ensure comprehensive temporal coverage.
• The system automatically applies perceptual deduplication to remove near-duplicate frames from uniform samples.
• CLI users trigger this behavior through watch.py with --detail balanced, while API users can call extract_scene_or_uniform directly.

Frequently Asked Questions

What triggers the uniform fallback in Claude Video?

The fallback triggers when extract_scene_candidates returns fewer than SCENE_MIN_FRAMES (default 8) distinct scenes. This typically occurs with static content like screen recordings or talking-head videos where visual changes fall below the SCENE_THRESHOLD sensitivity, causing the scene detector to identify insufficient cut points.

How does the fallback handle frame deduplication?

After uniform extraction, the system runs dedupe_perceptual to remove near-duplicate frames when the dedup parameter is True (the default). The metadata dictionary returned by extract_scene_or_uniform includes deduped_count, indicating exactly how many frames were removed from the initial uniform sample set.

Can I force uniform extraction without scene detection?

Yes. Bypass extract_scene_or_uniform and call extract directly from skills/watch/scripts/frames.py. This function uses ffmpeg's fps= filter for constant-rate sampling without scene analysis, though you will lose the automatic fallback intelligence and the unified metadata reporting that indicates which engine was used.

Where is the fallback logic implemented in the codebase?

The primary implementation lives in skills/watch/scripts/frames.py inside the extract_scene_or_uniform function. The CLI integration resides in skills/watch/scripts/watch.py, which determines which extraction engine to invoke based on the --detail flag and orchestrates the final report generation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}