How the Uniform Sampler Fallback Mechanism Works in claude-video

The uniform sampler fallback mechanism automatically activates when the scene detection or keyframe extraction engines fail to generate enough distinct frames, switching to evenly-spaced temporal sampling to guarantee a usable set of images.

In the bradautomates/claude-video repository, the uniform sampler serves as the critical safety net that ensures consistent frame extraction across diverse video types. When higher-level analysis engines cannot identify sufficient visual changes, this fallback mechanism seamlessly transitions to uniform temporal sampling to meet the requested frame budget.

When the Fallback Mechanism Triggers

The uniform sampler activates in two specific scenarios within skills/watch/scripts/frames.py:

Scene Detection Engine

When the scene-cut detector returns fewer than 8 distinct shots (SCENE_MIN_FRAMES), the video is treated as either static or insufficiently varied for scene-based analysis. The extract_scene_or_uniform() function (lines 54‑55) detects this condition and falls back to extract(), sampling frames at a constant rate calculated by auto_fps() for the effective duration of the clip.

Keyframe Extraction Engine

If fewer than 4 keyframes (KEYFRAME_MIN) are discovered—typically occurring in very short clips or strangely encoded files—the extract_keyframes() function (lines 636‑638) discards the sparse candidates and rebuilds the sample set using the same uniform extraction method.

The Fallback Execution Process

The uniform sampler follows a consistent four-step procedure to ensure reliable output:

1. Detect insufficient candidates

   if scene_count < SCENE_MIN_FRAMES:
       # Trigger fallback in extract_scene_or_uniform()
   
   if len(candidates) < KEYFRAME_MIN:
       # Trigger fallback in extract_keyframes()
   

2. Calculate optimal frame rate

   fps, _ = auto_fps(eff_duration, max_frames=budget)

   The auto_fps() function (lines 22‑36) computes a frame rate that respects the max_frames budget while distributing samples evenly across the video's effective duration.

3. Execute uniform extraction

   frames = extract(
       video_path, out_dir, fps=fps,
       resolution=resolution, max_frames=budget,
       start_seconds=start_seconds, end_seconds=end_seconds,
   )

   The extract() function (lines 71‑84) invokes ffmpeg with the fps={fps} filter to generate evenly-spaced JPEGs.

4. Apply optional deduplication

   if dedup:
       frames, n_dropped = dedupe_perceptual(frames)

   The dedupe_perceptual() function (lines 64‑71) removes visually identical frames that may result from static video segments.

Metadata and Engine Identification

Regardless of which engine initially attempted extraction, the fallback mechanism consistently marks the operation in the returned metadata dictionary (lines 68‑73 and 662‑668):

meta = {
    "engine": "uniform",
    "fallback": True,
    "frame_count": len(frames)
}

This allows downstream components in the /watch skill to report that uniform sampling was used instead of the primary extraction method.

Code Examples

Scene Engine with Fallback

from skills.watch.scripts.frames import extract_scene_or_uniform
from pathlib import Path

frames, meta = extract_scene_or_uniform(
    video_path="example.mp4",
    out_dir=Path("./out"),
    target_frames=30,
    max_frames=30,
)

print(meta["engine"])    # "scene" or "uniform"

print(meta["fallback"])  # False for scene, True for uniform fallback

Keyframe Extraction with Automatic Fallback

from skills.watch.scripts.frames import extract_keyframes
from pathlib import Path

frames, meta = extract_keyframes(
    video_path="short_clip.mp4",
    out_dir=Path("./out_kf"),
    max_frames=10,
)

print(meta["engine"])    # "keyframe" or "uniform"

print(meta["fallback"])  # True when keyframe count < 4

Summary

• The uniform sampler fallback mechanism activates when scene detection yields fewer than 8 frames or keyframe extraction finds fewer than 4 candidates.
• The system calculates an optimal frame rate using auto_fps() that respects the max_frames budget while ensuring even temporal distribution.
• The extract() function in skills/watch/scripts/frames.py uses ffmpeg with a constant FPS filter to generate the fallback frame set.
• Optional perceptual deduplication removes visually identical frames after extraction.
• Metadata flags (engine="uniform", fallback=True) ensure downstream processes can identify when the fallback was invoked.

Frequently Asked Questions

What triggers the uniform sampler fallback in claude-video?

The fallback triggers when the primary extraction engines cannot produce enough distinct frames. Specifically, when the scene-cut detector finds fewer than 8 shots (SCENE_MIN_FRAMES) or when keyframe extraction yields fewer than 4 candidates (KEYFRAME_MIN). In both cases, the system automatically switches to uniform temporal sampling to guarantee a usable frame set.

How does the uniform sampler calculate how many frames to extract?

The auto_fps() function (lines 22‑36 in frames.py) calculates a frame rate that distributes the requested max_frames budget evenly across the video's effective duration. This ensures the output contains the optimal number of evenly-spaced frames without exceeding the user's specified limits.

Can I prevent the uniform sampler from running if the video is static?

No, the fallback is automatic and designed as a safety mechanism. However, you can apply the optional dedupe_perceptual() filter after extraction, which removes visually identical frames that often result from static video content. This function is available at lines 64‑71 in skills/watch/scripts/frames.py.

Where is the uniform sampler fallback implemented in the codebase?

The core implementation resides in skills/watch/scripts/frames.py in the bradautomates/claude-video repository. The extract_scene_or_uniform() function handles the scene detection fallback, while extract_keyframes() manages the keyframe extraction fallback. Both functions call the extract() function (lines 71‑84) to perform the actual uniform sampling using ffmpeg.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}