Keyframe Minimum Threshold and Uniform Fallback for Sparse Video Sources in Claude-Video

The keyframe minimum threshold is 4 keyframes, and the uniform fallback is triggered whenever a video segment yields fewer than 4 keyframes.

The claude-video repository implements a robust keyframe extraction engine in its watch skill that automatically detects sparse video sources and switches to uniform sampling when keyframe coverage is insufficient. This fallback mechanism ensures reliable frame extraction across all video types, including very short clips and oddly-encoded streams.

Understanding the KEYFRAME_MIN Constant

The hard lower bound for acceptable keyframe counts is defined in skills/watch/scripts/frames.py at lines 27-30:

# Minimum keyframes needed before we trust the coverage

KEYFRAME_MIN = 4

This constant, KEYFRAME_MIN = 4, represents the absolute minimum number of keyframes the engine requires before it considers the coverage reliable for downstream tasks.

When the Uniform Fallback Activates

The fallback logic executes during the extract_keyframes function. Here's the decision flow:

1. Keyframe candidate extraction — The engine scans the video segment for I-frames (keyframes).
2. Threshold comparison — If len(candidates) < KEYFRAME_MIN, the sparse video condition triggers.
3. Uniform fallback invocation — The engine calls the standard extract function instead.

The conditional check appears at lines 36-38 in skills/watch/scripts/frames.py:

if len(candidates) < KEYFRAME_MIN:
    print(f"[sparse] Too few keyframes ({len(candidates)}), falling back to uniform sampling")
    fallback = True

The actual fallback execution occurs at lines 63-68:

if fallback:
    fps, _ = auto_fps(duration_seconds, max_frames=max_frames)
    return extract(
        video_path, output_dir, fps=fps, resolution=resolution,
        start_seconds=start_seconds, end_seconds=end_seconds,
        max_frames=max_frames, dedup=dedup
    ), {"engine": "uniform", "fallback": True}

Fallback Metadata and Behavior

When the uniform fallback activates, the returned metadata contains two critical flags:

Field	Value	Meaning
engine	"uniform"	Indicates uniform sampling was used instead of keyframe extraction
fallback	True	Confirms the sparse-video condition triggered the alternative path

The fallback path preserves the original max_frames budget, computes appropriate FPS via auto_fps, and optionally applies perceptual deduplication—matching the primary keyframe path's capabilities.

Practical Examples

Detecting Fallback at Runtime

from pathlib import Path
from skills.watch.scripts.frames import extract_keyframes

video_path = "example.mp4"
out_dir = Path("frames_out")

frames, meta = extract_keyframes(
    video_path,
    out_dir,
    resolution=512,
    max_frames=50,
    start_seconds=None,
    end_seconds=None,
    dedup=True,
)

# Inspect which engine actually ran

print("Engine used:", meta["engine"])          # "keyframe" or "uniform"

print("Fallback triggered:", meta["fallback"]) # True when < 4 keyframes

print("Frames extracted:", len(frames))

Manual Uniform Extraction for Known-Sparse Sources

from pathlib import Path
from skills.watch.scripts.frames import extract, auto_fps, get_metadata

video = "tiny_clip.mp4"
out_dir = Path("uniform_out")

# Compute duration and FPS budget

metadata = get_metadata(video)
duration = metadata["duration_seconds"]
fps, _ = auto_fps(duration, max_frames=30)

# Force uniform sampling (bypassing keyframe logic entirely)

frames = extract(
    video,
    out_dir,
    fps=fps,
    resolution=512,
    max_frames=30,
)

print(f"Extracted {len(frames)} uniformly-spaced frames")

Verifying Fallback in Tests

from pathlib import Path
from skills.watch.scripts.frames import extract_keyframes

# Static 1-second clip with minimal keyframes

static_clip = Path("tests/fixtures/static_clip.mp4")
out_dir = Path("tmp")

frames, meta = extract_keyframes(str(static_clip), out_dir, max_frames=50)

# Assertions validate fallback behavior

assert meta["fallback"] is True
assert meta["engine"] == "uniform"

Source File Reference

All threshold and fallback logic resides in skills/watch/scripts/frames.py:

• Lines 27-30: KEYFRAME_MIN constant definition
• Lines 36-38: Sparse-video detection conditional
• Lines 62-68: Uniform fallback execution with metadata

The CLI entry point in skills/watch/scripts/watch.py indirectly relies on this fallback mechanism when routing between the efficient keyframe engine and the scene-aware engine.

Summary

• Keyframe minimum threshold: Fixed at 4 keyframes via KEYFRAME_MIN in skills/watch/scripts/frames.py
• Uniform fallback trigger: Activates when candidate keyframes fall below the threshold
• Fallback behavior: Switches to extract function with uniform FPS-based sampling
• Metadata transparency: Returns {"engine": "uniform", "fallback": True} for downstream awareness
• Budget preservation: Maintains original max_frames and resolution parameters across both paths

Frequently Asked Questions

Why is the keyframe minimum threshold set to 4 specifically?

The value 4 represents a practical trade-off between coverage reliability and sparse-video tolerance. According to the claude-video source code, fewer than 4 keyframes provide insufficient temporal distribution for meaningful scene representation in most video analysis tasks, while still allowing very short clips to process through the fallback path.

How does the uniform fallback handle the max_frames budget differently than keyframe extraction?

Both paths respect the same max_frames budget, but the uniform fallback computes FPS via auto_fps to distribute frames evenly across the time range rather than selecting existing keyframe positions. This guarantees the requested frame count regardless of source encoding.

Can I force uniform sampling even for videos with sufficient keyframes?

Yes. Directly call extract from skills/watch/scripts/frames.py instead of extract_keyframes. Bypassing the keyframe logic entirely avoids the threshold check and any fallback considerations.

What video characteristics typically trigger the uniform fallback?

The fallback activates for very short clips (under typical keyframe intervals), oddly-encoded streams with aggressive GOP structures, static content with minimal I-frame generation, and corrupted or truncated video segments that report fewer detectable keyframes than actually exist.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}