When Does the System Fall Back from Keyframe Extraction to Uniform Extraction?

The system falls back from keyframe extraction to uniform extraction whenever fewer than four keyframes are detected in the requested video segment.

The bradautomates/claude-video repository implements an intelligent frame extraction system that attempts to use keyframes (I-frames) for efficiency. However, when source material lacks sufficient keyframes, the system must fall back from keyframe extraction to uniform extraction to ensure adequate frame coverage. This fallback mechanism is triggered by a specific threshold that developers should understand when processing diverse video content.

Conditions That Trigger Fallback from Keyframe Extraction to Uniform Extraction

The fallback mechanism is governed by a hard-coded constant defined in skills/watch/scripts/frames.py. At lines 27-30, the code establishes the minimum threshold:

KEYFRAME_MIN = 4

This value represents the absolute minimum number of keyframes required to justify using the keyframe extraction engine. If a video segment contains fewer than four keyframes—common in static screen recordings, very short clips, or videos with unusual encoding settings—the system triggers the fallback sequence.

Detection Logic in extract_keyframes

The actual check occurs inside the extract_keyframes function at lines 36-38 of skills/watch/scripts/frames.py. After attempting to decode I-frames using FFmpeg with the -skip_frame nokey flag, the function evaluates the candidate count:

if len(candidates) < KEYFRAME_MIN:   # ← fallback condition

    # … uniform extraction logic follows

When this condition evaluates to True, the engine immediately pivots from keyframe-based sampling to uniform frame extraction over the same time range.

The Fallback Execution Sequence

Upon triggering the fallback condition, the system executes a four-step recovery process:

1. Cleanup: Deletes any partially-extracted keyframe JPEGs to prevent file pollution.
2. Frame-rate calculation: Calls auto_fps to compute a suitable sampling rate for the effective segment duration.
3. Uniform extraction: Invokes the generic extract function to perform uniform sampling across the time range.
4. Deduplication: Optionally runs perceptual deduplication to remove near-duplicate frames.

This sequence ensures that even videos with sparse keyframes still yield a representative frame set for downstream processing.

Metadata and Verification

The fallback status is transparently reported through the function's metadata return value. When uniform extraction is triggered as a fallback, the metadata dictionary reflects this change:

{
    "engine": "uniform",
    "fallback": True,
    # other fields …

}

The repository documentation in README.md explicitly confirms this behavior, stating that "if a clip has fewer than 4 keyframes it falls back to uniform sampling" (lines 25-27).

Unit tests in tests/test_frames.py validate this logic through the test_keyframe_fallback_on_static_clip test (lines 30-36), which asserts that static clips correctly trigger the uniform engine with fallback=True.

Practical Examples

The following examples demonstrate automatic engine selection based on video content:

from pathlib import Path
import frames

# Example 1: Normal video with regular scene cuts

out, meta = frames.extract_keyframes(
    video_path="cut_clip.mp4",
    out_dir=Path("frames/"),
    max_frames=50,
)
print(meta["engine"])   # → "keyframe"

print(meta["fallback"]) # → False

# Example 2: Static screen recording with minimal keyframes

out, meta = frames.extract_keyframes(
    video_path="static_clip.mp4",
    out_dir=Path("frames/"),
    max_frames=50,
)
print(meta["engine"])   # → "uniform"

print(meta["fallback"]) # → True

Summary

• The fallback threshold is 4 keyframes: The constant KEYFRAME_MIN = 4 in skills/watch/scripts/frames.py defines the minimum viable keyframe count.
• Detection occurs post-decoding: After FFmpeg attempts to extract I-frames using -skip_frame nokey, the system checks if len(candidates) < 4.
• Automatic recovery: The system cleans up partial results, calculates appropriate frame rates, and switches to uniform sampling without user intervention.
• Transparent reporting: The metadata return value indicates "engine": "uniform" and "fallback": True when the switch occurs.

Frequently Asked Questions

What is the minimum number of keyframes required to avoid fallback?

The system requires at least 4 keyframes to maintain keyframe extraction mode. This threshold is hard-coded as KEYFRAME_MIN = 4 in skills/watch/scripts/frames.py and applies to all video segments processed by the extract_keyframes function.

How does the system handle the transition to uniform extraction?

When fallback triggers, the system first deletes any partially extracted JPEG files, then calls auto_fps to determine an appropriate sampling rate for the segment duration. It subsequently invokes the generic extract function to perform uniform temporal sampling across the same time range, optionally followed by perceptual deduplication.

Where is the fallback behavior documented?

The fallback logic is documented in the repository's README.md at lines 25-27, which explicitly states: "if a clip has fewer than 4 keyframes it falls back to uniform sampling." Additionally, the implementation details are evident in the source code at skills/watch/scripts/frames.py lines 27-38.

How can I verify that fallback occurred in my processed video?

Check the metadata dictionary returned by extract_keyframes. If fallback occurred, the metadata will contain "engine": "uniform" and "fallback": True. You can also inspect the unit test test_keyframe_fallback_on_static_clip in tests/test_frames.py (lines 30-36) to see how the repository validates this behavior programmatically.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}