Frame Cap Table Explained: How claude-video Allocates Frames by Duration Thresholds

The frame cap table in claude-video uses two distinct allocation strategies—auto_fps for full-video scans and auto_fps_focus for time-windowed segments—to determine target frame counts based on video duration, with both paths enforcing a hard 2 fps ceiling.

The frame cap table governs how many frames the watch skill extracts from a video before sending them to a vision model. Rather than sampling uniformly, the system in bradautomates/claude-video tieres allocation by duration to balance detail against token cost. This logic lives in skills/watch/scripts/frames.py and is invoked from skills/watch/scripts/watch.py depending on whether the user parses a full video or a focused time window.

The Two Allocation Paths

The codebase maintains separate helpers for full-video versus focused analysis. Each implements its own duration-to-target mapping, then applies a universal 2 fps clamp.

auto_fps: Full-Video Overview Mode

Called when no --start or --end flags are present. The mapping prioritizes recognizable coverage without oversampling long content:

# Source: skills/watch/scripts/frames.py#L122-L138

if dur <= 30:
    target = min(max_frames, max(12, round(dur)))
elif dur <= 60:
    target = min(max_frames, 40)
elif dur <= 180:
    target = min(max_frames, 60)
elif dur <= 600:
    target = min(max_frames, 80)
else:
    target = max_frames

• ≤ 30 seconds: Dense sampling—at least 12 frames, or roughly 1 frame per second.
• 30–60 seconds: Capped at 40 frames.
• 60 seconds–3 minutes: Capped at 60 frames.
• 3–10 minutes: Capped at 80 frames.
• > 10 minutes: Uses the user-supplied max_frames (default 100).

auto_fps_focus: Time-Windowed Detail Mode

Called when the user supplies --start/--end bounds. The frame cap table scales more aggressively because the user explicitly requested granular inspection:

# Source: skills/watch/scripts/frames.py#L141-L158

if dur <= 5:
    target = min(max_frames, max(10, round(dur * 6)))
elif dur <= 15:
    target = min(max_frames, max(30, round(dur * 4)))
elif dur <= 30:
    target = min(max_frames, 60)
elif dur <= 60:
    target = min(max_frames, 80)
elif dur <= 180:
    target = max_frames
else:
    target = max_frames

• ≤ 5 seconds: Up to 6× multiplier (max 10 floor), yielding 10–30 frames.
• 5–15 seconds: 4× multiplier, floor at 30 frames.
• 15–30 seconds: Hard cap at 60 frames.
• 30 seconds–3 minutes: Hard cap at 80 frames then max_frames.
• > 3 minutes: Falls back to max_frames.

The 2 FPS Hard Ceiling

Both helpers feed into _clamp_fps, which enforces MAX_FPS = 2.0 regardless of duration:

# Source: skills/watch/scripts/frames.py#L49-L53

MAX_FPS = 2.0

def _clamp_fps(fps, duration_seconds, max_frames):
    fps = min(fps, MAX_FPS)
    # Additional logic ensures target frames never exceed max_frames

    return fps, target

This protects against pathological cases where the raw math would suggest 10+ fps on very short clips.

How Frame Caps Drive Extraction

In skills/watch/scripts/watch.py, the driver selects the appropriate helper based on the focused flag:

# Source: skills/watch/scripts/watch.py#L330-L334

if focused:
    fps, target = auto_fps_focus(effective_duration, max_frames=max_frames)
else:
    fps, target = auto_fps(effective_duration, max_frames=max_frames)

The returned target becomes the frame cap passed to downstream extractors (extract, extract_keyframes, extract_scene_or_uniform). These functions respect the limit via ffmpeg -frames:v or internal even-sampling logic.

Practical Examples

from skills.watch.scripts.frames import auto_fps, auto_fps_focus

# Full-video scan: 2-minute clip, default max_frames=100

fps, cap = auto_fps(duration_seconds=120)
print(f"fps={fps:.2f}, frame-cap={cap}")  # fps≈2.00, cap≈60

# Focused window: 8-second segment, max_frames=80

fps, cap = auto_fps_focus(duration_seconds=8, max_frames=80)
print(f"fps={fps:.2f}, frame-cap={cap}")  # fps≈2.00, cap≈48 (6×8, rounded)

The computed fps and cap values flow directly into ffmpeg -vf fps={fps} and frame-limiting arguments.

Key Configuration Files

File	Role in Frame Allocation
skills/watch/scripts/frames.py	Hosts auto_fps, auto_fps_focus, _clamp_fps, and extraction routines
skills/watch/scripts/watch.py	Orchestrates mode detection and helper invocation
skills/watch/scripts/config.py	Defines default max_frames and tunable limits
tests/test_frames.py	Validates cap table outputs against expected targets

Summary

• The frame cap table uses duration thresholds to prevent over-sampling short clips and under-representing long ones.
• auto_fps allocates 12–100 frames for full-video scans; auto_fps_focus allocates 10–100 frames with higher density for short windows.
• All paths enforce 2 fps maximum via _clamp_fps in skills/watch/scripts/frames.py.
• The target frame count drives ffmpeg invocation and downstream sampling logic.

Frequently Asked Questions

What happens if a video exceeds the 10-minute threshold in auto_fps?

The system defaults to the user-supplied max_frames value, typically 100. The _clamp_fps function ensures the fps never exceeds 2.0, so a 20-minute video yields 100 frames at 0.083 fps (one frame every 12 seconds).

Why does focused mode use a 6× multiplier for clips under 5 seconds?

Short segments often contain rapid action or precise gestures. The 6× multiplier (capped by max_frames) prioritizes temporal resolution when the user explicitly isolates a narrow window, capturing up to 30 frames for a 5-second clip rather than the 5 frames a 1:1 ratio would provide.

Can I override the default max_frames value?

Yes. The default lives in skills/watch/scripts/config.py and can be passed as a parameter to auto_fps or auto_fps_focus. The CLI also exposes this as a configurable flag that propagates through watch.py to the allocation helpers.

Where does the actual frame extraction happen?

The target and fps values computed by the frame cap table flow into functions like extract(), extract_keyframes(), and extract_scene_or_uniform() within skills/watch/scripts/frames.py. These functions invoke ffmpeg with -vf fps={fps} and -frames:v {target} to enforce the limits at the codec level.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}