How the Frame Budget System Scales with Video Duration in Claude-Video

The frame budget system uses tiered caps in auto_fps() and auto_fps_focus() to allocate dense frames for short clips while strictly limiting extraction for videos over 10 minutes, ensuring predictable token costs regardless of input length.

The bradautomates/claude-video repository implements an adaptive frame budget mechanism within its watch skill to control how many visual frames are extracted from videos of varying lengths. This system, located primarily in skills/watch/scripts/frames.py, automatically adjusts sampling rates to balance visual detail against AI token consumption. By scaling non-linearly with duration, the pipeline prevents excessive processing costs for long-form content while preserving rich frame density for short segments.

Tiered Budget Calculation for Full Videos

The auto_fps() function in skills/watch/scripts/frames.py (lines 22-38) implements a duration-based tier system that caps the total frame count. The function accepts duration_seconds and an optional max_frames parameter (defaulting to 100), returning a calculated frames-per-second rate and final target count.

Short Clips (≤30 seconds)

For videos under 30 seconds, the system allocates approximately one frame per second with a minimum floor of 12 frames. The logic min(max_frames, max(12, int(round(duration_seconds)))) ensures that even a 5-second clip receives dense coverage, while respecting the global cap.

if duration_seconds <= 30:
    target = min(max_frames, max(12, int(round(duration_seconds))))

This tier provides the richest sampling density, capturing nearly every second of action for brief content.

Medium Duration (30 seconds – 10 minutes)

As duration increases, the frame budget scales sub-linearly through fixed caps:

• 30-60 seconds: Hard cap at 40 frames
• 1-3 minutes: Hard cap at 60 frames
• 3-10 minutes: Hard cap at 80 frames

elif duration_seconds <= 60:
    target = min(max_frames, 40)
elif duration_seconds <= 180:   # 3 min

    target = min(max_frames, 60)
elif duration_seconds <= 600:   # 10 min

    target = min(max_frames, 80)

These steps prevent token costs from growing proportionally with video length, ensuring a 5-minute video uses only 60 frames rather than the 300 frames that a linear 1 FPS rate would require.

Long Videos (>10 minutes)

Content exceeding 10 minutes receives the strictest limit. The system defaults to the user-specified max_frames value (typically 100) as a hard ceiling:

else:
    target = max_frames

The selected target is converted to an FPS value via target / duration_seconds and passed to _clamp_fps(), which enforces a global MAX_FPS = 2.0 ceiling before returning the final tuple.

Focused Range Scaling with auto_fps_focus

When users specify a start/end timestamp, the system switches to auto_fps_focus() in skills/watch/scripts/frames.py (lines 41-58), applying the budget only to the selected slice rather than the full duration.

This function uses accelerated sampling rates for very short windows:

• ≤5 seconds: 6 FPS minimum (minimum 10 frames)
• 5-15 seconds: 4 FPS minimum (minimum 30 frames)
• 15-60 seconds: Caps of 60 and 80 frames respectively
• >60 seconds: Falls back to max_frames

def auto_fps_focus(duration_seconds: float, max_frames: int = 100) -> tuple[float, int]:
    if duration_seconds <= 5:
        target = min(max_frames, max(10, int(round(duration_seconds * 6))))
    elif duration_seconds <= 15:
        target = min(max_frames, max(30, int(round(duration_seconds * 4))))
    # ... additional tiers

This allows dense analysis of specific moments—such as a 10-second action sequence receiving 40 frames—without processing the surrounding footage.

Integration in the Watch Pipeline

The entry point in skills/watch/scripts/watch.py (lines 55-63) selects the appropriate scaling function based on user input:

if focused:
    fps, target = auto_fps_focus(effective_duration, max_frames=budget_cap)
else:
    fps, target = auto_fps(effective_duration, max_frames=budget_cap)

The budget_cap variable pulls from either the --max-frames CLI argument or the configured default for the selected detail level (e.g., 100 for "balanced" mode). The calculated fps value is then passed to extraction functions like extract_scene_or_uniform() or extract_keyframes(), which respect both the target count and the global rate ceiling.

Command-Line Examples

Override the automatic scaling using the watch CLI:

# Default balanced mode with automatic duration scaling

watch https://youtu.be/xyz123

# Focused 15-second window receives denser sampling than full video

watch https://youtu.be/xyz123 --start 00:30 --end 00:45

# Override budget for exhaustive analysis (token-burner mode)

watch https://youtu.be/xyz123 --detail token-burner --max-frames 500

# Force low FPS for very long content

watch https://youtu.be/xyz123 --detail balanced --fps 0.5

Performance and Cost Implications

The non-linear scaling addresses three critical constraints in video processing pipelines.

Token cost control: Each extracted frame consumes image tokens when sent to Claude. By capping long videos at 100 frames regardless of duration, the system keeps API costs predictable for multi-hour content while allowing short clips to retain visual narrative density.

Compute efficiency: Fewer frames reduce ffmpeg processing overhead, memory usage, and storage requirements. A 2-hour video capped at 100 frames processes significantly faster than a naive 1 FPS extraction that would generate 7,200 frames.

User granularity: The --detail presets (efficient, balanced, token-burner) in skills/watch/scripts/config.py provide default caps, while --max-frames and --fps flags allow precise manual overrides when scene density or analysis depth requirements demand deviation from the automatic scaling.

Summary

• The frame budget system scales sub-linearly with video duration, applying strict tiered caps (40, 60, 80, 100 frames) at 30-second, 1-minute, 3-minute, and 10-minute boundaries.
• Short clips (≤30s) receive approximately 1 FPS sampling, while long videos (>10min) are hard-capped at the user-defined maximum (default 100).
• Focused ranges use auto_fps_focus() to apply the budget only to selected time windows, enabling 4-6 FPS sampling for brief segments under 15 seconds.
• The _clamp_fps() function enforces a global 2.0 FPS ceiling regardless of duration or settings.
• Users can override automatic scaling via --max-frames, --detail, or --fps arguments in the watch command.

Frequently Asked Questions

What is the maximum number of frames extracted from a 15-minute video?

A 15-minute video (900 seconds) exceeds the 10-minute threshold in auto_fps(), so it receives the default max_frames value of 100 frames unless the user overrides it with --max-frames or --detail token-burner. The system calculates an effective FPS of approximately 0.11 (100/900) and clamps it to the 2.0 FPS ceiling.

How does the focused range mode differ from full-video scanning?

Focused range mode activates when using --start and --end flags, triggering auto_fps_focus() instead of auto_fps(). This function applies the frame budget only to the selected duration slice, allowing higher densities (up to 6 FPS for clips under 5 seconds) than would be economical for the full video length.

Can users override the automatic frame budget scaling?

Yes. The budget_cap variable in watch.py accepts values from the --max-frames CLI argument, bypassing the automatic tier calculations. Additionally, the --fps flag allows direct specification of extraction rates, and the --detail preset (configured in config.py) adjusts the default cap across all duration tiers.

What is the highest frame rate allowed by the budget system?

The system enforces a global 2.0 FPS ceiling through the _clamp_fps() helper function in frames.py. Even if the tiered logic calculates a higher rate (e.g., 6 FPS for a 5-second focused window), the returned FPS value is clamped to 2.0 to prevent excessive token consumption and processing load.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}