How Auto-FPS Adapts to Video Duration and Frame Budget in claude-video

The auto-fps calculation dynamically adjusts frame extraction rates using duration-based tiering, applying higher density to short clips and strict caps to long videos while never exceeding a 2 FPS maximum or the user-defined frame budget.

The claude-video repository implements an intelligent frame sampling system that balances visual detail against token limits. Its auto-fps calculation automatically scales the extraction rate according to video length and a configurable frame budget, ensuring short segments remain granular while preventing long content from exploding processing costs. This adaptive logic resides in skills/watch/scripts/frames.py and drives the main analysis pipeline.

Core Algorithm in frames.py

The adaptive logic centers on two primary functions that handle different scanning modes, both enforcing safety limits through a shared clamping helper.

Auto-FPS for Full-Video Scans

When processing an entire video without time restrictions, the auto_fps function (lines 122-138) applies duration-based tiering to select the target frame count:

• ≤ 30 seconds: target = min(max_frames, max(12, round(duration))) — approximately one frame per second for very short content
• ≤ 60 seconds: target = min(max_frames, 40)
• ≤ 180 seconds (3 minutes): target = min(max_frames, 60)
• ≤ 600 seconds (10 minutes): target = min(max_frames, 80)
• > 600 seconds: target = max_frames (the user-supplied hard cap)

Auto-FPS for Focused Ranges

For detailed segment analysis when --start and --end flags are provided, auto_fps_focus (lines 141-158) allocates a denser frame budget because the user is "zooming in" for detail:

• ≤ 5 seconds: Up to 6× duration frames (capped by max_frames)
• ≤ 15 seconds: Up to 4× duration frames
• ≤ 30 seconds: target = min(max_frames, 60)
• ≤ 60 seconds: target = min(max_frames, 80)
• ≤ 180 seconds or longer: target = max_frames

FPS Clamping and Safety Mechanisms

Both functions route through _clamp_fps (lines 49-53), which enforces three critical constraints before returning the final values:

1. Maximum rate: FPS capped at MAX_FPS = 2.0
2. Minimum output: Guarantees at least one frame via max(1, ...)
3. Budget compliance: Respects the max_frames parameter from --max-frames or detail-level configuration

The clamping function calculates the effective FPS as target / duration, then applies these guards to prevent impossible extraction rates. Both auto_fps and auto_fps_focus also guard against non-positive durations (duration_seconds <= 0), immediately returning (1 fps, 1 frame) to avoid division-by-zero errors.

Integration with the Watch Pipeline

The main entry point skills/watch/scripts/watch.py selects the appropriate calculation path based on user input:

if focused:
    fps, target = auto_fps_focus(effective_duration, max_frames=budget_cap)
else:
    fps, target = auto_fps(effective_duration, max_frames=budget_cap)

The returned fps value drives the extraction engine (extract, extract_keyframes, or extract_scene_or_uniform), while target reports the budget allocation in the final markdown output. Default budgets originate from skills/watch/scripts/config.py, which maps detail levels (low, balanced, high) to specific frame caps.

Practical Usage Examples

# Direct API usage for testing adaptive logic

from skills.watch.scripts.frames import auto_fps, auto_fps_focus

# 45-second video with default 100-frame budget

fps, frame_count = auto_fps(45.0)

# Returns approximately 0.89 fps, yielding 40 frames

# 8-second focused segment with 30-frame budget

fps_focus, frame_count_focus = auto_fps_focus(8.0, max_frames=30)

# Returns 2.0 fps capped, yielding 16 frames

# CLI invocation triggering focused mode

watch https://youtu.be/example \
      --detail balanced \
      --max-frames 120 \
      --start 00:01:30 --end 00:02:00

# Calls auto_fps_focus(30s, max_frames=120) → ~2 fps, 60 frames

# Custom script inspecting extraction parameters

from pathlib import Path
from skills.watch.scripts.frames import get_metadata, auto_fps, extract

video_path = "lecture.mp4"
meta = get_metadata(video_path)
duration = meta["duration_seconds"]

fps, target = auto_fps(duration, max_frames=80)
frames = extract(video_path, Path("./frames"), fps=fps, max_frames=target)
print(f"Extracted {len(frames)} frames at {fps:.2f} fps")

Summary

• The auto-fps calculation uses tiered duration thresholds to determine optimal frame counts, keeping short videos dense and long videos manageable according to the frame budget.
• Two calculation modes exist: auto_fps for full-video scans and auto_fps_focus for time-restricted segments, with the latter applying multipliers (6×, 4×) for enhanced detail.
• The _clamp_fps helper enforces a global 2 FPS maximum and guarantees at least one frame, ensuring valid ffmpeg parameters.
• All calculations respect the frame budget (max_frames), which defaults to 100 but is configurable via CLI flags or detail-level presets in config.py.
• Source files: skills/watch/scripts/frames.py (logic), skills/watch/scripts/watch.py (integration), skills/watch/scripts/config.py (budget defaults).

Frequently Asked Questions

What is the maximum frame rate the auto-fps calculation will ever request?

The calculation hard-caps the extraction rate at 2.0 FPS via the MAX_FPS constant defined in skills/watch/scripts/frames.py. Even if the duration-based math suggests a higher rate, the _clamp_fps function (lines 49-53) limits the output to prevent excessive frame generation and token consumption.

How does the frame budget differ between full-video and focused analysis modes?

Focused ranges (--start/--end specified) receive a denser allocation through auto_fps_focus, which multiplies short durations by 6× (for ≤5s) or 4× (for ≤15s). In contrast, auto_fps for full videos uses fixed caps (12, 40, 60, 80 frames) based on duration tiers. Both modes ultimately respect the user-supplied max_frames ceiling.

Why does the algorithm guarantee at least one frame even for very short or invalid durations?

The _clamp_fps function includes a safety guard using max(1, ...) to ensure the frame count never falls below one. Additionally, both auto_fps and auto_fps_focus check for non-positive durations (duration_seconds <= 0) and immediately return (1 fps, 1 frame) to prevent ffmpeg errors or division-by-zero issues during extraction.

Where does the default frame budget of 100 frames originate?

The default value is defined in skills/watch/scripts/config.py, which maps detail levels (low, balanced, high) to specific frame caps. The balanced preset typically sets the budget to 100 frames, though users can override this via the --max-frames CLI argument or by modifying the configuration mapping.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}