Claude Video Auto-FPS Algorithm and Frame Budgets Explained

Claude Video uses a dual-mode auto-fps system that caps extraction at 2 FPS while dynamically scaling frame budgets based on clip duration, ensuring short videos remain dense and long videos stay within token limits.

Claude Video's intelligent frame sampling system automatically determines optimal extraction rates to balance visual detail with LLM token constraints. The algorithm resides in skills/watch/scripts/frames.py and implements a tiered budget approach that adjusts frame density based on whether you're analyzing full videos or focused snippets.

Core Constants and Constraints

The auto-fps logic relies on strict constants defined at the top of skills/watch/scripts/frames.py【/cache/repos/github.com/bradautomates/claude-video/main/skills/watch/scripts/frames.py#L19-L26】:

• MAX_FPS = 2.0 – Hard ceiling preventing extraction above 2 frames per second to control token costs
• SCENE_MIN_FRAMES = 8 – Minimum threshold for scene-based fallback strategies
• KEYFRAME_MIN = 4 – Minimum keyframe requirement for alternative extraction modes

These constraints ensure that even when the algorithm calculates higher theoretical rates, the actual extraction never exceeds 2 FPS.

Uniform Auto-FPS for Full-Video Analysis

The auto_fps function handles frame budget calculation when scanning entire videos without user-specified ranges:

def auto_fps(duration_seconds: float, max_frames: int = 100) -> tuple[float, int]:
    # … (logic) …

The algorithm applies duration-based tiering:

• Very short clips (≤ 30 seconds) – Targets roughly one frame per second, but never fewer than 12 frames: target = min(max_frames, max(12, int(round(duration_seconds))))
• Medium clips (≤ 60 seconds) – Caps extraction at 40 frames total
• Long clips (≤ 180 seconds) – Limits output to 60 frames
• Up to 10 minutes – Restricts budget to 80 frames
• Longer content – Receives the full max_frames allocation (default 100)

The raw FPS calculation then passes through _clamp_fps【/cache/repos/github.com/bradautomates/claude-video/main/skills/watch/scripts/frames.py#L49-L53】, which enforces the MAX_FPS ceiling and rounds to integer frame counts.

Focused Auto-FPS for Detail Extraction

When users specify start/end timestamps via --start and --end arguments, the system switches to auto_fps_focus for higher density analysis of specific segments:

def auto_fps_focus(duration_seconds: float, max_frames: int = 100) -> tuple[float, int]:
    # … (logic) …

This mode applies aggressive multipliers for brief windows:

• Tiny ranges (≤ 5 seconds) – Multiplies duration by 6, guaranteeing at least 10 frames
• Brief segments (≤ 15 seconds) – Multiplies by 4 with a minimum floor of 30 frames
• Short clips (≤ 30 seconds) – Caps at 60 frames
• Medium windows (≤ 60 seconds) – Limits to 80 frames
• Longer selections – Falls back to the full max_frames budget

This focused approach ensures that zoomed-in analysis retains sufficient visual granularity for detailed LLM reasoning.

Budget Enforcement Mechanisms

The _clamp_fps helper function (lines 49-53) implements dual constraints:

1. FPS never exceeds the MAX_FPS = 2.0 constant
2. Target frame counts are limited to both the calculated FPS × duration and the user-supplied max_frames parameter

This enforcement ensures that even when duration-based calculations suggest higher frame counts, the final extraction respects both the hard rate limit and the total token budget.

Implementation Flow and CLI Integration

The execution flow in skills/watch/scripts/frames.py【/cache/repos/github.com/bradautomates/claude-video/main/skills/watch/scripts/frames.py#L30-L38】 follows this pattern:

1. CLI driver parses optional --start/--end arguments to compute effective duration
2. Selects auto_fps for full-video scans or auto_fps_focus for ranged extractions
3. Invokes extract() with the computed FPS and target frame count
4. Applies optional de-duplication to preserve visual diversity while respecting the calculated budget

Practical Code Examples

Full-video extraction for a 2-minute clip:

fps, target = auto_fps(duration_seconds=120)   # → fps≈0.5, target≈60

Focused extraction for a 4-second excerpt:

fps, target = auto_fps_focus(duration_seconds=4)  # → fps≈1.5, target=10 (enforced minimum)

Both functions return a tuple containing the exact FPS value and the precise number of frames that will be extracted for downstream LLM processing.

Summary

• Claude Video implements two distinct auto-fps modes: uniform sampling for full videos and focused sampling for user-defined ranges
• All extraction rates are hard-capped at 2 FPS via MAX_FPS to prevent token overflow
• Frame budgets scale from 12-100 frames based on duration, with higher density allocated to short clips and focused segments
• The _clamp_fps helper enforces both rate and budget constraints before extraction
• Source logic resides in skills/watch/scripts/frames.py with specific duration thresholds defined in the auto_fps and auto_fps_focus functions

Frequently Asked Questions

What is the maximum FPS Claude Video will extract?

Claude Video enforces a hard limit of 2 FPS through the MAX_FPS constant defined in skills/watch/scripts/frames.py. Even if duration calculations suggest higher rates, the _clamp_fps function automatically reduces the extraction speed to stay within this ceiling, ensuring token costs remain predictable for LLM processing.

How does focused mode differ from uniform auto-fps?

Uniform mode (auto_fps) analyzes entire videos with conservative frame budgets (12-100 frames) that prioritize coverage over density. Focused mode (auto_fps_focus) activates when users specify time ranges via CLI arguments, applying multipliers up to 6× for very short segments to ensure detailed visual analysis of specific moments while maintaining the same 2 FPS ceiling.

What happens if I don't specify a frame budget?

The system defaults to 100 frames via the max_frames parameter in both auto_fps and auto_fps_focus functions. However, duration-based rules may reduce this further—for example, a 20-second video automatically receives only 20 frames under uniform mode, while a 4-hour video would receive the full 100-frame allocation.

Where does the actual frame extraction occur?

While the auto-fps logic lives in skills/watch/scripts/frames.py, the orchestration happens in skills/watch/scripts/watch.py, which handles video download, metadata retrieval, and invokes the appropriate auto-fps function based on whether start/end timestamps are provided.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}