Frame Budget System in Claude-Video: How It Adapts to Video Length

The frame budget system dynamically targets a maximum number of frames based on video duration rather than using a fixed sampling rate, ensuring short videos get dense coverage while long videos stay within predictable token-cost limits.

The frame budget system is implemented in the bradautomates/claude-video repository to optimize video processing for AI analysis. Unlike traditional extraction methods that use a constant frames-per-second (FPS) rate, this system calculates an adaptive budget—a hard cap on total extractable frames—that scales according to the video's total duration or a user-specified time range.

How the Frame Budget System Works

Core Budget Philosophy

As documented in skills/watch/scripts/frames.py, the system operates on a fundamental principle: token costs scale linearly with frame count. The inline documentation explains that the logic "targets a frame budget, not a fixed rate" to keep short videos dense and long videos capped. When users specify a focused time range, the system switches to a denser allocation mode because they are "zooming in for detail."

Full-Video Budget Calculation (auto_fps)

For complete video analysis, the auto_fps() function computes a suitable FPS that respects the duration-based tiering while staying within the max_frames limit:

• ≤ 30 seconds: Up to 12 FPS
• ≤ 60 seconds: Up to 40 FPS
• > 10 minutes: Falls back to user-provided max_frames
• Hard ceiling: FPS is clamped to MAX_FPS (2 FPS) regardless of duration

The function calculates a target frame count and ensures the final extraction never exceeds the budget. For example, a 2-minute 5-second video (125 seconds) receives approximately 0.63 FPS, yielding roughly 80 frames—well within a typical 100-frame budget.

Focused-Range Budget Calculation (auto_fps_focus)

When users specify a start/end range via auto_fps_focus(), the system allocates a denser budget because the analysis is concentrated on a shorter window:

• ≤ 5 seconds: Up to 6 × duration frames
• ≤ 15 seconds: Up to 4 × duration frames
• Longer ranges: Progressive scaling down to base rates

This allocation still respects the global max_frames parameter to prevent excessive token consumption, but provides higher visual fidelity for detailed inspection of specific segments.

Implementing Frame Budget Extraction

To apply the frame budget system to a full video, use the auto_fps function to calculate parameters before extraction:

from pathlib import Path
from skills.watch.scripts.frames import auto_fps, extract

video = "example.mp4"
metadata = get_metadata(video)               # → {"duration_seconds": 125, …}

fps, target = auto_fps(metadata["duration_seconds"], max_frames=100)

frames = extract(
    video_path=video,
    out_dir=Path("./frames"),
    fps=fps,
    max_frames=target,
    resolution=512,
)

For focused analysis on a specific time window, use auto_fps_focus() to obtain denser sampling:

from skills.watch.scripts.frames import auto_fps_focus, extract

# Extract frames only between 30s and 45s (user-specified detail window)

fps, target = auto_fps_focus(duration_seconds=15, max_frames=100)

frames = extract(
    video_path="example.mp4",
    out_dir=Path("./detail"),
    fps=fps,
    max_frames=target,
    start_seconds=30,
    end_seconds=45,
)

In this example, the 15-second window receives up to 60 frames (approximately 4 FPS), providing significantly denser coverage than the full-video default would allocate for the same clip.

FFmpeg Integration and Budget Enforcement

The frame budget is enforced at the extraction level in skills/watch/scripts/frames.py through FFmpeg command construction. The system passes the calculated FPS and hard frame limit directly to the FFmpeg filter:

• -vf fps={fps} sets the dynamic sampling rate
• -frames:v {max_frames} provides a hard cap on total output

This dual-parameter approach guarantees that extraction terminates if the frame count reaches the budget before the video ends, effectively preventing token-cost overruns regardless of input duration.

Summary

• The frame budget system replaces fixed-FPS extraction with duration-aware capping to manage token costs.
• Short videos (≤30s) receive dense sampling at up to 12 FPS, while long videos (≥10min) fall back to strict max_frames limits.
• The system clamps all sampling to a maximum of 2 FPS (MAX_FPS) to prevent excessive frame generation.
• Focused-range mode (auto_fps_focus) allocates denser budgets (up to 6× duration) for user-specified time windows.
• Budget enforcement occurs via FFmpeg arguments -vf fps={fps} and -frames:v {max_frames} in the extraction pipeline.

Frequently Asked Questions

What is the maximum FPS allowed by the frame budget system?

The system enforces a hard cap of 2 FPS through the MAX_FPS constant in skills/watch/scripts/frames.py. Even if duration-based calculations suggest higher rates (such as 12 FPS for short clips), the final value is clamped to 2 FPS to ensure token costs remain manageable.

How does the system handle very short video clips under 5 seconds?

For focused-range requests of 5 seconds or less, auto_fps_focus() allocates a budget of up to 6 × duration frames. This means a 5-second clip could receive up to 30 frames, providing extremely dense visual coverage for detailed analysis of brief moments.

What happens when a user specifies a time range instead of analyzing the full video?

When start and end timestamps are provided, the system switches from auto_fps() to auto_fps_focus(), which applies denser sampling multipliers (6× for ≤5s, 4× for ≤15s). This "focused mode" assumes the user is zooming in for detail and therefore allocates a larger proportion of the total frame budget to that specific window.

Where is the default max_frames value configured?

The default maximum frame budget is defined in skills/watch/scripts/config.py, which watch.py imports and passes to the extraction functions. Users can override this default when calling auto_fps() or auto_fps_focus() by providing a custom max_frames parameter.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}