How the Frame Budget Algorithm Scales with Video Duration in Balanced Mode

In balanced mode, the frame budget algorithm caps extraction at 100 frames total and dynamically lowers the sampling rate as video duration increases to maintain consistent processing overhead.

The bradautomates/claude-video repository implements an intelligent balanced mode that manages computational resources when analyzing video content. This mode employs a frame budget algorithm that scales inversely with video length, ensuring short clips retain temporal detail while enforcing strict limits on long-form content.

Frame Cap Configuration

The balanced mode enforces a hard limit of 100 frames per video, defined in skills/watch/scripts/config.py. The frame_cap() function returns this ceiling when called with the "balanced" detail parameter.

# skills/watch/scripts/config.py (lines 65-74)

def frame_cap(detail_mode: str) -> int:
    caps = {
        "low": 50,
        "balanced": 100,  # Maximum frame budget

        "high": 200
    }
    return caps.get(detail_mode, 100)

This constant frame budget ensures predictable memory usage and API processing costs regardless of input video length.

Duration-Based Scaling Rules

The scaling logic resides in skills/watch/scripts/frames.py within the auto_fps() function (lines 22-38). This function calculates a target frame count based on video duration, then derives the FPS required to hit that target without exceeding MAX_FPS = 2.0.

The algorithm applies five distinct duration tiers:

• ≤ 30 seconds: Targets min(100, max(12, ⌈duration⌉)) frames (12–30 frames), yielding approximately 0.4–1 fps (roughly one frame per second)
• 30–60 seconds: Fixed target of 40 frames, resulting in 0.67–1.33 fps
• 60–180 seconds: Fixed target of 60 frames, resulting in 0.33–1 fps
• 180–600 seconds: Fixed target of 80 frames, resulting in 0.13–0.44 fps
• > 600 seconds (10 minutes): Uses the full 100-frame budget, calculating FPS as 100 / duration (very low sampling rates for long videos)

As video duration increases, the algorithm maintains the frame budget by proportionally decreasing the extraction frequency.

Implementation Workflow

The orchestration logic in skills/watch/scripts/watch.py coordinates the frame budget application. When processing begins, the script:

1. Retrieves the configured cap via frame_cap("balanced") → 100
2. Passes the video duration and cap to auto_fps(duration, max_frames)
3. Receives the calculated FPS and target frame count
4. Falls back to scene detection if the detected scenes yield fewer frames than the budget; otherwise applies uniform FPS extraction

This workflow ensures that the 100-frame ceiling is respected while preferring scene-based segmentation when temporally sparse.

Practical Usage

Calculate the exact FPS budget programmatically using the internal utilities:

from skills.watch.scripts.config import frame_cap
from skills.watch.scripts.frames import auto_fps

detail = "balanced"
max_frames = frame_cap(detail)          # Returns 100

fps, target = auto_fps(45, max_frames)  # 45-second video

print(f"FPS: {fps:.2f}, Target frames: {target}")

# Output: FPS: 0.89, Target frames: 40

From the command line, balanced mode is the default detail setting:

python -m skills.watch.scripts.watch my_video.mp4 --detail balanced

The CLI automatically invokes auto_fps() with the 100-frame cap, applying the duration-based scaling rules without requiring manual FPS calculation.

Summary

• Hard cap: Balanced mode enforces a strict 100-frame limit via frame_cap() in skills/watch/scripts/config.py
• Inverse scaling: The auto_fps() function decreases sampling rates as duration increases to maintain the budget
• Five tiers: Duration thresholds at 30s, 60s, 180s, and 600s trigger specific frame targets (12-30, 40, 60, 80, 100)
• FPS ceiling: Extraction never exceeds 2.0 FPS, even for very short clips
• Scene preference: If scene detection produces fewer frames than the budget, the engine uses scene boundaries instead of uniform sampling

Frequently Asked Questions

What is the maximum number of frames extracted in balanced mode?

The algorithm caps extraction at 100 frames regardless of video length. This limit is defined in skills/watch/scripts/config.py and applies uniformly to all videos processed in balanced mode.

How does the algorithm handle videos longer than 10 minutes?

For videos exceeding 600 seconds, the algorithm utilizes the full 100-frame budget. It calculates FPS as 100 / duration, resulting in very sparse sampling (e.g., a 20-minute video extracts at approximately 0.083 fps, or one frame every 12 seconds).

What happens if scene detection finds fewer frames than the budget?

When the scene detection engine identifies fewer keyframes than the calculated budget, the system prefers the scene-based frames over uniform FPS extraction. This ensures that natural transition points are preserved even if the total count falls below the 100-frame target.

Why is the FPS capped at 2.0 in balanced mode?

The MAX_FPS = 2.0 constant in skills/watch/scripts/frames.py prevents excessive sampling of short clips, ensuring that the frame budget is distributed intelligently rather than extracting redundant near-duplicate frames from high-frame-rate source material.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}