How the Frame Budget System Works in claude-video: A Technical Deep Dive

The frame budget system controls token costs by calculating an optimal sampling rate that never exceeds a user-defined cap while prioritizing cue frames over automatically extracted detail frames.

The frame budget system is the core token-management mechanism in bradautomates/claude-video, responsible for determining exactly how many frames are extracted from video content before sending them to Claude. By automatically adjusting frames-per-second (FPS) calculations and reserving capacity for specific timestamps, the system ensures predictable token usage regardless of video length. This guide examines the complete implementation across watch.py and frames.py, explaining how the pipeline balances quality against cost constraints.

The Three-Stage Frame Budget Pipeline

The system operates through a rigid three-stage pipeline that transforms user constraints into an executable extraction plan.

Stage 1: Cap Resolution and the User-Provided Limit

Every request begins with a hard cap derived from the --max-frames CLI argument or the default configuration value.

In skills/watch/scripts/watch.py, the entry point resolves the final budget ceiling:

budget_cap = max_frames if max_frames is not None else 100

This assignment (lines 78-81) establishes the immutable upper bound that downstream calculations cannot violate. If the user provides no limit, the system defaults to 100 frames, ensuring that even long videos cannot trigger unexpected token charges.

Stage 2: Automatic FPS Selection

Once the cap is established, the system calculates an appropriate sampling density using duration-aware helpers in skills/watch/scripts/frames.py.

For full-video scans, the auto_fps(duration, max_frames) function (lines 22-38) buckets the content into thresholds (≤30 seconds, ≤60 seconds, ≤3 minutes, ≤10 minutes) and assigns progressively lower FPS values to longer content. For focused analysis when users specify --start and --end timestamps, auto_fps_focus(duration, max_frames) (lines 41-57) allocates a denser budget to capture fine details in short windows.

Both functions invoke _clamp_fps (lines 49-53), which enforces the global MAX_FPS = 2.0 ceiling and mathematically guarantees that fps × duration never exceeds max_frames.

Stage 3: Cue Reservation and Detail Allocation

The final stage implements a reservation system that prioritizes user-specified cue frames over algorithmic extraction.

When timestamps are provided via --timestamps, the system subtracts their count from the total budget before calculating the remaining allocation for detail frames:

detail_budget = max_frames if max_frames is None else max(0, max_frames - len(cue_frames))

This logic (lines 95-97 in watch.py) ensures that manual selections are always honored. The remaining detail_budget is passed to extraction engines like extract_keyframes or extract_scene_or_uniform. If these engines generate more candidates than the budget allows, the _even_sample helper (lines 83-92) performs uniform downsampling to meet the limit exactly.

Key Implementation Files

File	Primary Responsibility	Critical Functions
skills/watch/scripts/frames.py	FPS calculation and sampling logic	auto_fps(), auto_fps_focus(), _clamp_fps(), _even_sample()
skills/watch/scripts/watch.py	CLI orchestration and budget enforcement	Cap resolution, cue reservation, result reporting
skills/watch/scripts/config.py	Default caps per detail level	frame_cap configuration values

Practical Code Examples

Calculating FPS for Full Video Scans

The auto_fps function selects conservative sampling rates for complete video analysis:

from frames import auto_fps

duration = 250.0  # 4-minute video

max_frames = 100

fps, target = auto_fps(duration, max_frames)
print(f"Chosen FPS: {fps:.2f}, target frames: {target}")

For a 250-second video with a 100-frame cap, this returns approximately 0.4 FPS, yielding 100 frames spread evenly across the duration while respecting the MAX_FPS = 2.0 constraint.

Dense Sampling for Focused Ranges

When analyzing specific segments, use auto_fps_focus for higher temporal resolution:

from frames import auto_fps_focus

duration = 12.0  # 12-second focused clip

max_frames = 80

fps, target = auto_fps_focus(duration, max_frames)
print(f"Focused FPS: {fps:.2f}, target frames: {target}")

This generates approximately 6 FPS for short windows, significantly denser than full-video scanning, yet still clamps to the 80-frame cap.

CLI Usage with Cue Frames

The reservation system works transparently through the command line:

watch https://youtu.be/example \
      --max-frames 150 \
      --timestamps "00:10,00:45,01:20,02:00,02:30"

This command:

1. Reserves 5 frames for the specified timestamps
2. Calculates detail_budget = 145 for automatic extraction
3. Selects an FPS based on the effective duration
4. Reports the final allocation in the execution summary

Summary

• The frame budget system enforces a hard cap via budget_cap resolution in watch.py (lines 78-81), defaulting to 100 frames when unspecified.
• Automatic FPS selection uses duration bucketing in auto_fps() and auto_fps_focus() to balance density against duration, with _clamp_fps() enforcing the 2.0 FPS maximum.
• Cue reservation prioritizes user-specified timestamps by subtracting their count from the total budget before detail extraction begins.
• Even sampling via _even_sample() ensures engines never exceed the calculated budget, trimming excess candidates uniformly when necessary.

Frequently Asked Questions

What happens if I request more cue frames than my max-frames limit?

The system prevents negative budgets by using max(0, max_frames - len(cue_frames)). If you specify 10 cue frames but set --max-frames 5, the detail_budget becomes zero, meaning no automatic frames will be extracted—only your 5 manually selected timestamps will be processed (assuming the system clips to the cap), and a warning may be generated depending on the CLI implementation.

Why is there a maximum FPS of 2.0 hardcoded in the system?

The MAX_FPS = 2.0 constant in frames.py serves as a guardrail against excessive token usage. Even if a user specifies a high cap and short duration, the _clamp_fps function prevents sampling rates above 2 frames per second, ensuring that Claude receives temporally distinct frames rather than near-duplicate images that waste context window space.

How does the frame budget system handle very long videos?

For videos exceeding 10 minutes, auto_fps() selects increasingly sparse sampling rates (potentially less than 0.1 FPS) to maintain the frame cap across the extended duration. The algorithm prioritizes coverage over density, ensuring the entire video timeline is represented without exceeding token limits.

Can I see exactly how my frame budget was allocated after extraction?

Yes. The watch.py script reports the target frame count, the effective cap, and the actual number of frames extracted in the final output (lines 96-102). This transparency allows you to verify that cue frames were reserved correctly and that the automatic sampling respected your specified --max-frames limit.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}