# How the Frame Budget System Works in claude-video: A Technical Deep Dive

> Understand claude-video's frame budget system. Learn how it optimizes token costs, prioritizes cue frames, and stays within your budget for efficient video analysis.

- Repository: [bradautomates/claude-video](https://github.com/bradautomates/claude-video)
- Tags: deep-dive
- Published: 2026-08-13

---

**The frame budget system controls token costs by calculating an optimal sampling rate that never exceeds a user-defined cap while prioritizing cue frames over automatically extracted detail frames.**

The **frame budget system** is the core token-management mechanism in `bradautomates/claude-video`, responsible for determining exactly how many frames are extracted from video content before sending them to Claude. By automatically adjusting frames-per-second (FPS) calculations and reserving capacity for specific timestamps, the system ensures predictable token usage regardless of video length. This guide examines the complete implementation across [`watch.py`](https://github.com/bradautomates/claude-video/blob/main/watch.py) and [`frames.py`](https://github.com/bradautomates/claude-video/blob/main/frames.py), explaining how the pipeline balances quality against cost constraints.

## The Three-Stage Frame Budget Pipeline

The system operates through a rigid three-stage pipeline that transforms user constraints into an executable extraction plan.

### Stage 1: Cap Resolution and the User-Provided Limit

Every request begins with a **hard cap** derived from the `--max-frames` CLI argument or the default configuration value.

In [`skills/watch/scripts/watch.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/watch.py), the entry point resolves the final budget ceiling:

```python
budget_cap = max_frames if max_frames is not None else 100

```

This assignment (lines 78-81) establishes the immutable upper bound that downstream calculations cannot violate. If the user provides no limit, the system defaults to **100 frames**, ensuring that even long videos cannot trigger unexpected token charges.

### Stage 2: Automatic FPS Selection

Once the cap is established, the system calculates an appropriate sampling density using duration-aware helpers in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py).

For **full-video scans**, the `auto_fps(duration, max_frames)` function (lines 22-38) buckets the content into thresholds (≤30 seconds, ≤60 seconds, ≤3 minutes, ≤10 minutes) and assigns progressively lower FPS values to longer content. For **focused analysis** when users specify `--start` and `--end` timestamps, `auto_fps_focus(duration, max_frames)` (lines 41-57) allocates a denser budget to capture fine details in short windows.

Both functions invoke `_clamp_fps` (lines 49-53), which enforces the global `MAX_FPS = 2.0` ceiling and mathematically guarantees that `fps × duration` never exceeds `max_frames`.

### Stage 3: Cue Reservation and Detail Allocation

The final stage implements a **reservation system** that prioritizes user-specified cue frames over algorithmic extraction.

When timestamps are provided via `--timestamps`, the system subtracts their count from the total budget before calculating the remaining allocation for detail frames:

```python
detail_budget = max_frames if max_frames is None else max(0, max_frames - len(cue_frames))

```

This logic (lines 95-97 in [`watch.py`](https://github.com/bradautomates/claude-video/blob/main/watch.py)) ensures that manual selections are always honored. The remaining `detail_budget` is passed to extraction engines like `extract_keyframes` or `extract_scene_or_uniform`. If these engines generate more candidates than the budget allows, the `_even_sample` helper (lines 83-92) performs uniform downsampling to meet the limit exactly.

## Key Implementation Files

| File | Primary Responsibility | Critical Functions |
|------|------------------------|-------------------|
| [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) | FPS calculation and sampling logic | `auto_fps()`, `auto_fps_focus()`, `_clamp_fps()`, `_even_sample()` |
| [`skills/watch/scripts/watch.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/watch.py) | CLI orchestration and budget enforcement | Cap resolution, cue reservation, result reporting |
| [`skills/watch/scripts/config.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/config.py) | Default caps per detail level | `frame_cap` configuration values |

## Practical Code Examples

### Calculating FPS for Full Video Scans

The `auto_fps` function selects conservative sampling rates for complete video analysis:

```python
from frames import auto_fps

duration = 250.0  # 4-minute video

max_frames = 100

fps, target = auto_fps(duration, max_frames)
print(f"Chosen FPS: {fps:.2f}, target frames: {target}")

```

For a 250-second video with a 100-frame cap, this returns approximately **0.4 FPS**, yielding 100 frames spread evenly across the duration while respecting the `MAX_FPS = 2.0` constraint.

### Dense Sampling for Focused Ranges

When analyzing specific segments, use `auto_fps_focus` for higher temporal resolution:

```python
from frames import auto_fps_focus

duration = 12.0  # 12-second focused clip

max_frames = 80

fps, target = auto_fps_focus(duration, max_frames)
print(f"Focused FPS: {fps:.2f}, target frames: {target}")

```

This generates approximately **6 FPS** for short windows, significantly denser than full-video scanning, yet still clamps to the 80-frame cap.

### CLI Usage with Cue Frames

The reservation system works transparently through the command line:

```bash
watch https://youtu.be/example \
      --max-frames 150 \
      --timestamps "00:10,00:45,01:20,02:00,02:30"

```

This command:
1. Reserves 5 frames for the specified timestamps
2. Calculates `detail_budget = 145` for automatic extraction
3. Selects an FPS based on the effective duration
4. Reports the final allocation in the execution summary

## Summary

- The **frame budget system** enforces a hard cap via `budget_cap` resolution in [`watch.py`](https://github.com/bradautomates/claude-video/blob/main/watch.py) (lines 78-81), defaulting to 100 frames when unspecified.
- **Automatic FPS selection** uses duration bucketing in `auto_fps()` and `auto_fps_focus()` to balance density against duration, with `_clamp_fps()` enforcing the 2.0 FPS maximum.
- **Cue reservation** prioritizes user-specified timestamps by subtracting their count from the total budget before detail extraction begins.
- **Even sampling** via `_even_sample()` ensures engines never exceed the calculated budget, trimming excess candidates uniformly when necessary.

## Frequently Asked Questions

### What happens if I request more cue frames than my max-frames limit?

The system prevents negative budgets by using `max(0, max_frames - len(cue_frames))`. If you specify 10 cue frames but set `--max-frames 5`, the `detail_budget` becomes zero, meaning no automatic frames will be extracted—only your 5 manually selected timestamps will be processed (assuming the system clips to the cap), and a warning may be generated depending on the CLI implementation.

### Why is there a maximum FPS of 2.0 hardcoded in the system?

The `MAX_FPS = 2.0` constant in [`frames.py`](https://github.com/bradautomates/claude-video/blob/main/frames.py) serves as a guardrail against excessive token usage. Even if a user specifies a high cap and short duration, the `_clamp_fps` function prevents sampling rates above 2 frames per second, ensuring that Claude receives temporally distinct frames rather than near-duplicate images that waste context window space.

### How does the frame budget system handle very long videos?

For videos exceeding 10 minutes, `auto_fps()` selects increasingly sparse sampling rates (potentially less than 0.1 FPS) to maintain the frame cap across the extended duration. The algorithm prioritizes coverage over density, ensuring the entire video timeline is represented without exceeding token limits.

### Can I see exactly how my frame budget was allocated after extraction?

Yes. The [`watch.py`](https://github.com/bradautomates/claude-video/blob/main/watch.py) script reports the target frame count, the effective cap, and the actual number of frames extracted in the final output (lines 96-102). This transparency allows you to verify that cue frames were reserved correctly and that the automatic sampling respected your specified `--max-frames` limit.