# How Claude Video Manages Frame Extraction for Predictable Token Costs

> Discover how Claude Video achieves predictable token costs with its budget driven frame extraction pipeline. Learn how to set frame rates based on video duration and caps for constant spending.

- Repository: [bradautomates/claude-video](https://github.com/bradautomates/claude-video)
- Tags: deep-dive
- Published: 2026-07-28

---

**Claude Video uses a budget-driven extraction pipeline that calculates target frame rates based on video duration and user-defined caps, ensuring image token costs remain constant regardless of video length.**

The `bradautomates/claude-video` repository implements a deterministic frame sampling strategy that maps video content to a fixed token budget rather than extracting frames at a constant rate. This approach prevents context window overflow when processing long videos through Claude's multimodal API. By treating frame extraction as a resource allocation problem, the tool guarantees predictable costs while preserving visual information critical for analysis.

## The Budget-Driven Extraction Pipeline

The frame extraction logic in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) operates through three coordinated stages that transform raw video into a capped set of candidate images.

### Step 1: Metadata Analysis and Auto-FPS Calculation

The pipeline begins with `get_metadata()`, which invokes `ffprobe` to retrieve duration, dimensions, and audio stream information. Using this duration data, the `auto_fps()` function (or `auto_fps_focus()` when processing a specific time range) computes a target frames-per-second value that satisfies the equation `fps = target_frames / duration` while respecting the hard constraint `MAX_FPS = 2`【/skills/watch/scripts/frames.py#L19-L38】.

The *focused* variant allocates a denser sampling budget when the user specifies `--start` and `--end` timestamps, allowing higher temporal resolution within bounded windows without exceeding the global frame cap.

### Step 2: Candidate Frame Generation via Detail Engines

Claude Video selects one of three extraction engines based on the `--detail` flag passed through the CLI:

- **Keyframe Engine** (`extract_keyframes`): Extracts only I-frames using `-skip_frame nokey` for maximum speed. This engine defaults to a 50-frame cap and works best for video navigation where scene cuts provide sufficient context【/skills/watch/scripts/frames.py#L75-L84】.

- **Scene Engine** (`extract_scene_candidates`): Runs ffmpeg's scene-change detector with `select='gt(scene,THRESHOLD)'` to identify every shot transition. This captures content boundaries more precisely than uniform sampling.

- **Uniform Fallback** (`extract`): Decodes frames at a constant FPS when scene detection yields insufficient variety. The threshold `SCENE_MIN_FRAMES = 8` determines when to switch engines; if fewer than eight scene changes are detected, the system automatically falls back to uniform sampling via `extract_scene_or_uniform()`【/skills/watch/scripts/frames.py#L21-L27】【/skills/watch/scripts/frames.py#L50-L58】.

### Step 3: Perceptual Deduplication and Even Sampling

After candidate generation, `dedupe_perceptual()` processes the JPEG frames to eliminate redundant content. The function generates 16×16 grayscale thumbnails through `_thumb_frames()` and calculates the mean absolute pixel difference between consecutive frames. Frames with a difference ≤ `DEDUP_THRESHOLD = 2.0` are discarded, removing duplicate slides or static screen recordings while preserving subtle visual changes like code diffs or terminal scrolls【/skills/watch/scripts/frames.py#L31-L38】【/skills/watch/scripts/frames.py#L63-L71】.

If the deduplicated set still exceeds the user-specified maximum, `_even_sample()` selects *n* evenly-spaced frames while preserving the first and last frames to maintain temporal boundaries【/skills/watch/scripts/frames.py#L83-L92】. The final output includes timestamps and extraction reason tags for each selected frame.

## CLI Integration and Token Budgeting

The orchestration logic in [`skills/watch/scripts/watch.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/watch.py) bridges the extraction pipeline with user-facing controls. The CLI calculates a dynamic budget using `detail_budget = max_frames - len(cue_frames)`, reserving capacity for any user-provided cue images before allocating the remainder to video frames【/skills/watch/scripts/watch.py#L95-L105】.

Based on the `--detail` argument, the system dispatches to the appropriate engine:
- `efficient`: Routes to `extract_keyframes`
- `balanced` or `token-burner`: Routes to `extract_scene_or_uniform`【/skills/watch/scripts/watch.py#L124-L132】

The final report displays the selected frames alongside a budget summary, enabling users to verify the exact image token consumption before sending the request to Claude's API. According to the repository documentation, each 512px-wide JPEG consumes approximately 197 tokens, allowing users to estimate costs precisely【/README.md#L90-L94】.

## Practical Usage Examples

Extract frames automatically based on video length with default budget caps:

```bash
/watch https://youtu.be/dQw4w9WgXcQ what happens at the 30-second mark?

```

Force a denser sampling rate for a specific time window while respecting the 2 FPS maximum and 80-frame cap:

```bash
/watch video.mp4 --start 1:15 --end 1:45 --detail balanced

```

Disable deduplication to capture every slide in a static presentation:

```bash
/watch lecture.mov --detail token-burner --no-dedup

```

Programmatically access the frame budget API from Python:

```python
from pathlib import Path
from skills.watch.scripts.frames import auto_fps, extract_scene_or_uniform

duration = 185.0  # seconds

fps, target = auto_fps(duration, max_frames=100)
frames, meta = extract_scene_or_uniform(
    video_path="example.mp4",
    out_dir=Path("./tmp"),
    fps=fps,
    target_frames=target,
    resolution=512,
    max_frames=100,
)
print(f"Selected {len(frames)} frames (engine={meta['engine']})")

```

## Summary

- **Budget-driven FPS calculation** replaces fixed frame rates with duration-aware sampling that respects user-defined caps (`max_frames`) and the `MAX_FPS = 2` safety limit.
- **Three extraction engines** (keyframe, scene detection, uniform fallback) adapt to video content density, with automatic fallback when scene changes are sparse.
- **Perceptual deduplication** removes redundant frames using 16×16 grayscale thumbnails and a threshold of 2.0 mean absolute difference, preventing token waste on static content.
- **Even sampling** ensures long videos never exceed the frame budget by distributing selections uniformly across the timeline while preserving start and end frames.
- **Explicit user controls** via `--max-frames`, `--fps`, `--resolution`, and `--no-dedup` allow precise token cost management while maintaining predictable defaults.

## Frequently Asked Questions

### How does Claude Video prevent token costs from increasing with video length?

Instead of extracting frames at a constant rate, Claude Video calculates a target FPS using `auto_fps()` based on the ratio of `max_frames` to video duration. This ensures a 10-minute video and a 1-hour video both respect the same frame cap (e.g., 80 frames), keeping image token costs constant regardless of source length【/skills/watch/scripts/frames.py#L19-L38】.

### What is the difference between the "efficient" and "balanced" detail modes?

The `efficient` mode uses `extract_keyframes()` to capture only I-frames, limiting output to 50 frames maximum for fast processing. The `balanced` mode employs `extract_scene_or_uniform()`, which detects scene changes or falls back to uniform sampling when fewer than `SCENE_MIN_FRAMES = 8` transitions are found, providing better coverage for dynamic content【/skills/watch/scripts/frames.py#L50-L58】【/skills/watch/scripts/watch.py#L124-L132】.

### How does the deduplication algorithm work without losing important content?

The `dedupe_perceptual()` function generates 16×16 grayscale thumbnails and compares mean absolute pixel differences between consecutive frames. With a conservative `DEDUP_THRESHOLD = 2.0`, the system removes only nearly identical frames (such as duplicate slides or static screens) while preserving subtle visual changes including code diffs and terminal scrolls【/skills/watch/scripts/frames.py#L63-L71】.

### Can I override the automatic frame budget calculations?

Yes. While the default logic protects against accidental overspend, users can specify exact values using `--max-frames` to set the global cap, `--fps` to force a specific extraction rate, or `--no-dedup` to disable perceptual deduplication. These parameters in [`watch.py`](https://github.com/bradautomates/claude-video/blob/main/watch.py) directly override the `auto_fps()` calculations and `dedupe_perceptual()` filtering【/skills/watch/scripts/watch.py#L95-L105】.