How Claude Video Manages Frame Extraction for Predictable Token Costs

Claude Video uses a budget-driven extraction pipeline that calculates target frame rates based on video duration and user-defined caps, ensuring image token costs remain constant regardless of video length.

The bradautomates/claude-video repository implements a deterministic frame sampling strategy that maps video content to a fixed token budget rather than extracting frames at a constant rate. This approach prevents context window overflow when processing long videos through Claude's multimodal API. By treating frame extraction as a resource allocation problem, the tool guarantees predictable costs while preserving visual information critical for analysis.

The Budget-Driven Extraction Pipeline

The frame extraction logic in skills/watch/scripts/frames.py operates through three coordinated stages that transform raw video into a capped set of candidate images.

Step 1: Metadata Analysis and Auto-FPS Calculation

The pipeline begins with get_metadata(), which invokes ffprobe to retrieve duration, dimensions, and audio stream information. Using this duration data, the auto_fps() function (or auto_fps_focus() when processing a specific time range) computes a target frames-per-second value that satisfies the equation fps = target_frames / duration while respecting the hard constraint MAX_FPS = 2【/skills/watch/scripts/frames.py#L19-L38】.

The focused variant allocates a denser sampling budget when the user specifies --start and --end timestamps, allowing higher temporal resolution within bounded windows without exceeding the global frame cap.

Step 2: Candidate Frame Generation via Detail Engines

Claude Video selects one of three extraction engines based on the --detail flag passed through the CLI:

• Keyframe Engine (extract_keyframes): Extracts only I-frames using -skip_frame nokey for maximum speed. This engine defaults to a 50-frame cap and works best for video navigation where scene cuts provide sufficient context【/skills/watch/scripts/frames.py#L75-L84】.

• Scene Engine (extract_scene_candidates): Runs ffmpeg's scene-change detector with select='gt(scene,THRESHOLD)' to identify every shot transition. This captures content boundaries more precisely than uniform sampling.

• Uniform Fallback (extract): Decodes frames at a constant FPS when scene detection yields insufficient variety. The threshold SCENE_MIN_FRAMES = 8 determines when to switch engines; if fewer than eight scene changes are detected, the system automatically falls back to uniform sampling via extract_scene_or_uniform()【/skills/watch/scripts/frames.py#L21-L27】【/skills/watch/scripts/frames.py#L50-L58】.

Step 3: Perceptual Deduplication and Even Sampling

After candidate generation, dedupe_perceptual() processes the JPEG frames to eliminate redundant content. The function generates 16×16 grayscale thumbnails through _thumb_frames() and calculates the mean absolute pixel difference between consecutive frames. Frames with a difference ≤ DEDUP_THRESHOLD = 2.0 are discarded, removing duplicate slides or static screen recordings while preserving subtle visual changes like code diffs or terminal scrolls【/skills/watch/scripts/frames.py#L31-L38】【/skills/watch/scripts/frames.py#L63-L71】.

If the deduplicated set still exceeds the user-specified maximum, _even_sample() selects n evenly-spaced frames while preserving the first and last frames to maintain temporal boundaries【/skills/watch/scripts/frames.py#L83-L92】. The final output includes timestamps and extraction reason tags for each selected frame.

CLI Integration and Token Budgeting

The orchestration logic in skills/watch/scripts/watch.py bridges the extraction pipeline with user-facing controls. The CLI calculates a dynamic budget using detail_budget = max_frames - len(cue_frames), reserving capacity for any user-provided cue images before allocating the remainder to video frames【/skills/watch/scripts/watch.py#L95-L105】.

Based on the --detail argument, the system dispatches to the appropriate engine:

• efficient: Routes to extract_keyframes
• balanced or token-burner: Routes to extract_scene_or_uniform【/skills/watch/scripts/watch.py#L124-L132】

The final report displays the selected frames alongside a budget summary, enabling users to verify the exact image token consumption before sending the request to Claude's API. According to the repository documentation, each 512px-wide JPEG consumes approximately 197 tokens, allowing users to estimate costs precisely【/README.md#L90-L94】.

Practical Usage Examples

Extract frames automatically based on video length with default budget caps:

/watch https://youtu.be/dQw4w9WgXcQ what happens at the 30-second mark?

Force a denser sampling rate for a specific time window while respecting the 2 FPS maximum and 80-frame cap:

/watch video.mp4 --start 1:15 --end 1:45 --detail balanced

Disable deduplication to capture every slide in a static presentation:

/watch lecture.mov --detail token-burner --no-dedup

Programmatically access the frame budget API from Python:

from pathlib import Path
from skills.watch.scripts.frames import auto_fps, extract_scene_or_uniform

duration = 185.0  # seconds

fps, target = auto_fps(duration, max_frames=100)
frames, meta = extract_scene_or_uniform(
    video_path="example.mp4",
    out_dir=Path("./tmp"),
    fps=fps,
    target_frames=target,
    resolution=512,
    max_frames=100,
)
print(f"Selected {len(frames)} frames (engine={meta['engine']})")

Summary

• Budget-driven FPS calculation replaces fixed frame rates with duration-aware sampling that respects user-defined caps (max_frames) and the MAX_FPS = 2 safety limit.
• Three extraction engines (keyframe, scene detection, uniform fallback) adapt to video content density, with automatic fallback when scene changes are sparse.
• Perceptual deduplication removes redundant frames using 16×16 grayscale thumbnails and a threshold of 2.0 mean absolute difference, preventing token waste on static content.
• Even sampling ensures long videos never exceed the frame budget by distributing selections uniformly across the timeline while preserving start and end frames.
• Explicit user controls via --max-frames, --fps, --resolution, and --no-dedup allow precise token cost management while maintaining predictable defaults.

Frequently Asked Questions

How does Claude Video prevent token costs from increasing with video length?

Instead of extracting frames at a constant rate, Claude Video calculates a target FPS using auto_fps() based on the ratio of max_frames to video duration. This ensures a 10-minute video and a 1-hour video both respect the same frame cap (e.g., 80 frames), keeping image token costs constant regardless of source length【/skills/watch/scripts/frames.py#L19-L38】.

What is the difference between the "efficient" and "balanced" detail modes?

The efficient mode uses extract_keyframes() to capture only I-frames, limiting output to 50 frames maximum for fast processing. The balanced mode employs extract_scene_or_uniform(), which detects scene changes or falls back to uniform sampling when fewer than SCENE_MIN_FRAMES = 8 transitions are found, providing better coverage for dynamic content【/skills/watch/scripts/frames.py#L50-L58】【/skills/watch/scripts/watch.py#L124-L132】.

How does the deduplication algorithm work without losing important content?

The dedupe_perceptual() function generates 16×16 grayscale thumbnails and compares mean absolute pixel differences between consecutive frames. With a conservative DEDUP_THRESHOLD = 2.0, the system removes only nearly identical frames (such as duplicate slides or static screens) while preserving subtle visual changes including code diffs and terminal scrolls【/skills/watch/scripts/frames.py#L63-L71】.

Can I override the automatic frame budget calculations?

Yes. While the default logic protects against accidental overspend, users can specify exact values using --max-frames to set the global cap, --fps to force a specific extraction rate, or --no-dedup to disable perceptual deduplication. These parameters in watch.py directly override the auto_fps() calculations and dedupe_perceptual() filtering【/skills/watch/scripts/watch.py#L95-L105】.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}