How the bradautomates/claude-video Frame Budget System Works: Cost-Controlled Video Processing

The bradautomates/claude-video frame budget system automatically calculates optimal frame extraction rates based on video duration, caps extraction at configurable limits, and applies perceptual deduplication to deliver predictable token costs for downstream LLM processing.

The bradautomates/claude-video repository implements an intelligent budgeting mechanism that prevents runaway token costs when processing video content through Claude or other LLMs. Located primarily in skills/watch/scripts/frames.py, this system dynamically adjusts frames-per-second (FPS) sampling and enforces hard caps to ensure that both short clips and long videos remain within predictable processing budgets.

Understanding the Three-Stage Budget Architecture

The frame budget system operates through a pipeline of three distinct stages: target calculation, constrained extraction, and intelligent deduplication. Each stage ensures that the final frame count aligns with the video's duration while respecting the global max_frames parameter (default: 100).

Stage 1: Calculating Target Frame Counts

The system employs two specialized functions to determine appropriate sampling rates: auto_fps() for full-video analysis and auto_fps_focus() for restricted time ranges.

Full-Video Budget Calculation with auto_fps()

For complete video scans, auto_fps() in skills/watch/scripts/frames.py implements a tiered budgeting strategy that allocates frames based on duration:

• Short clips (< 30 seconds): Generous budget of up to 12 frames plus one frame per second
• 30-60 seconds: Capped at 40 frames
• 1-3 minutes: Capped at 60 frames
• 3-10 minutes: Capped at 80 frames
• Longer videos: Limited to the user-specified max_frames value

The function also enforces MAX_FPS = 2.0, ensuring extraction never exceeds two frames per second regardless of duration.

# skills/watch/scripts/frames.py – auto_fps() implementation

def auto_fps(duration_seconds: float, max_frames: int = 100) -> tuple[float, int]:
    if duration_seconds <= 30:
        target = min(max_frames, max(12, int(round(duration_seconds)))
    elif duration_seconds <= 60:
        target = min(max_frames, 40)
    # ... additional duration tiers

    return _clamp_fps(target / duration_seconds, duration_seconds, max_frames)

Focused Segment Budget Calculation with auto_fps_focus()

When users specify time ranges using --start and --end parameters, auto_fps_focus() applies a steeper sampling rate to preserve detail in the restricted window. For segments under 5 seconds, it allocates up to 10 frames or 6 frames per second, whichever is higher.

# skills/watch/scripts/frames.py – auto_fps_focus() implementation

def auto_fps_focus(duration_seconds: float, max_frames: int = 100) -> tuple[float, int]:
    if duration_seconds <= 5:
        target = min(max_frames, max(10, int(round(duration_seconds * 6))))
    # ... additional logic for longer focused segments

    return _clamp_fps(target / duration_seconds, duration_seconds, max_frames)

Stage 2: Constrained Frame Extraction with ffmpeg

Once the target FPS is calculated, the extract() function passes both the sampling rate and global max_frames limit to an ffmpeg subprocess. The -frames:v parameter acts as a hard ceiling, automatically truncating extraction if the FPS calculation yields more frames than the budget allows.

# skills/watch/scripts/frames.py – extract() method

cmd = [
    "ffmpeg", "-hide_banner", "-loglevel", "error", "-y",
    "-i", str(Path(video_path).resolve()),
    "-vf", f"fps={fps},{_scale_filter(resolution)}",
    "-frames:v", str(max_frames), "-q:v", "4", output_pattern,
]

This dual-layer enforcement ensures that even if duration calculations drift, the physical extraction halts at the defined budget boundary.

Stage 3: Perceptual Deduplication and Final Sampling

After extraction, frames undergo dedupe_perceptual() to eliminate visually redundant content before final budget enforcement. This process:

1. Generates 16x16 grayscale thumbnails (DEDUP_THUMB = 16) for each frame
2. Computes mean-pixel deltas between consecutive frames
3. Drops frames falling below DEDUP_THRESHOLD = 2.0

The surviving frames then pass through _even_sample(), which performs stratified sampling to reach the final target count while preserving the first and last frames for temporal context.

# skills/watch/scripts/frames.py – deduplication and sampling logic

def dedupe_perceptual(candidates, threshold=DEDUP_THRESHOLD):
    # ... thumbnail generation and delta calculation

    return _dedupe_by_deltas(candidates, thumbs, threshold)

def _even_sample(candidates, n):
    selected = [candidates[i] for i in _even_indices(len(candidates), n)]
    # ... return evenly distributed frame set

Practical Implementation Examples

Running a Full-Video Extraction

Execute the default budget calculation against an entire video file:

python -m skills.watch.scripts.frames /path/to/video.mp4 /tmp/frames

This automatically invokes auto_fps() to determine the appropriate sampling rate and extracts frames according to the duration-based budget tiers.

Extracting Focused Segments

For analyzing specific time ranges with higher sampling density:

python -m skills.watch.scripts.frames \
    /path/to/video.mp4 /tmp/frames --start 1:30 --end 2:00

This triggers auto_fps_focus(), applying the steeper budget curve suitable for short temporal windows while maintaining the same max_frames safety cap.

Summary

• Duration-based budgeting: The system automatically allocates more frames per second to short videos while capping long videos at fixed ceilings (40-80 frames depending on length).
• Dual enforcement: Budget limits are enforced both mathematically in auto_fps() and physically via ffmpeg -frames:v parameters.
• Perceptual deduplication: Near-identical frames are eliminated using 16x16 grayscale thumbnails before final sampling, ensuring budget slots hold unique visual information.
• Configurable constants: Key thresholds like MAX_FPS = 2.0 and DEDUP_THRESHOLD = 2.0 reside in skills/watch/scripts/config.py for easy adjustment.
• Range-specific optimization: The auto_fps_focus() function provides denser sampling when analyzing video segments specified by --start and --end timestamps.

Frequently Asked Questions

What is the maximum FPS limit in the frame budget system?

The bradautomates/claude-video frame budget system enforces a hard ceiling of MAX_FPS = 2.0 frames per second, as defined in skills/watch/scripts/config.py. This limit applies regardless of video duration or target calculations, ensuring that even very short clips never exceed two extracted frames per second.

How does the system handle short videos versus long videos?

Short videos receive preferential treatment with denser sampling: clips under 30 seconds receive up to 12 frames plus one frame per second, while segments under 5 seconds (when using focused mode) receive up to 6 frames per second. Long videos transition to fixed caps: 40 frames for 30-60 seconds, 60 frames for 1-3 minutes, and 80 frames for up to 10 minutes, preventing excessive token costs from extended content.

What is perceptual deduplication and how does it affect the frame budget?

Perceptual deduplication is a post-extraction optimization in dedupe_perceptual() that removes visually redundant frames before final budget enforcement. It creates 16x16 grayscale thumbnails of each extracted frame and compares mean-pixel deltas; frames with differences below DEDUP_THRESHOLD = 2.0 are discarded. This ensures the final frame budget contains only visually distinct content, maximizing information density per allocated token.

How do I extract frames from a specific time range using the budget system?

Specify --start and --end timestamps in HH:MM:SS or MM:SS format when invoking the frames module. This triggers auto_fps_focus() instead of auto_fps(), applying a steeper sampling curve appropriate for the restricted duration while maintaining the same max_frames safety cap. For example: python -m skills.watch.scripts.frames video.mp4 /output --start 1:30 --end 2:00.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}