How Scene-Change Detection Works in claude-video's Balanced vs. Token-Burner Modes

Scene-change detection in claude-video uses ffmpeg's scene filter to identify visual transitions, with balanced mode capping results at 100 frames and token-burner mode keeping every detected scene change. Both modes share identical detection logic but differ only in their max_frames parameter passed to extract_scene_or_uniform.

claude-video, an open-source video analysis tool, offers two scene-aware processing modes: balanced and token-burner. This article explains how both leverage ffmpeg scene detection and where they diverge in frame sampling strategy, based on the source code in the bradautomates/claude-video repository.

Scene-Change Detection Pipeline

Both modes execute the same four-stage pipeline defined in skills/watch/scripts/frames.py. The entry point extract_scene_or_uniform orchestrates detection, fallback, deduplication, and final sampling.

FFmpeg Scene Filter Execution

The core detection happens in extract_scene_candidates, which constructs an ffmpeg video filter:

vf = f"select='eq(n\\,0)+gt(scene\\,{threshold})',{_scale_filter(resolution)},showinfo"

This filter string performs three critical functions:

• eq(n\,0) — always selects the first frame (timestamp zero)
• gt(scene, threshold) — selects frames where ffmpeg's internal scene metric exceeds SCENE_THRESHOLD = 0.20
• showinfo — emits presentation timestamps for parsing candidate frames

The threshold of 0.20 (20% difference) filters out minor lighting changes while capturing genuine shot transitions.

Uniform Sampling Fallback

If a video contains fewer than SCENE_MIN_FRAMES = 8 scene cuts, the engine assumes static content (like screen recordings). It automatically falls back to simple uniform sampling via the extract function. This guarantees every video produces usable frames regardless of visual variety.

Perceptual Deduplication

After raw candidates are collected, dedupe_perceptual processes them unless --no-dedup is specified. The function:

1. Generates 16×16 grayscale thumbnails (DEDUP_THUMB = 16)
2. Computes mean-pixel difference between consecutive frames
3. Drops frames with difference ≤ 2.0 (DEDUP_THRESHOLD = 2.0)

This removes near-identical frames from the same shot while preserving distinct visual content.

Balanced Mode: Capped Scene-Aware Sampling

Balanced mode targets typical use cases with a 100-frame ceiling.

The configuration in skills/watch/scripts/config.py defines:

DEFAULT_DETAIL = "balanced"

def frame_cap(detail: str) -> Optional[int]:
    caps = {
        "low": 20,
        "medium": 50,
        "balanced": 100,
        "high": 200,
        "token-burner": None,  # uncapped

    }
    return caps.get(detail)

After deduplication, _even_sample distributes frames evenly: it always keeps the first and last frames, then spaces intermediates uniformly until reaching the cap. This ensures temporal coverage without clustering around high-activity regions.

Invoke balanced mode:

watch https://example.com/video.mp4 --detail balanced

Or programmatically:

from pathlib import Path
from skills.watch.scripts.frames import extract_scene_or_uniform

frames, meta = extract_scene_or_uniform(
    video_path="myvideo.mp4",
    out_dir=Path("./out"),
    fps=1.0,                # ignored in scene mode

    target_frames=100,
    resolution=512,
    max_frames=100,         # cap enforces balanced behavior

    start_seconds=None,
    end_seconds=None,
    dedup=True,
)

Token-Burner Mode: Uncapped Scene Extraction

Token-burner mode removes the ceiling entirely, returning every deduplicated scene change with no thinning.

The same frame_cap function returns None for this mode, which propagates to _even_sample as max_frames=None. The sampling function detects this and returns all frames without reduction.

This mode suits forensic analysis, animation review, or any scenario where missing a single shot transition carries high cost.

Invoke token-burner mode:

watch https://example.com/video.mp4 --detail token-burner

Or programmatically:

from pathlib import Path
from skills.watch.scripts.frames import extract_scene_or_uniform

frames, meta = extract_scene_or_uniform(
    video_path="myvideo.mp4",
    out_dir=Path("./out"),
    fps=1.0,
    target_frames=100,      # ignored when max_frames=None

    resolution=512,
    max_frames=None,        # uncapped: all scene changes kept

    start_seconds=None,
    end_seconds=None,
    dedup=True,
)

Mode Selection Logic in watch.py

The dispatch around line 214 in skills/watch/scripts/watch.py determines which mode activates:

detail_budget = frame_cap(detail)  # 100 for balanced, None for token-burner

frames, frame_meta = extract_scene_or_uniform(
    video_path=video_path,
    out_dir=out_dir,
    fps=fps,
    target_frames=target_frames,
    resolution=resolution,
    max_frames=detail_budget,  # the only differing parameter

    start_seconds=start_seconds,
    end_seconds=end_seconds,
    dedup=not no_dedup,
)

Both branches call identical functions; only the max_frames argument changes. This architectural choice eliminates duplication while providing clear behavioral separation.

Summary

• Scene-change detection uses ffmpeg's scene filter with a 0.20 threshold in extract_scene_candidates (frames.py)
• Balanced mode caps results at 100 frames via _even_sample, ensuring consistent token usage
• Token-burner mode sets max_frames=None, preserving all detected scene changes after deduplication
• Both modes rely on perceptual deduplication (16×16 thumbnails, difference threshold 2.0) to remove redundant frames
• The uniform fallback triggers for videos with fewer than 8 scene cuts, preventing empty results on static content

Frequently Asked Questions

How does claude-video decide between scene detection and uniform sampling?

The extract_scene_or_uniform function attempts scene detection first. If fewer than SCENE_MIN_FRAMES = 8 candidates emerge, it automatically falls back to uniform sampling. This ensures screen recordings and static videos still yield representative frames.

Can I adjust the scene detection sensitivity?

The threshold is hardcoded at SCENE_THRESHOLD = 0.20 in frames.py. Modifying this value requires editing the source; there is no command-line flag. Lower values increase sensitivity (more frames), while higher values reduce it.

Why does token-burner mode still perform deduplication?

Deduplication runs unless --no-dedup is passed. Token-burner removes the frame cap, not redundancy filtering. Near-identical frames (mean difference ≤ 2.0) still collapse, preventing wasted tokens on visually duplicate content.

What ffmpeg version is required for scene detection?

Any ffmpeg build with the select filter supports scene detection. The scene metric has been stable since ffmpeg 3.0 (2016). No external dependencies beyond standard ffmpeg are needed.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}