How Claude Video Detects Scene Changes in Balanced Detail Mode

In balanced detail mode, Claude Video uses FFmpeg's select='gt(scene,0.20)' filter to detect luminance changes exceeding 20%, falls back to uniform sampling for static videos with fewer than 8 detected shots, deduplicates perceptually similar frames, and caps the output at 100 frames.

Claude Video's balanced detail mode intelligently extracts scene-aware frames by analyzing video content rather than sampling at fixed intervals. According to the source code in the bradautomates/claude-video repository, this mode orchestrates a multi-stage pipeline involving FFmpeg scene detection, perceptual deduplication, and intelligent fallback mechanisms to ensure comprehensive video coverage.

How Balanced Mode Selects the Scene Engine

The watch command initiates scene-aware frame extraction when the user specifies detail=balanced or sets the environment variable WATCH_DETAIL=balanced. In skills/watch/scripts/watch.py, the engine selection logic routes balanced mode to the scene detection pipeline with a default frame cap of 100.

# watch.py – engine selection (balanced → scene-aware frames)

# source: watch.py L214-L218

if detail == "balanced":
    frames, frame_meta = extract_scene_or_uniform(
        video_path=local_path,
        out_dir=frame_dir,
        fps=fps,
        target_frames=target_frames,
        resolution=resolution,
        max_frames=target_frames,
        dedup=True,
    )

This invocation triggers extract_scene_or_uniform in skills/watch/scripts/frames.py, which implements the complete detection workflow.

FFmpeg Scene-Change Detection Pipeline

The core detection relies on FFmpeg's scene change filter. The extract_scene_candidates function constructs a video filter that selects the first frame plus every frame where the scene score exceeds the threshold.

# frames.py – ffmpeg scene-change filter

# source: frames.py L52-L55

vf = f"select='eq(n\\,0)+gt(scene\\,{threshold})',{_scale_filter(resolution)},showinfo"

SCENE_THRESHOLD is defined as 0.20 (20% luminance change) in the configuration section of frames.py. This threshold determines the sensitivity of cut detection—higher values require more dramatic visual changes to register as new scenes.

The FFmpeg filter outputs frame metadata including timestamps, which the parser converts into structured candidate objects for downstream processing.

Candidate Collection and Frame Metadata

After FFmpeg processing, extract_scene_candidates constructs a list of dictionaries containing frame metadata. Each entry includes the frame index, timestamp, file path, and a reason field indicating whether the frame represents the first frame or a detected scene change.

# frames.py – candidate construction

# source: frames.py L72-L79

out.append({
    "index": i,
    "timestamp_seconds": ts,
    "path": str(path),
    "reason": "first-frame" if i == 0 else "scene-change",
})

This metadata persists through the pipeline and appears in the final extraction report, allowing users to trace which frames represent actual scene boundaries versus uniform samples.

Fallback Logic for Static Content

The balanced mode implements adaptive engine selection to handle static or low-motion videos. After detecting scene candidates, the algorithm evaluates the total shot count against SCENE_MIN_FRAMES (defined as 8 in frames.py.

If scene_count >= 8, the system proceeds with the scene engine. Otherwise, it falls back to uniform sampling (evenly spaced frames) to guarantee coverage for videos lacking distinct cuts. This ensures that static presentations or single-shot recordings still yield useful frame extracts rather than empty results.

Perceptual Deduplication and Frame Capping

When the scene engine remains active, the pipeline invokes dedupe_perceptual to remove near-identical frames. This function compares perceptual hashes and drops frames with a difference ≤ DEDUP_THRESHOLD (2.0), preventing multiple captures from the same scene transition.

# frames.py – deduplication (when enabled)

# source: frames.py L43-L45

deduped, n_dropped = dedupe_perceptual(scene_frames) if dedup else (scene_frames, 0)

Following deduplication, _even_sample enforces the 100-frame cap (configurable via target_frames) by evenly distributing selections across the remaining frames while preserving the first and last frames. This sampling ensures temporal coverage without exceeding API token budgets.

The function returns comprehensive metadata including the engine used ("scene"), candidate count, deduplication statistics, and fallback status, which watch.py renders in the final report (lines L886-L898).

Practical Usage Examples

Running Claude Video in balanced mode from the command line:

python -m skills.watch.scripts.watch \
    --detail balanced \
    https://www.youtube.com/watch?v=example

Programmatic use of the scene engine:

from skills.watch.scripts.frames import extract_scene_or_uniform
from pathlib import Path

video = "sample.mp4"
out_dir = Path("./frames")
fps, _ = 2.0, None            # fps is ignored for the scene engine

target_frames = 100           # balanced default cap

frames, meta = extract_scene_or_uniform(
    video_path=video,
    out_dir=out_dir,
    fps=fps,
    target_frames=target_frames,
    resolution=512,
    max_frames=target_frames,
    dedup=True,
)

print("Engine:", meta["engine"])                # → "scene"

print("Detected shots:", meta["candidate_count"])
print("Frames kept:", len(frames))

Inspecting the metadata:

print(meta)

# {

#   "engine": "scene",

#   "candidate_count": 27,

#   "deduped_count": 5,

#   "selected_count": 22,

#   "fallback": False,

# }

Summary

• Balanced mode triggers FFmpeg scene detection with a 20% luminance threshold (SCENE_THRESHOLD=0.20) via the select='gt(scene,0.20)' filter.
• The system requires a minimum of 8 detected shots (SCENE_MIN_FRAMES=8) to use the scene engine; otherwise, it falls back to uniform sampling.
• Perceptual deduplication removes redundant frames with a hash difference threshold of 2.0 to avoid capturing multiple frames from the same cut.
• Output is capped at 100 frames by default, with even sampling applied after deduplication to maximize temporal coverage.
• All logic resides in skills/watch/scripts/frames.py and is invoked from skills/watch/scripts/watch.py when detail=balanced is specified.

Frequently Asked Questions

What threshold does Claude Video use for scene detection?

Claude Video uses a 20% luminance change threshold (0.20) for scene detection in balanced mode. This value is defined as SCENE_THRESHOLD in skills/watch/scripts/frames.py and passed to FFmpeg's select='gt(scene,0.20)' filter.

When does balanced mode fall back to uniform sampling?

Balanced mode falls back to uniform sampling when the video contains fewer than 8 distinct scene changes (SCENE_MIN_FRAMES=8). This check occurs in extract_scene_or_uniform after candidate detection, ensuring static videos or single-shot content still yield usable frame extracts.

How does the deduplication work in balanced mode?

After scene detection, the dedupe_perceptual function calculates perceptual hashes for each frame and removes duplicates where the hash difference is ≤ 2.0 (DEDUP_THRESHOLD). This eliminates redundant frames from the same scene transition while preserving visually distinct content.

What is the maximum number of frames extracted in balanced mode?

Balanced mode caps extraction at 100 frames by default, configurable via the target_frames parameter. After deduplication, the _even_sample function distributes selections evenly across the timeline to respect this cap while maintaining first and last frame coverage.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}