How Claude-Video Performs Scene-Change Detection in Balanced Mode

In balanced mode, Claude-Video uses ffmpeg's scene detection filter to identify frames with greater than 20% luminance change, deduplicates perceptually similar shots, and caps the output at 100 frames while falling back to uniform sampling for static content.

The bradautomates/claude-video repository implements intelligent frame extraction through multiple detail modes. When you invoke the balanced detail mode, the system prioritizes scene-change detection to capture distinct visual moments without exceeding context window limits. This approach ensures comprehensive coverage of video content by focusing on editorial cuts and significant visual transitions.

Engine Selection in Watch.py

The entry point for balanced mode resides in skills/watch/scripts/watch.py, where the detail parameter determines the extraction strategy. When set to "balanced" (or via the WATCH_DETAIL=balanced environment variable), the code invokes extract_scene_or_uniform with a default frame budget of 100.

# watch.py – engine selection (balanced → scene-aware frames)

# source: watch.py L214-L218

if detail == "balanced":
    frames, frame_meta = extract_scene_or_uniform(
        # ... parameters including target_frames=100

    )

This function call initiates the full scene-detection pipeline, which returns both the selected frames and metadata about the extraction process.

FFmpeg Scene Detection Implementation

The core detection logic lives in skills/watch/scripts/frames.py within the extract_scene_candidates function. This implementation leverages ffmpeg's scene filter to identify meaningful visual boundaries.

Scene Threshold Configuration

The system defines a 20% luminance change threshold through the SCENE_THRESHOLD constant (0.20) at lines 20-21. The ffmpeg filter expression combines this threshold with scaling and showinfo filters to output relevant frames.

# frames.py – ffmpeg scene-change filter

# source: frames.py L52-L55

vf = f"select='eq(n\\,0)+gt(scene\\,{threshold})',{_scale_filter(resolution)},showinfo"

This filter always captures the first frame (eq(n,0)) plus any frame where the scene score exceeds 0.20, ensuring you receive both the opening shot and all subsequent scene transitions.

Candidate Frame Extraction

Each detected frame becomes a structured dictionary containing temporal and spatial metadata. The function returns a list of candidates with their capture reason marked as either "first-frame" or "scene-change".

# frames.py – candidate construction

# source: frames.py L72-L79

out.append({
    "index": i,
    "timestamp_seconds": ts,
    "path": str(path),
    "reason": "first-frame" if i == 0 else "scene-change",
})

Fallback and Frame Optimization

After initial detection, the system validates whether the scene engine produced sufficient visual diversity to justify its use.

Minimum Scene Requirements

The code checks if scene_count meets or exceeds SCENE_MIN_FRAMES (defined as 8 at lines 26-27). If the video contains fewer than 8 distinct shots—indicating static or single-scene content—the system abandons the scene engine in favor of uniform sampling. This fallback guarantees meaningful coverage even for footage without editorial cuts.

Perceptual Deduplication

When the scene engine produces enough candidates, dedupe_perceptual removes near-duplicate frames using a hash difference threshold of 2.0 (DEDUP_THRESHOLD). This step eliminates redundant frames from gradual transitions or false positives while preserving distinct visual states.

# frames.py – deduplication (when enabled)

# source: frames.py L43-L45

deduped, n_dropped = dedupe_perceptual(scene_frames) if dedup else (scene_frames, 0)

Sampling and Output Constraints

Balanced mode enforces a hard cap of 100 frames (target_frames) to manage token consumption. After deduplication, the _even_sample function distributes selections evenly across the timeline, always preserving the first and last frames to maintain temporal boundaries.

# frames.py – even sampling of the deduped list

# source: frames.py L45-L48

cap = len(deduped) if max_frames is None else max_frames
selected = _even_sample(deduped, cap)

The final metadata object reports the engine used ("scene"), candidate counts, deduplication statistics, and fallback status, providing transparency into the extraction process.

Practical Usage Examples

Run Claude-Video in balanced mode from the command line:

python -m skills.watch.scripts.watch \
    --detail balanced \
    https://www.youtube.com/watch?v=example

Programmatically invoke the scene engine with custom parameters:

from skills.watch.scripts.frames import extract_scene_or_uniform
from pathlib import Path

video = "sample.mp4"
out_dir = Path("./frames")
target_frames = 100

frames, meta = extract_scene_or_uniform(
    video_path=video,
    out_dir=out_dir,
    fps=2.0,  # Ignored by scene engine but required

    target_frames=target_frames,
    resolution=512,
    max_frames=target_frames,
    dedup=True,
)

print("Engine:", meta["engine"])                # → "scene"

print("Detected shots:", meta["candidate_count"])
print("Frames kept:", len(frames))

Inspect the metadata to verify extraction behavior:

print(meta)

# {

#   "engine": "scene",

#   "candidate_count": 27,

#   "deduped_count": 5,

#   "selected_count": 22,

#   "fallback": False,

# }

Summary

• Balanced mode activates the scene engine via extract_scene_or_uniform in skills/watch/scripts/frames.py.
• Scene detection relies on ffmpeg's select='gt(scene,0.20)' filter to identify 20% luminance changes.
• The system requires a minimum of 8 distinct scenes; otherwise, it falls back to uniform sampling.
• Perceptual deduplication removes similar frames using a hash threshold of 2.0 before final selection.
• Output is capped at 100 frames by default, with even sampling applied to distribute selections temporally.

Frequently Asked Questions

What luminance threshold triggers scene detection in balanced mode?

Claude-Video uses a 20% luminance change threshold (SCENE_THRESHOLD = 0.20) to identify distinct shots. When ffmpeg calculates a frame-to-frame difference exceeding this value via the scene filter, it marks that frame as a scene boundary. This threshold effectively catches editorial cuts while filtering out minor lighting fluctuations and camera noise.

How does the system handle videos with minimal scene changes?

If the detector identifies fewer than 8 distinct scenes (SCENE_MIN_FRAMES), the engine automatically falls back to uniform sampling. This ensures static videos, screen recordings, or single-shot content still provide useful frame coverage across the entire timeline. The fallback mechanism guarantees the 100-frame budget is always populated with temporally distributed visual data.

Can I disable perceptual deduplication when using balanced mode?

Yes, you can disable deduplication by passing dedup=False to the extract_scene_or_uniform function. When enabled (the default), the system uses perceptual hashing to remove frames with hash differences of 2.0 or less. Disabling this feature preserves all scene-change candidates, which may be useful when analyzing videos with subtle visual variations that should be treated as distinct frames.

Is the 100-frame maximum configurable in balanced mode?

Absolutely. While balanced mode defaults to 100 frames, you can override this by modifying the target_frames parameter in your Python code or adjusting the relevant CLI argument. However, the underlying scene threshold (0.20) and minimum scene requirements (8 frames) are constants defined in skills/watch/scripts/frames.py that require source modification to change.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}