How Scene-Change Detection Works with SCENE_THRESHOLD and When Uniform Sampling Kicks In

Scene-change detection in claude-video uses FFmpeg's scene filter with a SCENE_THRESHOLD of 0.20 to detect visual cuts, and automatically falls back to uniform sampling when fewer than 8 scene changes are detected.

The bradautomates/claude-video repository implements an adaptive keyframe extraction system that prioritizes meaningful scene transitions over arbitrary timestamps. This article explains the mechanics of scene-change detection, the role of the SCENE_THRESHOLD constant, and the precise conditions that trigger uniform sampling.

What is SCENE_THRESHOLD and How It Detects Scene Changes

The SCENE_THRESHOLD constant defines the sensitivity of FFmpeg's scene-change detection filter. As implemented in skills/watch/scripts/frames.py (line 20), the default value is 0.20:

SCENE_THRESHOLD: float = 0.20

This threshold is passed to the extract_scene_candidates function, which constructs an FFmpeg video filter expression (lines 224–226):

vf = f"select='eq(n\\,0)+gt(scene\\,{threshold})',{_scale_filter(resolution)},showinfo"

The filter components break down as follows:

• eq(n,0) — always select the first frame as a baseline reference
• gt(scene,{threshold}) — select frames where the scene-change score exceeds the threshold
• _scale_filter(resolution) — downscale frames for efficient processing
• showinfo — emit metadata about selected frames

FFmpeg computes the scene metric by comparing histogram differences between consecutive frames. When this difference exceeds SCENE_THRESHOLD, FFmpeg emits that frame as a candidate keyframe. Higher thresholds yield fewer, more dramatic scene changes; lower thresholds capture subtle transitions.

When the System Falls Back to Uniform Sampling

Scene-change detection alone cannot guarantee sufficient coverage for all video types. The extract_scene_or_uniform function (lines 510–543) implements a reliability check:

def extract_scene_or_uniform(
    video_path: Path,
    out_dir: Path,
    max_frames: Optional[int] = None,
    dedupe: bool = True,
) -> Tuple[List[Path], Dict]:
    # ... scene detection attempt ...

    scene_frames = extract_scene_candidates(...)
    scene_count = len(scene_frames)
    
    if scene_count >= SCENE_MIN_FRAMES:  # SCENE_MIN_FRAMES = 8

        return scene_frames, {"engine": "scene", "count": scene_count}
    
    # Fall back to uniform sampling

    timestamps = _generate_uniform_timestamps(video_path, target_count=8)
    uniform_frames = extract_at_timestamps(video_path, timestamps, out_dir)
    return uniform_frames, {"engine": "uniform", "count": len(uniform_frames)}

The fallback condition is straightforward: if fewer than SCENE_MIN_FRAMES (8) scene changes are detected, the system switches to uniform sampling. This handles:

• Static videos with minimal visual change
• Slow-motion or single-shot footage
• Screen recordings with gradual transitions

The Uniform Sampling Implementation

Uniform sampling distributes frames evenly across the video duration. The extract_at_timestamps function receives precomputed timestamps and extracts frames at those specific points, ensuring consistent coverage regardless of visual content.

Key differences between the two engines:

Engine	Selection Criteria	Use Case
Scene	Visual difference > 0.20	Videos with distinct shots, dialogue cuts, camera movements
Uniform	Evenly spaced timestamps	Static content, slideshows, single-take recordings

Practical Code Examples

Inspecting the Threshold and Engine Selection

from skills.watch.scripts.frames import (
    SCENE_THRESHOLD,
    SCENE_MIN_FRAMES,
    extract_scene_or_uniform,
)
from pathlib import Path

# Check default constants

print(f"Scene threshold: {SCENE_THRESHOLD}")      # 0.20

print(f"Minimum scene frames: {SCENE_MIN_FRAMES}")  # 8

# Extract frames with automatic engine selection

frames, metadata = extract_scene_or_uniform(
    video_path=Path("interview.mp4"),
    out_dir=Path("output/"),
    max_frames=16,
)

print(f"Engine used: {metadata['engine']}")
print(f"Frames extracted: {metadata['count']}")

Handling a Static Video (Forces Uniform Sampling)

# A screen recording with no cuts will trigger uniform sampling

frames, metadata = extract_scene_or_uniform(
    video_path=Path("static_recording.mp4"),
    out_dir=Path("uniform_output/"),
)

assert metadata["engine"] == "uniform"
assert metadata["count"] >= 8  # Guaranteed minimum coverage

Key Files and Functions in the Repository

• skills/watch/scripts/frames.py — Core implementation containing SCENE_THRESHOLD, extract_scene_candidates(), and extract_scene_or_uniform()
• skills/watch/scripts/watch.py — CLI entry point that orchestrates frame extraction based on the --detail flag
• tests/test_frames.py — Test coverage validating scene engine selection and uniform fallback behavior

Summary

• SCENE_THRESHOLD (0.20) controls FFmpeg's scene-change filter sensitivity—higher values detect only major cuts, lower values capture subtle transitions
• Scene detection runs first via extract_scene_candidates(), which builds an FFmpeg filter selecting frames where scene > threshold
• Uniform sampling triggers automatically when fewer than 8 scene changes are detected, ensuring minimum frame coverage for downstream processing
• The engine metadata field ("scene" or "uniform") records which method was used for each extraction

Frequently Asked Questions

How do I make scene detection more sensitive?

Lower the threshold by modifying SCENE_THRESHOLD in skills/watch/scripts/frames.py or passing a custom value to extract_scene_candidates(). Values below 0.15 detect more subtle transitions but may include false positives from minor lighting changes.

Why 8 frames as the minimum for scene detection?

The SCENE_MIN_FRAMES constant (line 19 in frames.py) ensures sufficient visual diversity for typical downstream tasks like video summarization or CLIP-based analysis. This value balances coverage against redundancy.

Can I force uniform sampling even for videos with many scene changes?

Currently extract_scene_or_uniform() does not expose a force-uniform parameter. You can call extract_at_timestamps() directly with custom timestamps, or modify the threshold logic to always fail the scene count check.

Does the threshold value affect processing speed?

No—FFmpeg evaluates the scene metric for every frame regardless of threshold. The threshold only filters which frames are emitted. Processing time depends primarily on video resolution and the _scale_filter() downscaling factor.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}