Claude Video Scene-Change Detection Algorithm: Technical Deep Dive

Claude Video detects scene changes by using ffmpeg's built-in scene filter with a configurable 0.20 threshold, followed by perceptual deduplication of 16×16 grayscale thumbnails and even sampling to respect token budgets.

The scene-change detection algorithm powers the /watch slash command in the bradautomates/claude-video repository, enabling efficient video analysis by extracting only visually significant frames. Implemented primarily in skills/watch/scripts/frames.py, this system balances computational cost against coverage through three distinct detail modes.

Core Algorithm Components

The algorithm relies on ffmpeg's native scene detection capabilities combined with post-processing filters to eliminate redundancy and enforce frame caps.

Scene Threshold and Constants

At the heart of the implementation are several hardcoded constants defined in skills/watch/scripts/frames.py:

• SCENE_THRESHOLD = 0.20 – The minimum scene-change score required for ffmpeg to flag a frame as a cut point
• SCENE_MIN_FRAMES = 8 – The fallback threshold; if fewer than 8 scene changes are detected, the system switches to uniform sampling
• MAX_READ_DIMENSION = 1998 – Maximum JPEG height constraint for processing

These values determine the sensitivity of detection and the minimum viable granularity for scene-based extraction.

Frame Extraction Pipeline

The primary entry point extract_scene_candidates invokes ffmpeg with a complex filtergraph:

ffmpeg -i input.mp4 -vf "select='eq(n\,0)+gt(scene,0.20)',scale=...,showinfo" ...

This filter expression performs two critical functions:

1. eq(n\,0) – Always selects the first frame to ensure coverage from the start
2. gt(scene,0.20) – Selects any frame where the scene-change score exceeds the threshold

The showinfo filter emits timestamp data, which the algorithm parses using the regex pattern SHOWINFO_TS_RE to map frame indices to precise temporal positions.

Perceptual Deduplication

After candidate extraction, the dedupe_perceptual function eliminates near-identical frames that may result from minor encoding variations or static scenes with subtle noise.

The deduplication process:

1. Resizes each JPEG to a 16×16 grayscale thumbnail via _thumb_frames
2. Computes the mean absolute difference between consecutive thumbnails
3. Discards frames where the difference is ≤ 2.0 (DEDUP_THRESHOLD)

This perceptual hashing approach ensures that only visually distinct shots are retained, preventing token waste on redundant imagery.

Even Sampling Strategy

When the candidate count exceeds the requested budget, the _even_indices utility (used by _even_sample) distributes selections evenly across the timeline:

• Always preserves the first and last frames to maintain temporal boundaries
• Calculates step intervals to spread remaining selections uniformly
• Respects the max_frames cap specified by the detail mode

This guarantees comprehensive coverage regardless of video length or scene density.

Implementation in frames.py

The orchestration logic resides in extract_scene_or_uniform, which implements the decision tree for extraction strategies:

# From skills/watch/scripts/frames.py

def extract_scene_or_uniform(
    video_path,
    out_dir,
    fps=None,
    target_frames=None,
    resolution=512,
    max_frames=None,
    start_seconds=None,
    end_seconds=None,
    dedup=True,
):
    # 1. Attempt scene-based extraction (uncapped initially)

    candidates = extract_scene_candidates(
        video_path, out_dir, resolution, start_seconds, end_seconds
    )
    
    # 2. Fallback check

    if len(candidates) < SCENE_MIN_FRAMES:
        # Switch to uniform sampling for static content

        return extract(video_path, out_dir, fps, target_frames, resolution)
    
    # 3. Optional deduplication

    if dedup:
        candidates = dedupe_perceptual(candidates)
    
    # 4. Apply frame cap via even sampling

    if max_frames and len(candidates) > max_frames:
        indices = _even_indices(len(candidates), max_frames)
        candidates = [candidates[i] for i in indices]
    
    return candidates

This function returns a list of dictionaries containing index, timestamp_seconds, path, and reason for each selected frame, along with metadata about the engine used and deduplication statistics.

Detail Modes and Configuration

The repository exposes three extraction strategies via the /watch command, configured through the detail parameter:

Mode	Frame Source	Cap	Engine	Use Case
efficient	ffmpeg key-frames	50	keyframe	Quick previews, low token usage
balanced (default)	Scene-change frames	100	scene	General analysis with fallback
token-burner	Scene-change frames	Uncapped	scene	Maximum detail, highest coverage

The balanced mode implements intelligent fallback logic: if extract_scene_candidates returns fewer than 8 frames (indicating a static video like a screen recording or talking-head), the system automatically switches to uniform sampling to ensure adequate representation.

Usage Examples

Python Integration

Clients invoke the scene detection through the high-level API in skills/watch/scripts/watch.py:

from skills.watch.scripts.frames import extract_scene_or_uniform, auto_fps

# Calculate target FPS based on duration and cap

fps, target = auto_fps(duration_seconds, max_frames=100)

frames, frame_meta = extract_scene_or_uniform(
    video_path="lecture.mp4",
    out_dir="./frames",
    fps=fps,
    target_frames=target,
    resolution=512,
    max_frames=100,  # Set to None for token-burner mode

    dedup=True,
)

print(f"Extracted {len(frames)} frames using {frame_meta['engine']}")

Command-Line Execution

For debugging or standalone usage, run the module directly:

python -m skills.watch/scripts/frames.py \
    input.mp4 \
    ./output \
    --fps 2.0 \
    --max-frames 100 \
    --resolution 512

Add --no-dedup to disable perceptual deduplication and retain all scene-change candidates.

Summary

• Claude Video scene-change detection leverages ffmpeg's native scene filter with a default threshold of 0.20 to identify significant visual transitions
• The algorithm in skills/watch/scripts/frames.py implements a three-stage pipeline: candidate extraction, 16×16 grayscale perceptual deduplication, and even sampling to enforce frame caps
• Automatic fallback to uniform sampling occurs when fewer than 8 scene changes are detected, preventing poor coverage on static content
• Three detail modes (efficient, balanced, token-burner) allow users to trade token consumption against analysis granularity
• All frames retain precise timestamp metadata extracted from ffmpeg's showinfo output, enabling accurate temporal referencing

Frequently Asked Questions

How does the scene-change detection handle videos with minimal motion?

If the algorithm detects fewer than SCENE_MIN_FRAMES (8) scene changes, it deems the video "effectively static" and automatically falls back to uniform sampling via the extract function. This prevents situations where screen recordings or single-shot interviews would yield only 1-2 frames while respecting the token budget through evenly spaced temporal sampling.

What is the perceptual deduplication threshold and how does it work?

The dedupe_perceptual function resizes each candidate frame to a 16×16 grayscale thumbnail and calculates the mean absolute difference between consecutive frames. If the difference is ≤ 2.0 (DEDUP_THRESHOLD), the frames are considered visually identical and the latter is discarded. This removes redundant frames caused by encoding artifacts or subtle lighting changes without removing genuinely distinct content.

Can I adjust the scene-change sensitivity threshold?

The SCENE_THRESHOLD constant is hardcoded to 0.20 in skills/watch/scripts/frames.py. While the current implementation does not expose this as a command-line parameter, you can modify the constant directly in the source or patch the extract_scene_candidates function to accept a custom threshold value for the gt(scene,THRESH) ffmpeg filter expression.

How does the token-burner mode differ from balanced mode?

Both modes use the scene-change detection engine, but balanced mode applies a hard cap of 100 frames (detail_cap=100) and enables automatic fallback to uniform sampling. Token-burner mode sets max_frames=None, allowing unlimited scene-change candidates (subject only to deduplication), making it suitable for detailed analysis of complex, fast-cutting content where maximum visual coverage justifies higher token consumption.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}