How Scene Detection Works for Frame Selection in Claude Video

Claude Video detects scene changes using ffmpeg's scene metric filter, deduplicates perceptually similar frames, and falls back to uniform sampling when videos lack enough visual cuts.

The bradautomates/claude-video repository implements an intelligent frame selection pipeline that prioritizes scene boundaries over fixed intervals. By analyzing video content with ffmpeg and applying post-processing filters, the system ensures that only the most visually informative frames are extracted for downstream analysis.

Scene-Change Detection with ffmpeg

The detection logic resides in skills/watch/scripts/frames.py within the extract_scene_candidates function (lines 217-281). Claude Video invokes ffmpeg with the select filter using the expression:

eq(n,0)+gt(scene,<threshold>)

This expression serves two purposes:

• eq(n,0) forces retention of the first frame regardless of content.
• gt(scene,threshold) selects any frame where the scene change metric exceeds the default 0.20 threshold defined by SCENE_THRESHOLD on line 20.

The filter simultaneously scales frames to the requested resolution and injects the showinfo filter so ffmpeg prints presentation timestamps (PTS) to stderr. These timestamps are later captured using the SHOWINFO_TS_RE regular expression defined on line 39.

Timestamp Extraction and Candidate Generation

After ffmpeg execution completes, the pipeline parses the stderr output to extract PTS timestamps for each selected frame (lines 68-70). These values populate the timestamp_seconds field in the returned frame dictionaries.

The raw output from ffmpeg represents candidate frames—potential scene boundaries that have not yet been validated or deduplicated. Each candidate includes its index, extracted timestamp, and file path, forming the input for the post-processing stages.

Post-Processing Pipeline

Minimum Shots Guard

If the initial scan produces fewer than 8 scene cuts (SCENE_MIN_FRAMES), the engine assumes the video is static content (such as a screen recording) and triggers a fallback mechanism. The extract_scene_or_uniform function (starting at line 511) detects this condition and switches from scene-based selection to uniform temporal sampling.

Perceptual Deduplication

Raw scene candidates often contain near-duplicate frames. The dedupe_perceptual function (lines 63-66) removes these by generating 16×16 grayscale thumbnails for each JPEG via _thumb_frames. It then calculates the mean absolute pixel difference between consecutive thumbnails. Frames with a difference ≤ 2.0 (DEDUP_THRESHOLD on line 38) are considered duplicates, removed from the candidate list, and deleted from disk.

Even Sampling to Frame Budget

After deduplication, the remaining frames may still exceed the user-defined budget. The _even_indices helper (lines 83-92) implements _even_sample to select evenly-spaced indices across the candidate list. This method guarantees that the first and last frames are always retained while distributing the remaining selections uniformly throughout the video duration.

Implementation Details and Code Example

The orchestration function extract_scene_or_uniform returns both the selected frames and a metadata dictionary indicating which engine was used (scene vs uniform), the number of candidates found, how many were deduplicated, and whether a fallback occurred.

from pathlib import Path
from skills.watch.scripts.frames import extract_scene_or_uniform

video = Path("example.mp4")
out_dir = Path("frames_out")
out_dir.mkdir(parents=True, exist_ok=True)

# Ask for a maximum of 80 frames; let the function decide fps automatically.

frames, info = extract_scene_or_uniform(
    str(video), out_dir,
    fps=0,                     # 0 → auto-fps is computed internally

    target_frames=80,
    resolution=512,
    max_frames=80,
    start_seconds=None,
    end_seconds=None,
    dedup=True,
)

print("Engine used:", info["engine"])
for f in frames:
    print(f"{f['index']:03d} – {f['timestamp_seconds']}s – {f['reason']}")

Executing this snippet creates JPEGs named frame_0000.jpg, frame_0001.jpg, and so on in frames_out/, while printing whether the scene-detection engine succeeded or fell back to uniform sampling.

Summary

• Scene detection in Claude Video relies on ffmpeg's select filter with a default threshold of 0.20 to identify visual cuts.
• The pipeline extracts timestamps from ffmpeg's stderr using regex pattern SHOWINFO_TS_RE, then validates that at least 8 scene cuts exist before accepting scene-based results.
• Perceptual deduplication uses 16×16 grayscale thumbnails and a mean-absolute-difference threshold of 2.0 to eliminate redundant frames.
• When scene cuts are insufficient, the system automatically falls back to uniform sampling via extract_scene_or_uniform.
• Final frame selection uses even-spaced indexing to respect the max_frames budget while preserving first and last frames.

Frequently Asked Questions

What ffmpeg expression does Claude Video use for scene detection?

Claude Video uses the expression eq(n,0)+gt(scene,<threshold>) within ffmpeg's select filter. The eq(n,0) component ensures the first frame is always selected, while gt(scene,0.20) selects frames where the scene change metric exceeds the default threshold defined by SCENE_THRESHOLD on line 20 of frames.py.

How does Claude Video handle videos without scene changes?

When fewer than 8 scene cuts are detected (SCENE_MIN_FRAMES), the extract_scene_or_uniform function treats the video as static content and automatically falls back to uniform temporal sampling based on frames-per-second calculations rather than scene boundaries.

What is the perceptual deduplication threshold?

The dedupe_perceptual function uses a DEDUP_THRESHOLD of 2.0 (defined on line 38), representing the maximum mean absolute pixel difference allowed between 16×16 grayscale thumbnails before consecutive frames are considered duplicates and removed from the selection.

Which function orchestrates the entire frame extraction pipeline?

The extract_scene_or_uniform function starting at line 511 of skills/watch/scripts/frames.py coordinates the complete workflow, calling extract_scene_candidates for detection, dedupe_perceptual for deduplication, and managing the fallback to uniform extraction when scene-based detection yields insufficient results.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}