How Frame Delta Deduplication Handles Slow Fades vs. Sudden Cuts in Video Analysis

Frame delta deduplication collapses slow fades into single representative frames while preserving sudden cuts by comparing mean pixel differences against a threshold of 2.0 in 16×16 grayscale thumbnails.

The bradautomates/claude-video repository implements an intelligent frame extraction system that uses frame delta deduplication to minimize redundant visual data. This perceptual deduplication technique, found in the watch skill, analyzes successive video frames to determine whether gradual transitions or abrupt changes should be kept for downstream AI processing.

The Perceptual Deduplication Pipeline

The deduplication logic resides in skills/watch/scripts/frames.py and operates on candidate frames extracted via scene-change detection, key-frame extraction, or uniform sampling. The system processes each frame through three distinct stages to identify visual redundancy.

Thumbnail Generation

Before comparison, each candidate JPEG is downscaled to a tiny grayscale thumbnail of size DEDUP_THUMB × DEDUP_THUMB (16 × 16 pixels). This aggressive reduction minimizes computational overhead while preserving enough luminance information to detect significant visual changes. The thumbnail creation occurs in lines 31–38 of the frames module.

Mean-Pixel Difference Calculation

The helper function _frame_delta computes the average absolute difference per pixel between two thumbnails. Implemented in lines 15–22 of skills/watch/scripts/frames.py, this function converts both thumbnails to grayscale arrays and calculates the mean absolute deviation across all pixel values. This yields a single scalar representing the visual distance between consecutive frames.

Threshold-Based Filtering

Frames whose mean difference is ≤ DEDUP_THRESHOLD (default 2.0) are considered "near-identical" and the later frame is discarded. This threshold operates on the 0–255 grayscale range, meaning two frames must differ by less than approximately 0.8% average intensity to trigger deduplication. The filtering logic appears in lines 32–38 and lines 80–99 of the source file.

How Slow Fades Are Processed

During a slow fade, visual content changes gradually over multiple seconds. Consecutive thumbnails differ only by a few grayscale levels, typically staying below the 2.0 threshold. The greedy loop in _dedupe_by_deltas therefore treats the entire fade as a single shot, keeping only the first frame of the fade and discarding the intermediate ones.

This behavior prevents a flood of nearly identical frames from inflating token usage during video analysis. For example, a 5-second crossfade between scenes generates approximately 10–15 candidate frames at 2 fps, but the deduplication algorithm collapses these into a single representative frame, reducing redundancy by over 90% while maintaining visual context.

How Sudden Cuts Are Processed

A hard cut produces a large visual jump between consecutive frames. The thumbnail delta between the frame before the cut and the frame after the cut exceeds DEDUP_THRESHOLD, so the later frame is not dropped. The cut is retained as a distinct shot, preserving the abrupt change that is crucial for downstream scene analysis and content understanding.

This threshold-based distinction ensures that the algorithm captures narrative breakpoints while smoothing over gradual transitions. The system specifically checks these deltas in the while loop spanning lines 80–99 of _dedupe_by_deltas, comparing each frame against the last kept frame rather than its immediate predecessor.

Practical Implementation

You can observe this behavior directly using the watch skill's Python API:

from pathlib import Path
from skills.watch.scripts.frames import extract_scene_or_uniform, dedupe_perceptual

video = "example.mp4"
out_dir = Path("./tmp")
out_dir.mkdir(exist_ok=True)

# 1. Extract candidate frames (scene detection + uniform fallback)

candidates, meta = extract_scene_or_uniform(
    video_path=video,
    out_dir=out_dir,
    fps=2.0,
    target_frames=30,
    resolution=512,
    max_frames=30,
    dedup=True,          # <-- enable perceptual deduplication

)

print("Engine used:", meta["engine"])
print("Frames before dedup:", len(candidates))

# 2. Run the deduplication step directly (optional)

deduped, dropped = dedupe_perceptual(candidates)
print("Dropped near-identical frames:", dropped)

Running this script on a clip containing a 5-second slow fade outputs a single frame for that fade, whereas a clip with a hard cut retains both sides of the transition. The dedupe_perceptual function wraps the internal _dedupe_by_deltas logic, returning both the filtered frame list and a count of dropped duplicates.

Summary

• Frame delta deduplication in bradautomates/claude-video uses 16×16 grayscale thumbnails to compare visual similarity between candidate frames.
• The _frame_delta function calculates mean absolute pixel differences, with a default threshold of 2.0 determining deduplication boundaries.
• Slow fades remain below the threshold and are collapsed into single representative frames to prevent token bloat.
• Sudden cuts exceed the threshold and are preserved as distinct shots for accurate scene segmentation.
• The implementation resides in skills/watch/scripts/frames.py, specifically within the _dedupe_by_deltas function (lines 80–99).

Frequently Asked Questions

What is the default deduplication threshold and can it be modified?

The default DEDUP_THRESHOLD is 2.0 on a 0–255 grayscale scale. While this constant is hardcoded in skills/watch/scripts/frames.py at the module level, you can modify the source directly or patch the value during import for specialized use cases requiring higher or lower sensitivity to visual changes.

How does the algorithm handle partially transparent overlays or watermarks?

Partially transparent overlays that change gradually between frames (such as fading watermarks) typically produce pixel differences below the 2.0 threshold. Consequently, frame delta deduplication treats these similarly to slow fades, keeping only the first frame where the overlay appears significantly different from the previous content.

Why does the system use 16×16 thumbnails instead of full-resolution images?

The DEDUP_THUMB size of 16×16 pixels represents a trade-off between computational efficiency and perceptual accuracy. This resolution captures sufficient luminance information to detect significant scene changes while reducing memory footprint and processing time by approximately 99.9% compared to comparing full 512×512 frames.

Which function actually performs the greedy deduplication loop?

The _dedupe_by_deltas function implements the greedy comparison loop, processing frames sequentially and maintaining a reference to the last kept frame. As implemented in lines 80–99 of skills/watch/scripts/frames.py, it compares each new frame against the last kept frame using _frame_delta, discarding those that fall below the threshold.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}