Frame Deduplication in the /watch Skill: How bradautomates/claude-video Eliminates Redundant Frames

Frame deduplication in the /watch skill works by converting extracted frames to 16×16 grayscale thumbnails, computing the mean absolute pixel difference between consecutive frames, and greedily removing frames that fall below a configurable threshold (default 2.0) to eliminate static or near-identical content before analysis.

The bradautomates/claude-video repository provides a /watch skill that extracts frames from video files for AI analysis. To reduce redundancy and token consumption when processing long videos or static content like screen recordings, the skill implements an efficient frame deduplication algorithm in skills/watch/scripts/frames.py that collapses visually similar frames while preserving scene changes.

The Five-Step Deduplication Pipeline

The deduplication logic follows a perceptual hashing approach optimized for speed and memory efficiency. According to the source code in skills/watch/scripts/frames.py, the pipeline processes extracted JPEGs before they are passed to downstream analysis.

1. Downscaling to 16×16 Grayscale Thumbnails

Each extracted frame is converted to a tiny perceptual hash using FFmpeg. The _thumb_frames function (lines 24-33) decodes each JPEG into a DEDUP_THUMB × DEDUP_THUMB (16×16) grayscale thumbnail.

This design keeps the Python implementation pure standard-library because FFmpeg handles the heavy lifting of image decoding and resizing. The function returns raw grayscale bytes to Python for comparison, minimizing memory overhead even for long videos.

2. Computing Mean Absolute Difference

The _frame_delta helper (lines 15-22) calculates the mean absolute difference between two thumbnail byte-arrays. If the input arrays differ in length—indicating corrupted or mismatched frames—the function returns float("inf") to ensure the fail-open behavior never collapses unlike frames.

This per-pixel comparison works effectively for static content (slides, screen recordings) while preserving distinct shots that differ in luminance.

3. Greedy Chronological Deduplication

The dedupe_perceptual function (lines 64-71) orchestrates the comparison logic by calling _dedupe_by_deltas (lines 78-87). The algorithm walks through chronologically-ordered candidates and keeps the first frame, then compares each subsequent thumbnail to the last kept thumbnail.

If the mean difference is ≤ DEDUP_THRESHOLD (default 2.0), the frame is considered a duplicate and is immediately deleted from disk using Path(...).unlink(). Otherwise, it becomes the new "last kept" frame. This greedy "last-kept" comparison avoids cascading deletions; a frame is only dropped if it matches the most recent survivor, not merely the previous raw frame.

4. Cleanup and Re-indexing

Within _dedupe_by_deltas (lines 99-107), all dropped JPEGs are permanently removed from the filesystem. The surviving frames are then re-indexed so that the index field runs contiguously from 0 to N-1. The function returns a tuple containing the list of surviving frame dictionaries and the count of dropped frames.

5. Integration with Extraction Pipelines

The deduplication step runs automatically unless disabled. In skills/watch/scripts/watch.py (lines 64-73), the command-line flag --no-dedup flips the dedup boolean to False. Throughout the engine pipelines—including extract_scene_or_uniform, extract_keyframes, and extract—the code invokes dedupe_perceptual before any cap or sampling step, recording the drop count in the meta-object as deduped_count.

Configuring Frame Deduplication Behavior

You can control deduplication through both CLI flags and Python API parameters.

Disable deduplication entirely by passing --no-dedup when running the skill:

python -m skills.watch.scripts.watch video.mp4 --no-dedup

Adjust the sensitivity by modifying the DEDUP_THRESHOLD constant in skills/watch/scripts/frames.py (default 2.0). Lower values keep more frames; higher values are more aggressive at removing near-duplicates. The threshold is inclusive—deltas exactly equal to the threshold count as duplicates.

Implementation Code Examples

Running Deduplication from the Command Line

Extract frames with default deduplication enabled:

python -m skills.watch.scripts.frames \
    video.mp4 frames_out \
    --fps 4.0 --max-frames 100

The script runs dedupe_perceptual automatically and prints a JSON summary including "deduped_count".

Disabling Deduplication in CLI

To keep every extracted frame without collapsing duplicates:

python -m skills.watch.scripts.frames \
    video.mp4 frames_out \
    --fps 4.0 --max-frames 100 --no-dedup

The "deduped_count" field will report 0 in the output metadata.

Using the Python API Directly

Import the deduplication functions for custom workflows:

from skills.watch.scripts import frames

# candidates is a list of frame dicts from extract()

candidates = [
    {"index": 0, "timestamp_seconds": 0.0, "path": "frame_0000.jpg", "reason": "uniform"},
    {"index": 1, "timestamp_seconds": 0.5, "path": "frame_0001.jpg", "reason": "uniform"},
]

# Collapse near-duplicates with default threshold (2.0)

survivors, dropped = frames.dedupe_perceptual(candidates)

print(f"Kept {len(survivors)} frames, dropped {dropped} duplicates")

Custom Threshold (Advanced)

For stricter deduplication that keeps only highly similar frames:

from pathlib import Path

# Generate thumbnails manually for custom threshold

thumbs = frames._thumb_frames([Path(f["path"]) for f in candidates])

# Use threshold of 1.0 instead of default 2.0

survivors, dropped = frames._dedupe_by_deltas(
    candidates, 
    thumbs=thumbs, 
    threshold=1.0
)

Summary

• Frame deduplication in the /watch skill uses 16×16 grayscale thumbnails generated by FFmpeg to create lightweight perceptual hashes of each frame.
• The _frame_delta function computes mean absolute pixel differences, with the greedy _dedupe_by_deltas algorithm (lines 78-87) keeping the first frame and comparing subsequent frames only against the last survivor.
• Duplicates are deleted from disk immediately when their difference falls at or below the DEDUP_THRESHOLD (default 2.0), and remaining frames are re-indexed contiguously.
• Control the feature via the --no-dedup CLI flag in watch.py (line 64) or by calling dedupe_perceptual() directly with dedup=False.
• The threshold is inclusive (<=), and mismatched frame sizes return float("inf") to prevent accidental deletion of corrupted or heterogeneous inputs.

Frequently Asked Questions

What threshold should I use for frame deduplication?

The default threshold of 2.0 works well for most screen recordings and presentation videos where slides remain static for multiple seconds. For high-motion content like sports or action scenes, consider raising the threshold to 3.0-4.0 to prevent over-aggressive removal, or disable deduplication entirely with --no-dedup. For extremely static content like terminal recordings, lowering to 1.0 may remove more redundant frames without losing information.

Does deduplication run before or after frame sampling?

Deduplication runs before the cap or sampling steps. According to the integration in extract_scene_or_uniform and related functions, dedupe_perceptual processes the full set of extracted candidates first, then the remaining frames undergo any maximum frame limits or sampling strategies. This ensures that duplicate frames do not consume slots in your --max-frames quota.

Why does the algorithm use 16×16 thumbnails instead of full resolution?

The 16×16 grayscale thumbnails strike a balance between perceptual accuracy and computational efficiency. Full-resolution comparisons would be prohibitively slow for long videos and sensitive to compression artifacts, while smaller thumbnails might miss subtle but meaningful visual changes. The 256-byte representation (16×16) enables rapid byte-array comparisons in pure Python without requiring heavy image processing libraries.

How does the greedy algorithm handle scene transitions?

The greedy "last-kept" approach handles scene transitions naturally by only comparing each candidate to the most recent surviving frame, not merely the previous frame in the extraction sequence. When a scene change occurs, the new frame will differ significantly from the last kept frame (exceeding the threshold), causing it to be preserved and becoming the new reference point for subsequent comparisons. This prevents the algorithm from drifting or collapsing distinct scenes that happen to be adjacent in the timeline.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}