How bradautomates/claude-video Uses Mean Absolute Difference for Frame Deduplication

bradautomates/claude-video removes visually duplicate frames by computing Mean Absolute Difference (MAD) on 16×16 grayscale thumbnails, discarding frames with a MAD score of 2.0 or less compared to the last kept frame.

The claude-video repository implements a lightweight, pure-Python frame deduplication pipeline that leverages Mean Absolute Difference (MAD) to eliminate redundant video frames without external machine learning dependencies. By downsampling extracted frames to tiny thumbnails and applying a deterministic greedy comparison algorithm, the tool efficiently preserves distinct visual moments while collapsing static sequences and frozen screens.

How MAD Frame Deduplication Works

The deduplication process operates in three tightly coupled stages defined in skills/watch/scripts/frames.py. Each stage minimizes computational overhead while maintaining accuracy through pixel-level comparison.

Step 1: Generating 16×16 Grayscale Thumbnails

After candidate frames are extracted as JPEGs, the _thumb_frames function (lines 424–460) downscales each image to a 16×16 grayscale thumbnail using a single ffmpeg pass. The constant DEDUP_THUMB = 16 defines this dimension, producing a 256-byte array per frame that represents the luminance profile of the original image. This approach keeps the implementation within the Python standard library while ensuring consistent input for the MAD calculation.

Step 2: Computing the Mean Absolute Difference

The _frame_delta function (lines 315–322) calculates the MAD between two thumbnail byte-arrays a and b. It sums the absolute per-pixel differences and divides by the total pixel count (256), yielding a normalized score in the range 0–255. A score of 0.0 indicates identical thumbnails, while higher values represent increasing visual divergence.

# Conceptual implementation based on _frame_delta

def _frame_delta(a: bytes, b: bytes) -> float:
    """Calculate Mean Absolute Difference between two 16×16 thumbnails."""
    total = sum(abs(x - y) for x, y in zip(a, b))
    return total / 256.0  # DEDUP_THUMB * DEDUP_THUMB

Step 3: Greedy Duplicate Removal

The _dedupe_by_deltas function (lines 380–400) implements a greedy filtering algorithm. Starting with the first candidate frame, it compares each subsequent frame's thumbnail to the last kept thumbnail using _frame_delta. If the MAD is ≤ DEDUP_THRESHOLD (2.0), the frame is classified as a duplicate and deleted; otherwise, it becomes the new "last kept" reference. This strategy guarantees that only distinct visual moments survive, preventing cascade effects that might unintentionally drop unique shots separated by minor variations.

Integration with Extraction Engines

The public API dedupe_perceptual (lines 360–368) orchestrates the entire pipeline by calling _thumb_frames followed by _dedupe_by_deltas. Higher-level extraction engines—including extract_scene_or_uniform, extract_keyframes, and extract—invoke this function when the dedup=True parameter is passed (enabled by default).

from pathlib import Path
import frames

video = Path("lecture.mp4")
out_dir = Path("output")

# Extract with automatic MAD-based deduplication

frame_list, meta = frames.extract_scene_or_uniform(
    video_path=video,
    out_dir=out_dir,
    fps=1.0,
    target_frames=30,
    dedup=True  # Triggers MAD comparison

)

print(f"Kept {len(frame_list)} frames, dropped {meta['deduped_count']} duplicates")

Configuring Deduplication Parameters

The deduplication sensitivity is controlled by two constants in skills/watch/scripts/frames.py:

• DEDUP_THUMB = 16: Sets the thumbnail resolution. Lower values increase speed but reduce accuracy; 16×16 provides sufficient granularity to detect scene changes while maintaining minimal memory overhead.
• DEDUP_THRESHOLD = 2.0: Defines the maximum MAD score for duplicate classification. This inclusive threshold (frames with delta exactly 2.0 are dropped) effectively collapses truly identical frames—such as static slides or frozen screens—while preserving genuine scene transitions that typically exhibit much higher deltas.

To disable deduplication entirely, pass dedup=False to any extraction engine, bypassing the MAD computation and retaining all candidate frames.

Testing and Validation

The repository validates the MAD logic through comprehensive unit tests in tests/test_dedup.py. Key test cases include:

• test_frame_delta_identical_is_zero: Confirms that identical thumbnails produce a delta of exactly 0.0.
• test_dedupe_collapses_identical_run: Verifies that sequences of duplicate frames collapse to a single survivor.
• test_dedupe_keeps_all_distinct: Ensures frames with large MAD values above the threshold are all retained.
• test_dedupe_threshold_is_inclusive: Demonstrates that a delta exactly equal to 2.0 still triggers duplicate removal.

These tests confirm that the greedy algorithm behaves deterministically across edge cases and threshold boundaries.

Summary

• Mean Absolute Difference drives frame deduplication in claude-video through efficient per-pixel comparison of 16×16 thumbnails.
• The _frame_delta function in skills/watch/scripts/frames.py computes MAD scores ranging from 0–255, with identical frames scoring 0.0.
• A greedy algorithm (_dedupe_by_deltas) compares each frame only to the last kept frame, removing duplicates with MAD ≤ 2.0.
• The dedupe_perceptual API integrates seamlessly with all extraction engines, requiring zero external ML dependencies.
• Comprehensive tests in tests/test_dedup.py verify deterministic behavior at threshold boundaries and across identical frame sequences.

Frequently Asked Questions

What is Mean Absolute Difference in video processing?

Mean Absolute Difference (MAD) is a statistical measure that calculates the average absolute pixel difference between two images. In claude-video, MAD quantifies visual similarity between 16×16 grayscale thumbnails, providing a computationally inexpensive metric to identify near-duplicate frames without requiring complex perceptual hashing or feature detection algorithms.

Why does claude-video use 16×16 thumbnails for MAD calculation?

The 16×16 resolution (DEDUP_THUMB = 16) strikes a balance between detection accuracy and processing speed. This size contains enough pixel data to distinguish scene changes while generating only 256 bytes per frame, enabling rapid in-memory comparison using pure Python standard library functions without external image processing dependencies.

How does the greedy deduplication algorithm work?

The greedy algorithm maintains a reference to the last kept frame and compares each new candidate only to that reference. If the MAD score falls at or below the 2.0 threshold, the candidate is deleted; otherwise, it becomes the new reference. This approach prevents "drift" that could occur when comparing against an average or initial frame, ensuring that visually distinct shots are preserved even if they appear in rapid succession.

Can I adjust the deduplication sensitivity?

Yes. You can modify the DEDUP_THRESHOLD constant in skills/watch/scripts/frames.py (currently set to 2.0) to make the filter more or less aggressive. Lower values (e.g., 1.0) retain more frames by requiring near-identical matches, while higher values (e.g., 5.0) aggressively remove frames with minor variations. Note that the threshold is inclusive—frames matching the threshold exactly are considered duplicates and removed.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}