How bradautomates/claude-video Handles Frame Deduplication Using Mean Absolute Difference

bradautomates/claude-video removes visually duplicate frames by computing Mean Absolute Difference (MAD) between 16×16 grayscale thumbnails, keeping only frames that differ by more than 2.0 on a 0–255 scale.

The claude-video repository implements a lightweight, pure-Python frame deduplication system that eliminates redundant video frames without external dependencies. By leveraging ffmpeg for thumbnail generation and Mean Absolute Difference for pixel-level comparison, the tool efficiently collapses static sequences while preserving distinct visual moments.

The Three-Step MAD Deduplication Pipeline

The deduplication logic in skills/watch/scripts/frames.py operates through three tightly coupled stages that process candidate frames after initial extraction.

Step 1: Generating 16×16 Grayscale Thumbnails

Before comparison, each candidate frame is downscaled to a tiny grayscale thumbnail to minimize computational overhead. In frames._thumb_frames (lines 424–460), the system invokes ffmpeg to convert extracted JPEGs into 16×16 pixel grayscale arrays (DEDUP_THUMB = 16). This single-pass approach keeps the implementation within standard library constraints while producing 256-byte representations suitable for rapid comparison.

Step 2: Computing Mean Absolute Difference

The core comparison occurs in frames._frame_delta (lines 315–322). For two thumbnail byte-arrays a and b, the function calculates the Mean Absolute Difference by summing absolute per-pixel differences and dividing by the total pixel count (256). This yields a normalized score in the range 0–255, where identical frames produce 0.0 and maximally different frames approach 255.0.

Step 3: Greedy Duplicate Removal

The deduplication algorithm in frames._dedupe_by_deltas (lines 380–400) implements a greedy selection strategy. Starting with the first candidate frame, each subsequent thumbnail is compared to the last kept frame using _frame_delta. If the MAD is ≤ DEDUP_THRESHOLD (2.0), the frame is considered a duplicate and deleted; otherwise, it becomes the new reference for subsequent comparisons. This approach ensures that only visually distinct moments survive while avoiding cascade effects that might unintentionally drop unique shots.

Implementation Details in frames.py

The deduplication system exposes a clean public API through frames.dedupe_perceptual (lines 360–368). This function orchestrates the thumbnail generation and comparison pipeline, returning both the surviving frame list and the count of dropped duplicates.

Key implementation characteristics include:

• Deterministic thresholding: The hardcoded threshold of 2.0 specifically targets truly identical frames (static slides, frozen screens) while preserving genuine scene cuts
• Memory efficiency: Processing occurs on byte arrays rather than full-resolution images
• Pure stdlib operation: No computer vision libraries required; ffmpeg handles all image decoding

Higher-level extraction engines—including extract_scene_or_uniform, extract_keyframes, and extract—automatically invoke dedupe_perceptual when the dedup=True parameter is passed.

Integration with Frame Extraction Engines

Frame deduplication operates as an optional post-processing step within the video analysis pipeline. After any extraction engine writes JPEG candidates to disk, the system checks the dedup parameter:

1. Extraction: Engines like extract_scene_candidates generate initial frame sets based on scene detection or uniform sampling
2. Deduplication: If enabled, dedupe_perceptual processes the candidate list through the MAD pipeline
3. Capping: The surviving frames may then be evenly sampled via _even_sample to respect max_frames limits

This workflow ensures that deduplication occurs before frame count constraints are applied, maximizing the diversity of the final selected frames.

Code Examples

Manual Deduplication of Frame Lists

from pathlib import Path
import frames

# Assume candidates is a list of frame dicts from an extractor

candidates = [
    {"index": 0, "timestamp_seconds": 0.0, "path": "frame_0000.jpg", "reason": "scene-change"},
    {"index": 1, "timestamp_seconds": 0.1, "path": "frame_0001.jpg", "reason": "scene-change"},
]

# Run MAD-based deduplication with default threshold (2.0)

survivors, dropped = frames.dedupe_perceptual(candidates)

print(f"Kept {len(survivors)} frames, dropped {dropped} duplicates.")

Enabling Deduplication in High-Level Extraction

from pathlib import Path
import frames

video = "example.mp4"
out_dir = Path("frames_out")

# Extract using scene detection with automatic deduplication

frames_out, meta = frames.extract_scene_or_uniform(
    video_path=video,
    out_dir=out_dir,
    fps=2.0,
    target_frames=50,
    max_frames=100,
    dedup=True,  # Triggers MAD comparison

)

print("Engine:", meta["engine"])
print("Deduped frames:", meta["deduped_count"])

Disabling Deduplication

frames_out, meta = frames.extract_scene_or_uniform(
    video_path=video,
    out_dir=out_dir,
    fps=2.0,
    target_frames=50,
    max_frames=100,
    dedup=False,  # All frames retained regardless of similarity

)

Testing and Validation

The repository includes comprehensive test coverage in tests/test_dedup.py that verifies the MAD logic across edge cases:

• test_frame_delta_identical_is_zero: Confirms that identical thumbnails produce a delta of 0.0
• test_dedupe_collapses_identical_run: Validates that sequences of duplicate frames collapse to a single survivor
• test_dedupe_keeps_all_distinct: Ensures frames with large MAD values are fully retained
• test_dedupe_threshold_is_inclusive: Demonstrates that deltas exactly equal to 2.0 count as duplicates

These tests prove that the Mean Absolute Difference implementation correctly identifies visual redundancy without false positives.

Summary

• bradautomates/claude-video implements frame deduplication using Mean Absolute Difference computed on 16×16 grayscale thumbnails
• The greedy algorithm compares each frame against the last kept frame, removing duplicates with MAD ≤ 2.0
• Core functions reside in skills/watch/scripts/frames.py: _thumb_frames for generation, _frame_delta for comparison, and _dedupe_by_deltas for selection
• The system operates without external dependencies beyond ffmpeg, using pure-Python byte array operations
• Deduplication integrates seamlessly with all extraction engines via the dedupe_perceptual public API

Frequently Asked Questions

What threshold does claude-video use for frame deduplication?

The system uses a MAD threshold of 2.0 on a 0–255 scale, defined as DEDUP_THRESHOLD in skills/watch/scripts/frames.py. This conservative value ensures only visually identical frames (such as static presentation slides or frozen video segments) are removed, while preserving subtle motion and scene transitions that exceed the threshold.

Why does claude-video use 16×16 thumbnails for MAD calculation?

The 16×16 pixel resolution (DEDUP_THUMB = 16) provides sufficient granularity to detect meaningful visual changes while minimizing computational overhead. At 256 bytes per thumbnail, the Mean Absolute Difference calculation requires only 256 subtraction and absolute value operations per comparison, enabling rapid processing of long video sequences without GPU acceleration or heavy dependencies.

How does the greedy deduplication algorithm work?

The greedy approach implemented in _dedupe_by_deltas maintains a reference to the last kept frame and compares each subsequent candidate against this reference. If the MAD exceeds the threshold, the candidate becomes the new reference; otherwise, it is discarded. This ensures temporal coherence—if frame A differs from frame B, and B differs from C, all three are retained even if A and C might appear similar, preventing accidental loss of transitional content.

Can I adjust the deduplication sensitivity in claude-video?

While the DEDUP_THRESHOLD constant is hardcoded to 2.0 in the source, advanced users can modify this value directly in skills/watch/scripts/frames.py (line 315) to adjust sensitivity. Lower values (such as 1.0) retain more frames by requiring stricter similarity, while higher values (such as 5.0) aggressively remove frames with minor variations. The repository test suite in tests/test_dedup.py provides validation patterns for custom threshold calibration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}