How bradautomates/claude-video Performs Frame Deduplication Using Grayscale Thumbnails and Mean Absolute Difference

bradautomates/claude-video detects near-duplicate video frames by converting each extracted JPEG into a 16×16 grayscale thumbnail and computing the mean absolute per-pixel difference between successive thumbnails, removing frames where the difference is less than or equal to 2.0.

The claude-video repository implements a fast perceptual deduplication pipeline that processes video frames through ffmpeg-generated thumbnails and deterministic comparison algorithms. This system reduces redundant visual data before AI analysis by scaling high-resolution frames into compact grayscale representations and applying a greedy mean-absolute-difference strategy to identify true visual duplicates.

Grayscale Thumbnail Generation with ffmpeg

The deduplication process begins in skills/watch/scripts/frames.py with the _thumb_frames() function (lines 24-31), which generates compact representations of each extracted frame. The constant DEDUP_THUMB = 16 defines a fixed 16×16 pixel thumbnail size that balances computational efficiency with structural detail.

The function constructs a single ffmpeg command (lines 48-50) that reads the JPEG sequence, applies the scale=16:16 filter, forces the gray pixel format, and streams raw video bytes back to Python. The raw byte stream is split into a list of bytes objects—one per frame—each containing exactly DEDUP_THUMB × DEDUP_THUMB grayscale values representing the luminance structure of the original image.

Mean Absolute Difference Calculation

Once thumbnails are generated, the _frame_delta() function (lines 15-22 in skills/watch/scripts/frames.py) calculates the similarity between two frame buffers using mean absolute difference (MAD). This metric computes the average per-pixel absolute distance between two thumbnail byte buffers on a scale of 0-255.

def _frame_delta(a: bytes, b: bytes) -> float:
    # mean absolute per‑pixel difference (0‑255)

    if not a or len(a) != len(b):
        return float("inf")
    return sum(abs(x - y) for x, y in zip(a, b)) / len(a)

The function includes a critical safety guard: it returns float("inf") if the input buffers differ in length or are empty. This ensures that corrupted or mismatched frames are never incorrectly collapsed as duplicates, preventing data corruption in the deduplication pipeline.

Greedy Deduplication Algorithm

The dedupe_perceptual() function (lines 63-70) orchestrates the deduplication workflow by first generating thumbnails for all candidate frames and then invoking _dedupe_by_deltas() (lines 78-86) to perform the actual duplicate detection.

The algorithm uses a greedy chronological approach based on the constant DEDUP_THRESHOLD = 2.0. It keeps the first frame as a reference, then compares each subsequent thumbnail against the last kept frame using _frame_delta(). If the mean absolute difference is less than or equal to the threshold, the candidate frame is marked as a duplicate and scheduled for deletion; otherwise, it becomes the new reference frame.

def _dedupe_by_deltas(candidates, thumbs, threshold=DEDUP_THRESHOLD):
    kept = [candidates[0]]
    last = thumbs[0]
    dropped = []
    for cand, thumb in zip(candidates[1:], thumbs[1:]):
        if _frame_delta(thumb, last) <= threshold:
            dropped.append(cand)
        else:
            kept.append(cand)
            last = thumb
    # cleanup: delete dropped JPEGs & re‑index survivors

After processing, the function deletes the dropped JPEGs from disk and re-indexes the survivors to maintain sequential naming.

Integration Across Extraction Engines

The deduplication system integrates with every frame extraction engine in the repository, including extract(), extract_scene_candidates(), and extract_keyframes(). Unless the user passes the --no-dedup CLI flag via skills/watch/scripts/watch.py, the system automatically invokes dedupe_perceptual() on the extracted frame candidates.

The deduplication results are reported through the deduped_count field in the JSON metadata returned by the CLI, providing transparency about how many redundant frames were removed from the analysis set.

Code Examples

To deduplicate frames programmatically after extraction:

from skills.watch.scripts.frames import dedupe_perceptual

candidates = [
    {"path": "frame_0001.jpg", "timestamp_seconds": 0.0, "reason": "uniform"},
    {"path": "frame_0002.jpg", "timestamp_seconds": 0.5, "reason": "uniform"},
    # … additional frames …

]

# Uses default threshold of 2.0

unique_frames, n_dropped = dedupe_perceptual(candidates)

print(f"Kept {len(unique_frames)} frames, removed {n_dropped} duplicates")

From the command line, deduplication runs automatically:

python -m skills.watch.scripts.frames video.mp4 output_dir --max-frames 200

The output metadata includes "deduped_count": <n> indicating the number of duplicates removed.

Summary

• Grayscale thumbnails: The _thumb_frames() function in skills/watch/scripts/frames.py generates 16×16 pixel thumbnails using ffmpeg's scale and gray format filters.
• MAD calculation: _frame_delta() computes mean absolute per-pixel difference between thumbnail buffers, returning infinity for size mismatches to prevent data corruption.
• Greedy deduplication: _dedupe_by_deltas() compares each frame against the last kept frame using a default threshold of 2.0, removing duplicates when the difference falls below this value.
• Transparent reporting: The system reports removed frames via deduped_count in the CLI metadata and can be disabled with the --no-dedup flag.

Frequently Asked Questions

What is the default threshold for frame deduplication in claude-video?

The default threshold is 2.0, defined by the DEDUP_THRESHOLD constant in skills/watch/scripts/frames.py. Frames with a mean absolute difference of 2.0 or less (on a 0-255 per-pixel scale) are considered visual duplicates and removed from the candidate set. This conservative threshold ensures that only nearly identical frames are eliminated while preserving meaningful visual changes.

Why does claude-video use 16×16 grayscale thumbnails instead of full-resolution frames?

The DEDUP_THUMB = 16 constant specifies a 16×16 pixel resolution because processing full-resolution frames would be computationally expensive without improving deduplication accuracy. The grayscale conversion eliminates color channel noise while preserving the structural luminance information necessary to detect true visual duplicates. This approach processes video data through compact byte buffers that require minimal memory compared to full JPEG decoding.

How does the algorithm handle corrupted frames or size mismatches?

The _frame_delta() function includes explicit guards that return float("inf") when input buffers are empty or differ in length. This ensures that corrupted frames, partial reads, or resolution changes are never incorrectly collapsed as duplicates. The greedy deduplication algorithm treats these infinity values as non-matches, forcing the system to keep potentially corrupted frames rather than risk data loss through erroneous deletion.

Can I disable frame deduplication or adjust the threshold?

You can disable deduplication entirely by passing the --no-dedup flag to the CLI in skills/watch/scripts/watch.py. However, the threshold is hardcoded to 2.0 in the current implementation via DEDUP_THRESHOLD. To use a different sensitivity, you would need to modify the constant in skills/watch/scripts/frames.py or call dedupe_perceptual(candidates, threshold=your_value) directly from Python when using the programmatic API.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}