How Perceptual Frame Deduplication Works in Claude-Video: A Technical Deep Dive

Claude-Video removes near-duplicate frames by comparing 16×16 grayscale thumbnails using mean absolute pixel difference with a greedy filtering algorithm.

The bradautomates/claude-video repository implements a lightweight, perceptual deduplication system that collapses static or near-static video segments before sending frames to downstream LLMs. This reduces token costs without sacrificing meaningful visual changes. The algorithm runs entirely on compressed JPEG thumbnails, making it fast enough to process in a single FFmpeg pass.

Generating 16×16 Grayscale Thumbnails

Every deduplication operation begins with DEDUP_THUMB = 16 — a fixed thumbnail size that balances detail with computational efficiency.

In skills/watch/scripts/frames.py (lines 24-31), the _thumb_frames function generates these thumbnails through a single FFmpeg invocation:

def _thumb_frames(frame_paths: list[Path]) -> list[bytes]:
    """Return list of 16×16 grayscale raw bytes, one per frame."""
    cmd = [
        "ffmpeg",
        "-hide_banner", "-loglevel", "error",
        "-i", "-",
        "-vf", f"scale={DEDUP_THUMB}:{DEDUP_THUMB}:flags=lanczos,format=gray",
        "-f", "image2pipe", "-c:v", "rawvideo", "-pix_fmt", "gray", "-"
    ]
    # Pipelined JPEG → raw byte buffers

Key design choices:

• Lanczos scaling preserves edge sharpness during aggressive downscaling
• Grayscale conversion eliminates color information, focusing purely on luminance changes
• Raw pipe output avoids disk I/O — thumbnails stay in memory as byte buffers

This pipeline processes frames in batch, returning one bytes object per frame where each buffer contains exactly 256 bytes (16 × 16 pixels).

Computing Frame Similarity with Mean Absolute Difference

The core similarity metric lives in _frame_delta (lines 15-22 of skills/watch/scripts/frames.py):

def _frame_delta(a: bytes, b: bytes) -> float:
    """Mean absolute per-pixel difference between two thumbnail buffers."""
    if len(a) != len(b):
        return float('inf')  # Mismatched frames never collapse

    diff_sum = sum(abs(a[i] - b[i]) for i in range(len(a)))
    return diff_sum / len(a)

Critical safeguards:

• Length mismatch protection returns infinite distance, preventing accidental deduplication across different video resolutions or corruption
• Mean absolute error (MAE) is preferred over MSE or SSIM for speed — with only 256 pixels, structural analysis offers no practical advantage
• Float division yields a normalized 0-255 scale where identical frames score 0.0

Greedy Deduplication Algorithm

The dedupe_perceptual function orchestrates the filtering process (lines 63-71), delegating to _dedupe_by_deltas (lines 78-89) for the actual selection logic:

def dedupe_perceptual(frames: list[dict], threshold: float = DEDUP_THRESHOLD) -> tuple:
    """Greedy perceptual deduplication with configurable threshold."""
    if len(frames) <= 1:
        return frames, 0
    
    thumbs = _thumb_frames([f["path"] for f in frames])
    keep_mask = _dedupe_by_deltas(thumbs, threshold)
    
    # Apply mask, delete dropped files, re-index survivors

    ...

The greedy selection algorithm works as follows:

1. Initialize with first frame always kept
2. Iterate through remaining frames
3. Compare current frame's thumbnail delta to last kept frame (not previous frame)
4. Drop if delta ≤ DEDUP_THRESHOLD (default 2.0), keep otherwise
5. Delete dropped JPEG files and re-index metadata

This "last kept" reference point — rather than comparing to the immediate predecessor — prevents chain-collapse artifacts where small cumulative differences would otherwise accumulate.

Default Threshold and Tuning

The DEDUP_THRESHOLD = 2.0 default reflects empirical calibration:

Threshold	Behavior	Use Case
0.0	Only exact duplicates removed	Lossless archival
2.0	Default — near-static content collapsed	General LLM processing
5.0+	Aggressive compression, may miss subtle changes	High-motion content only

Thresholds are specified in mean pixel difference (0-255 scale), so 2.0 represents less than 1% average luminance change across the thumbnail.

Integration Across Extraction Engines

Perceptual deduplication applies universally unless disabled with --no-dedup. The three frame extraction engines all invoke dedupe_perceptual:

• extract_scene_or_uniform — lines 44-57: Scene detection + uniform sampling
• extract_keyframes — lines 65-68: FFmpeg keyframe extraction

Metadata tracking (lines 165-170) stores deduped_count for pipeline observability:

metadata = {
    "extracted_count": len(raw_frames),
    "deduped_count": len(dropped_frames),
    "final_count": len(surviving_frames)
}

Practical Usage Examples

Python API

from pathlib import Path
from skills.watch.scripts import frames

# Frame candidates from any extraction engine

candidates = [
    {"path": "frame_0001.jpg", "timestamp_seconds": 0.0},
    {"path": "frame_0002.jpg", "timestamp_seconds": 0.5},
    # ...

]

# Deduplicate with custom threshold (stricter than default)

deduped, dropped = frames.dedupe_perceptual(candidates, threshold=1.5)

print(f"Removed {dropped} frames, kept {len(deduped)}")

CLI Disable Flag

# Bypass deduplication entirely

python -m skills.watch.scripts.frames video.mp4 out_dir --no-dedup

Performance Characteristics

• Memory: O(n) for thumbnail buffers (256 bytes × frame count)
• Compute: O(n) delta calculations, each 256 absolute differences
• Disk: Only surviving frames retained; dropped JPEGs deleted immediately
• Bottleneck: FFmpeg thumbnail generation, not Python comparison loop

For a 10,000-frame video, thumbnail generation dominates at ~2-3 seconds; the deduplication loop completes in milliseconds.

Summary

• Thumbnail generation: Single FFmpeg pass produces 16×16 grayscale images via DEDUP_THUMB = 16
• Similarity metric: Mean absolute pixel difference in _frame_delta, with infinite-distance guard for mismatched buffers
• Selection algorithm: Greedy "last kept" comparison in _dedupe_by_deltas using DEDUP_THRESHOLD = 2.0
• Universal application: All three extraction engines (scene, uniform, keyframe) run deduplication unless --no-dedup is specified
• Metadata tracking: deduped_count field records compression achieved

Frequently Asked Questions

Why use 16×16 thumbnails instead of full-resolution frames?

Downscaling to 16×16 captures structural changes while eliminating noise and compression artifacts. At 256 pixels total, the comparison is fast enough to process thousands of frames per second, yet large enough to distinguish meaningful scene transitions from minor encoding variations. Full-resolution comparison would be orders of magnitude slower with no accuracy benefit for the duplicate-detection task.

What happens if frame dimensions vary within a video?

The _frame_delta function returns infinite distance for buffer length mismatches, as implemented on lines 17-18 of skills/watch/scripts/frames.py. This conservative fallback prevents any deduplication across clips with different resolutions. In practice, this shouldn't occur within a single video file — FFmpeg scaling normalizes dimensions during the thumbnail generation pass.

Can the threshold be adjusted per-video or per-use-case?

Yes — the dedupe_perceptual function accepts a threshold parameter defaulting to DEDUP_THRESHOLD (2.0). Lower values preserve more frames for fine-grained analysis; higher values aggressively compress static content like presentations or security footage. The threshold is not exposed in the top-level CLI, but direct API access allows full customization.

How does this compare to cryptographic or pixel-perfect deduplication?

Perceptual deduplication handles JPEG re-encoding, minor color shifts, and compression variation that would defeat hash-based methods. Identical frames compressed at different quality settings would share perceptual thumbnails but not cryptographic hashes. The MAE metric specifically tolerates small encoding differences while catching meaningful content changes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}