Frame Deduplication Using Perceptual Diff in Claude-Video: A Technical Deep Dive

Claude-Video removes near-identical frames by generating 16×16 grayscale thumbnails, computing mean-absolute-per-pixel differences, and greedily discarding frames whose delta falls below a configurable threshold.

The bradautomates/claude-video repository implements an efficient frame deduplication using perceptual diff strategy that reduces redundant visual data before LLM processing. This technique generates tiny perceptual hashes via FFmpeg scaling, compares frames using pixel-level difference metrics, and applies a greedy algorithm to preserve visual variety while minimizing token consumption.

How Perceptual Frame Deduplication Works

The core implementation resides in skills/watch/scripts/frames.py, where three private functions orchestrate the deduplication pipeline. The process converts full-resolution JPEGs into compact grayscale representations, calculates perceptual distances, and filters duplicates based on a similarity threshold.

Step 1: Thumbnail Generation with FFmpeg

The _thumb_frames function invokes a single FFmpeg pass that scales every extracted frame to DEDUP_THUMB × DEDUP_THUMB (default 16×16) grayscale images. This produces raw byte arrays representing simplified perceptual fingerprints rather than cryptographic hashes.

# Internal implementation at skills/watch/scripts/frames.py lines 24-33

def _thumb_frames(frames: list[dict]) -> list[bytes]:
    # FFmpeg scales to 16x16 grayscale, returning raw pixel bytes

    # for each frame in the candidate list

    pass

These tiny thumbnails preserve luminance patterns while discarding high-frequency noise and color variations, making the comparison resistant to minor encoding artifacts.

Step 2: Delta Calculation via Mean Absolute Difference

The _frame_delta function computes the mean absolute difference between two thumbnail byte arrays. This metric calculates the average absolute deviation per pixel, providing a normalized perceptual distance score.

# Implementation at skills/watch/scripts/frames.py lines 15-22

def _frame_delta(a: bytes, b: bytes) -> float:
    if len(a) != len(b):
        return float("inf")  # Safety check for decode failures

    return sum(abs(x - y) for x, y in zip(a, b)) / len(a)

If thumbnail arrays differ in length (indicating a decode hiccup), the function returns infinity to prevent erroneous deduplication.

Step 3: Greedy Deduplication Algorithm

The _dedupe_by_deltas function implements a greedy chronological filter. It keeps the first frame as a reference, then compares each subsequent thumbnail to the last kept frame—not the previous frame in the source list. When the delta falls below DEDUP_THRESHOLD (default 2.0), the frame is deleted; otherwise, it becomes the new reference.

# Implementation at skills/watch/scripts/frames.py lines 80-99

def _dedupe_by_deltas(frames: list[dict], thumbs: list[bytes], 
                      threshold: float = 2.0) -> tuple[list[dict], int]:
    kept = [frames[0]]
    last_thumb = thumbs[0]
    dropped = 0
    
    for frame, thumb in zip(frames[1:], thumbs[1:]):
        delta = _frame_delta(last_thumb, thumb)
        if delta <= threshold:
            dropped += 1
            # Remove the duplicate JPEG file

        else:
            kept.append(frame)
            last_thumb = thumb
    return kept, dropped

After processing, surviving frames are re-indexed sequentially to maintain contiguous naming.

Integration with Frame Extraction Engines

The public dedupe_perceptual helper (lines 64-71) ties thumbnail generation to the greedy filter, returning both the filtered candidate list and the drop count. This function integrates into three extraction pipelines:

1. Scene Detection Engine (extract_scene_or_uniform): Runs after scene-change detection but before the final frame cap
2. Keyframe Engine (extract_keyframes): Processes frames after FFmpeg keyframe extraction
3. Uniform Sampling: Applied when running frames.py directly as a standalone script

Each engine passes its candidate frame list through the same deduplication pipeline, ensuring consistent behavior across extraction strategies.

Configuration and Usage Examples

Control deduplication via the dedup boolean parameter and tune sensitivity with the DEDUP_THRESHOLD constant (default 2.0). Lower values preserve more frames; higher values aggressively remove near-duplicates.

Enabling Perceptual Deduplication in Scene Detection

from pathlib import Path
import skills.watch.scripts.frames as frames

candidates, meta = frames.extract_scene_or_uniform(
    video_path="example.mp4",
    out_dir=Path("frames"),
    fps=2.0,
    target_frames=50,
    dedup=True,  # Enable perceptual diff deduplication

)

print(f"Dropped {meta['deduped_count']} duplicate frames")
print(f"Retained {len(candidates)} unique frames")

Manual Deduplication on Existing Frames

# Extract candidates without deduplication

frames_list, _ = frames.extract_scene_candidates(
    video_path="example.mp4",
    out_dir=Path("tmp"),
    max_frames=None,
)

# Apply perceptual filtering manually with custom threshold

kept, dropped = frames.dedupe_perceptual(frames_list, threshold=1.5)
print(f"Removed {dropped} duplicates, {len(kept)} frames remain")

Disabling Deduplication for Debug

candidates, meta = frames.extract_keyframes(
    video_path="example.mp4",
    out_dir=Path("keyframes"),
    max_frames=50,
    dedup=False,  # Disable to preserve all extracted frames

)
assert meta["deduped_count"] == 0

Summary

• Perceptual deduplication in Claude-Video uses 16×16 grayscale thumbnails generated via FFmpeg to create lightweight frame signatures.
• The mean-absolute-per-pixel difference metric quantifies visual similarity between frames, with a default threshold of 2.0 determining collapse boundaries.
• A greedy chronological algorithm compares each frame to the last kept reference, ensuring O(n) complexity while preserving temporal visual variety.
• Implementation resides primarily in skills/watch/scripts/frames.py, accessible through dedupe_perceptual() or automatic integration with scene, keyframe, and uniform extraction engines.

Frequently Asked Questions

How does perceptual diff differ from cryptographic hashing for frame deduplication?

Cryptographic hashing (like MD5 or SHA-256) identifies bitwise-identical files, failing if a single pixel changes. Perceptual diff, as implemented in Claude-Video, computes the mean absolute pixel difference between 16×16 grayscale thumbnails, allowing frames to be considered duplicates even with minor compression artifacts or lighting changes. This tolerance prevents sending visually redundant yet technically distinct frames to downstream LLM processing.

Why does Claude-Video use 16×16 grayscale thumbnails instead of full-resolution images?

The DEDUP_THUMB constant (default 16) creates perceptually representative fingerprints weighing only 256 bytes per frame. This size provides sufficient luminance pattern information to detect scene changes while minimizing memory overhead and computation time. Grayscale conversion removes color channel variance that rarely affects semantic content understanding, focusing the comparison on structural similarity rather than chromatic noise.

What happens when the frame delta threshold is set to 0.0?

Setting threshold=0.0 configures the deduplicator to only remove bitwise-identical thumbnails. Since _frame_delta computes mean absolute difference, a zero threshold requires every pixel in the 16×16 grid to match exactly. In practice, this rarely triggers due to JPEG encoding variability between frames, effectively disabling practical deduplication while still incurring the thumbnail generation overhead.

Can frame deduplication be applied to existing frame directories without re-extracting?

The dedupe_perceptual() function requires the internal frame metadata structure produced by extraction functions, not raw JPEG files on disk. To deduplicate existing frames, you would need to reconstruct the candidate list format expected by _thumb_frames and _dedupe_by_deltas, including proper path references to the frame files. The current implementation is designed for integration during the extraction pipeline rather than post-hoc directory cleaning.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}