How Perceptual Frame Deduplication Works in Claude-Video

Claude-Video removes near-duplicate frames by comparing 16×16 grayscale thumbnails using a greedy algorithm that drops frames when the mean absolute pixel difference falls below a threshold of 2.0.

Perceptual frame deduplication in the bradautomates/claude-video repository eliminates redundant video frames before LLM processing, reducing token costs while preserving visual information. The system analyzes tiny perceptual thumbnails rather than full-resolution images to detect near-identical content efficiently. This implementation uses a single ffmpeg pass for thumbnail generation followed by a deterministic greedy filter.

Thumbnail Generation for Perceptual Deduplication

The deduplication pipeline begins by creating compact perceptual representations of each extracted frame. In skills/watch/scripts/frames.py, the _thumb_frames function (lines 24-31) orchestrates a single ffmpeg pass that downscales every JPEG to a 16×16 grayscale image (DEDUP_THUMB = 16).

This process returns a list of raw byte buffers, one per frame, minimizing memory overhead compared to loading full-resolution images. The thumbnail dimensions are deliberately small—just 256 total pixels—making subsequent comparisons computationally trivial while retaining enough structural information to detect meaningful visual changes.

Measuring Frame Similarity with Mean Absolute Difference

Once thumbnails are generated, the system quantifies visual differences using the _frame_delta function (lines 15-22). This utility computes the mean absolute per-pixel difference between two thumbnail buffers.

If the two buffers differ in length—indicating corruption or mismatched extraction—the function returns infinite distance. This safety mechanism ensures that malformed frames are never collapsed into valid sequences, preserving data integrity during the deduplication process.

The Greedy Perceptual Deduplication Algorithm

The core logic resides in dedupe_perceptual and its helper _dedupe_by_deltas (lines 63-71 and 78-89). The algorithm operates as follows:

1. Initialize with the first frame as the "last kept" reference.
2. Iterate through remaining frames, comparing each to the last kept thumbnail.
3. Drop the current frame if its delta is ≤ DEDUP_THRESHOLD (default 2.0).
4. Keep the frame and update the reference if the delta exceeds the threshold.

This greedy approach efficiently collapses static or near-static periods while guaranteeing that the first occurrence of any visual state is preserved. When frames are dropped, their corresponding JPEG files are deleted from disk, and the surviving frames are re-indexed to maintain sequential order. The deduplication logic is validated by unit tests in tests/test_dedup.py, which verify that identical frames are collapsed while distinct frames are preserved.

Integration with Frame Extraction Engines

Perceptual frame deduplication applies universally across all frame extraction strategies—including scene-change detection, uniform sampling, and keyframe extraction—unless explicitly disabled. The CLI exposes a --no-dedup flag (lines 165-170) to bypass this step for debugging or specific analysis requirements.

During processing, the extract_scene_or_uniform and extract_keyframes functions (lines 44-57 and 65-68) invoke the deduplication routine automatically. The system tracks efficiency by recording the number of removed frames in the metadata dictionary under deduped_count, allowing downstream consumers to audit compression ratios. End-to-end tests in tests/test_watch.py confirm that this metadata is reported correctly.

Implementation Example

To deduplicate frames programmatically using the default threshold:

from pathlib import Path
from skills.watch.scripts import frames

# Assume `candidates` is a list of dicts produced by any extraction engine:

#   [{"path": "frame_0001.jpg", "timestamp_seconds": 0.0, ...}, …]

# Perform perceptual deduplication (default threshold = 2.0)

deduped_frames, dropped = frames.dedupe_perceptual(candidates)

print(f"Removed {dropped} near‑duplicate frames")
for f in deduped_frames:
    print(f["timestamp_seconds"], f["path"])

To process a video without deduplication via the command line:

python -m skills.watch.scripts.frames video.mp4 out_dir --no-dedup

Summary

• Perceptual frame deduplication in Claude-Video uses 16×16 grayscale thumbnails generated via ffmpeg to minimize computational overhead.
• The mean absolute pixel difference metric quantifies similarity, with infinite distance returned for buffer mismatches to prevent data corruption.
• A greedy algorithm retains the first frame of each visual sequence and drops subsequent frames with deltas ≤ 2.0.
• The system integrates with all extraction engines and tracks removed frames via the deduped_count metadata field.

Frequently Asked Questions

What threshold does Claude-Video use for perceptual deduplication?

The default DEDUP_THRESHOLD is 2.0, representing the maximum mean absolute pixel difference between 16×16 thumbnails before frames are considered distinct. This tight threshold ensures only virtually identical frames are collapsed, preserving meaningful visual changes while eliminating static redundancy.

How does the algorithm handle corrupted or mismatched frame buffers?

The _frame_delta function returns infinite distance when comparing buffers of differing lengths. According to the source code in skills/watch/scripts/frames.py (lines 15-22), this prevents mismatched or corrupted frames from being incorrectly merged with valid frames, ensuring data integrity throughout the pipeline.

Can I disable perceptual frame deduplication when processing videos?

Yes. Pass the --no-dedup flag when invoking the CLI, as implemented in lines 165-170 of skills/watch/scripts/frames.py. This bypasses the dedupe_perceptual call entirely, retaining all extracted frames regardless of visual similarity. Disabling deduplication is useful for debugging or when analyzing frame-level metadata without compression.

What happens to frames that are removed during deduplication?

Dropped frames are permanently deleted from the output directory during the re-indexing phase. The _dedupe_by_deltas function (lines 78-89) removes the underlying JPEG files and compacts the remaining frames into a continuous sequence. The count of deleted frames is stored in the deduped_count metadata field for audit purposes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}