How Near-Duplicate Frames Are Detected in Claude Video Using Thumbnails

Claude Video detects near-duplicate frames by generating 16×16 grayscale thumbnails and greedily removing any frame whose mean absolute pixel difference to the previous kept frame is ≤ 2.0.

The deduplication system in bradautomates/claude-video provides a lightweight, perceptual approach to collapsing visually identical frames without relying on heavy computer vision models. This article breaks down the thumbnail-based detection algorithm implemented in the frame processing pipeline.

Thumbnail Generation with FFmpeg

The detection pipeline begins in skills/watch/scripts/frames.py with the _thumb_frames function. This utility converts extracted JPEG frames into compact grayscale thumbnails through a single optimized FFmpeg command.

The command scales each frame to DEDUP_THUMB × DEDUP_THUMB (16 × 16 pixels) in grayscale format, outputting raw video bytes that are then split into individual thumbnail arrays.

# From skills/watch/scripts/frames.py

DEDUP_THUMB = 16  # pixels

DEDUP_THRESHOLD = 2.0  # mean pixel difference threshold

Raw bytes from FFmpeg are parsed into one thumbnail per source frame, creating the compact representations used for all subsequent comparisons.

Mean Pixel Difference Calculation

The _frame_delta function computes similarity between two thumbnails. It takes two thumbnail byte arrays a and b and returns the mean absolute per-pixel difference on a 0-255 scale.

If thumbnails differ in length—indicating malformed or mismatched data—the function returns infinity to ensure maximally different treatment.

# Conceptual usage of _frame_delta

delta = _frame_delta(thumb_a, thumb_b)  # float in range [0, 255]

is_duplicate = delta <= DEDUP_THRESHOLD  # default: ≤ 2.0

This simple metric avoids cryptographic hashing pitfalls where visually similar frames produce wildly different hashes.

The Greedy Deduplication Algorithm

The dedupe_perceptual function orchestrates the removal process. It operates as a greedy chronological filter: keep the first frame, then discard any subsequent frame whose thumbnail delta to the most recently kept frame falls at or below the threshold.

from pathlib import Path
from skills.watch.scripts.frames import dedupe_perceptual

# Candidate frames from extraction pipeline

candidates = [
    {"index": 0, "timestamp_seconds": 0.0, "path": "out/frame_0000.jpg", "reason": "scene-change"},
    {"index": 1, "timestamp_seconds": 0.5, "path": "out/frame_0001.jpg", "reason": "scene-change"},
    # ...

]

# Run thumbnail-based deduplication

survivors, dropped = dedupe_perceptual(candidates)
print(f"Kept {len(survivors)} frames, dropped {dropped} near-duplicates.")

The helper _dedupe_by_deltas implements the core iteration logic:

1. Initialize with first candidate as "last kept"
2. For each subsequent candidate, compute thumbnail delta to last kept
3. If delta ≤ DEDUP_THRESHOLD, delete the JPEG file and continue
4. Otherwise, keep the frame and update "last kept"

Dropped frames are permanently removed from disk, and surviving frames receive reindexed metadata.

Configuration and Tuning

Two constants control the detection behavior near the top of frames.py:

Constant	Value	Purpose
DEDUP_THUMB	16	Thumbnail size in pixels (square)
DEDUP_THRESHOLD	2.0	Maximum mean pixel difference for duplicate classification

The 16×16 resolution strikes a balance: large enough to preserve structural information, small enough for fast comparison. The threshold of 2.0 (roughly 0.8% of the 0-255 range) captures near-identical frames while preserving subtle but meaningful visual changes.

Integration in Extraction Pipelines

Deduplication is optional and controlled via the --no-dedup CLI flag or dedup=False parameter. When enabled, it runs in three extraction modes as implemented in skills/watch/scripts/watch.py:

• Uniform extraction (extract): Frames deduplicated before return
• Scene-based extraction (extract_scene_or_uniform): Applied after scene detection
• Key-frame extraction (extract_keyframes): Near-identical keyframes collapsed

# Default: deduplication enabled

python -m skills.watch.scripts.watch https://youtube.com/... output_dir

# Disable to keep all extracted frames

python -m skills.watch.scripts.watch https://youtube.com/... output_dir --no-dedup

Performance Characteristics

The thumbnail approach offers significant advantages over alternatives:

• Speed: Single FFmpeg pass for thumbnail generation, O(n) comparison loop
• Memory: 16×16 grayscale = 256 bytes per thumbnail versus full frames
• Determinism: Fixed thresholds produce reproducible results
• Simplicity: No neural network dependencies or model weights

The mean pixel difference metric intentionally trades sophistication for predictability—unlike learned perceptual hashes, behavior is fully inspectable and tunable via the threshold constant.

Summary

• Claude Video detects near-duplicate frames using 16×16 grayscale thumbnails generated via FFmpeg in _thumb_frames
• Similarity is measured by mean absolute pixel difference in _frame_delta, bounded by DEDUP_THRESHOLD (2.0)
• The greedy chronological algorithm in dedupe_perceptual keeps first frames and drops subsequent frames below threshold
• Deduplication is configurable via --no-dedup flag and applies across uniform, scene-based, and key-frame extraction modes
• All implementation lives in skills/watch/scripts/frames.py with integration points in watch.py

Frequently Asked Questions

What resolution are the thumbnails used for duplicate detection?

Thumbnails are 16×16 pixels grayscale, defined by the DEDUP_THUMB = 16 constant in skills/watch/scripts/frames.py. This 256-byte representation is small enough for fast comparison while retaining enough structural information for perceptual similarity detection.

How does the threshold value of 2.0 translate to visual similarity?

The threshold represents the maximum mean absolute pixel difference on a 0-255 scale. A value of 2.0 means average pixel values can differ by at most roughly 0.8% of full range—essentially capturing frames that are visually indistinguishable to human perception while preserving frames with any meaningful change.

Can I disable near-duplicate detection when processing videos?

Yes. Pass the --no-dedup flag to the watch command, or set dedup=False when calling extraction functions programmatically. This preserves every extracted frame regardless of visual similarity to preceding frames.

Why use mean pixel difference instead of perceptual hashing algorithms like pHash?

Mean pixel difference on tiny thumbnails provides deterministic, inspectable behavior without external dependencies. The Claude Video implementation prioritizes simplicity and speed over handling adversarial cases (resized, rotated, or heavily compressed duplicates) that production perceptual hashes address.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}