Frame Deduplication in Claude-Video: How the 16×16 Grayscale Thumbnail Approach Works

Claude-Video implements frame deduplication by converting extracted JPEG frames into 16×16 grayscale thumbnails and using a greedy comparison algorithm to drop frames with a mean pixel difference below 2.0.

The bradautomates/claude-video repository uses a perceptual hashing technique to eliminate near-duplicate frames before processing. This 16×16 grayscale thumbnail approach reduces storage overhead and ensures that only visually distinct frames are passed to downstream analysis pipelines. The implementation relies on FFmpeg for efficient thumbnail generation and a simple pixel-wise comparison metric to determine similarity.

How the 16×16 Grayscale Thumbnail System Works

The deduplication system compresses each candidate frame into a tiny 256-byte representation (16×16 pixels, single channel). By comparing these lightweight thumbnails rather than full-resolution images, the tool achieves fast, memory-efficient duplicate detection even on large video files.

The workflow follows four distinct phases: thumbnail generation, delta calculation, greedy filtering, and cleanup.

Generating 16×16 Grayscale Thumbnails with FFmpeg

The _thumb_frames() function in skills/watch/scripts/frames.py generates thumbnails by streaming all candidate JPEGs through a single FFmpeg command. The constant DEDUP_THUMB = 16 defines the target resolution.

# From skills/watch/scripts/frames.py

DEDUP_THUMB = 16  # pixels

def _thumb_frames(candidate_frames: list[dict]) -> list[bytes]:
    """Generate 16x16 grayscale thumbnails for all candidates."""
    # FFmpeg command scales to 16x16 and forces gray format

    cmd = [
        "ffmpeg", "-y", "-i", "concat:" + "|".join(inputs),
        "-vf", f"scale={DEDUP_THUMB}:{DEDUP_THUMB}",
        "-pix_fmt", "gray", "-f", "rawvideo", "pipe:1"
    ]
    # Returns list of 256-byte strings (16*16 pixels)

This approach decodes the entire batch in one process, avoiding the overhead of spawning multiple FFmpeg instances. Each thumbnail consumes exactly 256 bytes (16 × 16 pixels × 1 byte per grayscale value).

Calculating Per-Pixel Differences

The _frame_delta() function computes the mean absolute difference between two thumbnail byte strings. It iterates through corresponding pixel values (0–255) and returns the average absolute deviation.

def _frame_delta(thumb_a: bytes, thumb_b: bytes) -> float:
    """Return mean absolute pixel difference between two thumbnails."""
    if len(thumb_a) != len(thumb_b):
        return float("inf")  # Mismatched dimensions = never duplicate

    
    total = sum(abs(a - b) for a, b in zip(thumb_a, thumb_b))
    return total / len(thumb_a)

If the thumbnail byte strings differ in length—indicating a processing error or corrupted frame—the function immediately returns infinity, ensuring mismatched frames are never collapsed into one another.

Greedy Deduplication Logic

The _dedupe_by_deltas() function implements a greedy chronological filter using the DEDUP_THRESHOLD = 2.0 constant. It walks through the list of candidates, comparing each frame against the last kept frame.

DEDUP_THRESHOLD = 2.0  # Maximum mean difference to consider "duplicate"

def _dedupe_by_deltas(candidate_frames: list[dict], thumbnails: list[bytes]):
    """Remove near-duplicates based on 16x16 grayscale similarity."""
    kept_indices = [0]  # Always keep first frame

    last_kept_thumb = thumbnails[0]
    
    for i, thumb in enumerate(thumbnails[1:], start=1):
        delta = _frame_delta(last_kept_thumb, thumb)
        if delta > DEDUP_THRESHOLD:
            kept_indices.append(i)
            last_kept_thumb = thumb
        else:
            # Delete the duplicate JPEG from disk

            Path(candidate_frames[i]["path"]).unlink()

When the mean difference is ≤ 2.0, the current frame is considered a near-duplicate. The function deletes the corresponding JPEG file from disk and excludes it from the returned list. The survivor becomes the new reference for subsequent comparisons.

Public API Entry Point

The dedupe_perceptual() function serves as the high-level interface. It validates input, orchestrates thumbnail generation, and invokes the delta-based filtering.

def dedupe_perceptual(candidate_frames: list[dict]) -> tuple[list[dict], int]:
    """Deduplicate frames using 16x16 grayscale thumbnail comparison.
    
    Returns:
        Tuple of (filtered_frames, dropped_count)
    """
    if len(candidate_frames) < 2:
        return candidate_frames, 0
    
    thumbnails = _thumb_frames(candidate_frames)
    return _dedupe_by_deltas(candidate_frames, thumbnails)

This function is called by all extraction pipelines—including extract, extract_scene_candidates, and extract_keyframes—unless the user explicitly disables deduplication.

Integration in the Video Processing Pipeline

The deduplication step runs immediately after frame extraction but before budget capping or uniform sampling. In skills/watch/scripts/frames.py, the extract_scene_or_uniform function invokes dedupe_perceptual() at lines 443–447:

# Inside extraction pipeline

if not args.no_dedup:
    candidates, dropped = dedupe_perceptual(candidates)
    logger.info(f"Dropped {dropped} near-duplicate frames")

The CLI entry point in skills/watch/scripts/watch.py (lines 212–224) parses the --no-dedup flag to toggle this behavior. By default, all video processing workflows apply the 16×16 grayscale thumbnail deduplication to maximize frame diversity within the specified budget.

Practical Usage Examples

Deduplicating Extracted Frames in Python

Use the frames module directly to deduplicate a list of candidate dictionaries:

from pathlib import Path
from skills.watch.scripts import frames

# Candidates must contain "path" keys pointing to JPEG files

candidates = [
    {"path": "out/frame_0000.jpg", "timestamp_seconds": 0.0},
    {"path": "out/frame_0001.jpg", "timestamp_seconds": 0.5},
    {"path": "out/frame_0002.jpg", "timestamp_seconds": 1.0},
]

deduped, dropped = frames.dedupe_perceptual(candidates)
print(f"Retained {len(deduped)} frames, removed {dropped} duplicates")

Running the Full Pipeline via CLI

Process a YouTube URL with deduplication enabled (default):

python -m skills.watch.scripts.watch \
    https://www.youtube.com/watch?v=example \
    --max-frames 100

To bypass deduplication and retain all extracted frames:

python -m skills.watch.scripts.watch \
    https://www.youtube.com/watch?v=example \
    --max-frames 100 \
    --no-dedup

Summary

• 16×16 grayscale thumbnails reduce each frame to a 256-byte fingerprint for fast comparison.
• Mean absolute difference between corresponding pixels determines visual similarity, with a threshold of 2.0 defining the duplicate boundary.
• Greedy chronological filtering keeps the first occurrence of a scene and drops subsequent frames that differ by less than the threshold.
• Automatic cleanup deletes duplicate JPEG files from disk during processing.
• Zero-config integration means all extraction pipelines use this method unless --no-dedup is specified.

Frequently Asked Questions

What is the 16×16 grayscale thumbnail approach in Claude-Video?

The 16×16 grayscale thumbnail approach is a perceptual deduplication technique where each video frame is scaled to a 16-pixel square and converted to grayscale. This creates a compact 256-byte representation that preserves structural information while enabling fast pixel-wise comparisons. The system uses these thumbnails to identify and remove near-duplicate frames before further processing.

Why does the deduplication use a threshold of 2.0 for the mean difference?

The DEDUP_THRESHOLD = 2.0 value represents the maximum average pixel difference (on a 0–255 scale) allowed between two 16×16 thumbnails before they are considered distinct. This threshold is strict enough to catch nearly identical frames—such as those from static scenes or slow camera movements—while preserving frames with subtle but meaningful changes. The value is defined as a constant in skills/watch/scripts/frames.py and can be modified for stricter or looser deduplication.

How does the greedy deduplication algorithm handle scene changes?

The greedy algorithm processes frames chronologically, maintaining a reference to the last kept frame. When a new frame differs from this reference by more than 2.0 mean absolute difference, it becomes the new reference. This ensures that the first frame of a new scene is always retained, while subsequent similar frames are dropped. The approach favors early frames in a sequence, which works well for extracting representative stills from video content.

Where can I find the unit tests for the deduplication logic?

The deduplication logic is validated in tests/test_dedup.py, which contains unit tests covering identical frames, completely distinct frames, and threshold edge cases. These tests verify that _frame_delta() returns infinity for mismatched thumbnail sizes and that dedupe_perceptual() correctly calculates the dropped count while preserving the temporal order of retained frames.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}