How Frame Deduplication Works in claude-video: 16×16 Grayscale Thumbnail Analysis

Claude-video removes near-duplicate frames by downsampling extracted frames to 16×16 grayscale thumbnails and dropping candidates whose mean absolute pixel difference falls below a threshold of 2.0.

The frame deduplication process in bradautomates/claude-video is a lightweight, perceptual-delta filter designed to conserve your final frame budget for visually distinct content. It runs automatically after any frame extraction engine—whether uniform sampling, scene-change detection, or keyframe extraction—comparing compressed grayscale representations rather than full-resolution images to maximize speed.

Generating 16×16 Grayscale Thumbnails with FFmpeg

The deduplication pipeline begins by transforming each extracted JPEG into a minimal perceptual fingerprint. Two constants control this behavior in skills/watch/scripts/frames.py:

• DEDUP_THUMB = 16 — the thumbnail width and height in pixels (lines 31–38)
• DEDUP_THRESHOLD = 2.0 — the maximum mean absolute difference for frames to be considered duplicates (lines 31–38)

The _thumb_frames function batches this operation through a single FFmpeg command that reads the JPEG sequence, scales every frame to 16×16, and converts to grayscale:

cmd = [
    "ffmpeg", "-hide_banner", "-loglevel", "error",
    "-start_number", str(int(digits)),
    "-i", pattern,
    "-vf", f"scale={DEDUP_THUMB}:{DEDUP_THUMB},format=gray",
    "-f", "rawvideo", "-"
]

This outputs raw grayscale bytes as a bytes object per frame, minimizing memory overhead compared to decoding full images (see _thumb_frames in frames.py, lines 42–50).

Computing Perceptual Similarity with Mean Absolute Difference

With thumbnails generated, the _frame_delta helper calculates perceptual distance between two frames. It computes the mean absolute per-pixel difference across all 256 pixels (range 0–255):

return sum(abs(x - y) for x, y in zip(a, b)) / len(a)

If thumbnail byte strings differ in length—indicating a processing failure—the function returns infinity, ensuring mismatched frames are never collapsed (see _frame_delta in frames.py, lines 15–21).

Greedy Deduplication Algorithm

The dedupe_perceptual function orchestrates the actual filtering through _dedupe_by_deltas. The algorithm uses a greedy, single-reference strategy:

1. Always retain the first frame as the initial reference
2. For each subsequent candidate, compute δ = _frame_delta(candidate_thumb, last_kept_thumb)
3. Drop the candidate if δ ≤ DEDUP_THRESHOLD (its JPEG file is deleted)
4. Keep the candidate and update the reference if δ > DEDUP_THRESHOLD

This approach efficiently collapses static bursts into single representatives while preserving visual changes that appear later. The reference updates immediately upon keeping a frame, so gradual transitions are tracked correctly (see _dedupe_by_deltas in frames.py, lines 78–100).

CLI and Programmatic Usage

The deduplication step is enabled by default in the main watch script. It can be controlled via command line or used directly in Python code.

Default Deduplication (Enabled)

python -m skills.watch.scripts.watch demo.mp4 output_dir

The JSON summary includes deduped_count showing how many frames were removed.

Disable Deduplication

python -m skills.watch.scripts.watch demo.mp4 output_dir --no-dedup

All extracted frames are preserved; deduped_count will be 0.

Programmatic Control

from skills.watch.scripts.frames import dedupe_perceptual, extract
from pathlib import Path

# Extract frames at 1 fps

candidates = extract(
    video_path="demo.mp4",
    out_dir=Path("tmp"),
    fps=1.0,
    max_frames=200,
)

# Remove near-duplicates

unique_frames, dropped = dedupe_perceptual(candidates)
print(f"Kept {len(unique_frames)}, dropped {dropped}")

Custom Threshold (Advanced)

Increase tolerance for more aggressive deduplication:

unique_frames, dropped = dedupe_perceptual(candidates, threshold=5.0)

Integration and Reporting

The deduplication filter is engine-agnostic. In watch.py, the driver explicitly invokes it after extraction completes, unless --no-dedup was passed (line 212):

if dedup:
    frames, deduped_count = dedupe_perceptual(frames)

The final report annotates results with deduplication statistics (lines 294–300), showing candidates and how many near-duplicates were dropped for each extraction engine.

Unit tests in tests/test_dedup.py verify greedy behavior, threshold boundary conditions, and graceful handling when thumbnail generation produces unexpected output lengths.

Summary

• 16×16 grayscale thumbnails provide a 256-byte perceptual fingerprint per frame via single-pass FFmpeg processing
• Mean absolute difference with threshold 2.0 determines visual similarity
• Greedy single-reference algorithm collapses static sequences while tracking visual changes
• Automatic by default with CLI override and full programmatic access
• Validated by unit tests covering edge cases and failure modes

Frequently Asked Questions

Why 16×16 pixels specifically?

This resolution strikes a balance between perceptual sensitivity and computational efficiency. At 256 grayscale values, the entire thumbnail fits in cache while still capturing coarse structural differences that matter for video understanding. Smaller sizes miss meaningful changes; larger sizes slow comparison without improving results for typical video content.

Can I adjust how aggressive the deduplication is?

Yes. Pass a custom threshold to dedupe_perceptual()—higher values treat more frames as duplicates. The default 2.0 mean absolute difference corresponds to roughly 0.8% average pixel variation. Values above 10.0 risk collapsing genuinely distinct frames, while values below 1.0 preserve subtle changes including compression artifacts.

What happens if FFmpeg fails to generate thumbnails?

The _frame_delta function detects length mismatches and returns infinite distance, guaranteeing those frames are never considered duplicates. The algorithm proceeds safely, and the user receives the deduplication count in output. Unit tests specifically verify this failure-safety path.

Does deduplication run before or after frame extraction engines?

After. The filter receives the candidate list from whichever extraction engine ran—uniform sampling, scene-change detection, or keyframe extraction—then applies perceptual filtering. This design lets engines focus on temporal selection while deduplication handles visual redundancy independently.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}