How the Frame Delta Algorithm Uses 16×16 Grayscale Thumbnails for Similarity Detection in Claude Video

The frame delta algorithm in bradautomates/claude-video converts extracted video frames to 16×16 grayscale thumbnails and compares them using mean absolute pixel difference to detect and remove visually duplicate frames.

The claude-video repository implements a lightweight, deterministic frame deduplication system that operates entirely on low-resolution thumbnails without external image libraries. This approach preserves rapid scene changes while eliminating redundant frames from static segments.

How 16×16 Thumbnails Are Generated

The thumbnail pipeline uses ffmpeg in a single pass to transform full-resolution frames into compact byte arrays ideal for fast comparison.

The DEDUP_THUMB Constant

The thumbnail dimensions are controlled by a single constant in skills/watch/scripts/frames.py:

# Line 31

DEDUP_THUMB = 16  # 16×16 pixel thumbnails

This produces 256-byte arrays (16 × 16 = 256) that fit efficiently in memory even for long videos.

The _thumb_frames Function

The _thumb_frames function (lines 24-34, 44-51) executes ffmpeg with specific scaling and format filters:

# ffmpeg scales to 16×16 and converts to grayscale

"scale=16:16,format=gray"

# Raw pixel bytes stream directly to Python

"-f", "rawvideo"

Each frame becomes a bytes object exactly 256 bytes in length. No image decoding libraries are required—pure stdlib operations handle the comparison.

The Mean Absolute Difference Metric

Similarity computation lives in _frame_delta (lines 15-22), which implements a robust pixel-wise comparison.

_frame_delta Implementation

The function computes mean absolute per-pixel difference on the 0-255 grayscale scale:

def _frame_delta(thumb_a: bytes, thumb_b: bytes) -> float:
    if len(thumb_a) != len(thumb_b):
        return float("inf")  # Safety: never merge mismatched thumbnails

    # Sum of absolute differences, averaged across 256 pixels

    return sum(abs(a - b) for a, b in zip(thumb_a, thumb_b)) / 256

The length check serves as a defensive guard against ffmpeg decoding errors or truncated streams.

Greedy Deduplication with Configurable Threshold

The dedupe_perceptual function orchestrates the full deduplication pipeline, while _dedupe_by_deltas performs the actual filtering.

Threshold Configuration

The default tolerance is defined on line 38:

DEDUP_THRESHOLD = 2.0  # Mean absolute difference ≤ 2.0 merges frames

This conservative default ensures only essentially identical frames are merged—preserving slide transitions, subtle animations, and camera movement.

The _dedupe_by_deltas Algorithm

Lines 82-88 implement chronological greedy deduplication:

1. Keep the first frame automatically
2. For each subsequent frame, compare its thumbnail to the last kept frame
3. If _frame_delta ≤ DEDUP_THRESHOLD: delete the JPEG and skip
4. Otherwise: keep the frame and update the comparison baseline

This approach maintains temporal ordering and avoids the O(n²) cost of pairwise comparison across all frames.

Complete Usage Examples

Running Deduplication Manually

from pathlib import Path
from skills.watch.scripts.frames import extract, dedupe_perceptual

# Extract full-resolution frames at 1 fps

frames = extract(
    video_path="example.mp4",
    out_dir=Path("tmp/frames"),
    fps=1.0,
    resolution=512,
    max_frames=200,
)

# Collapse near-identical frames using 16×16 grayscale thumbnails

deduped_frames, dropped = dedupe_perceptual(frames)

print(f"Kept {len(deduped_frames)} frames, dropped {dropped} duplicates")

Adjusting Similarity Sensitivity

from skills.watch.scripts.frames import dedupe_perceptual, DEDUP_THRESHOLD

# Raise threshold to retain more frames (less aggressive deduplication)

custom_threshold = 5.0
deduped, _ = dedupe_perceptual(frames, threshold=custom_threshold)

print(f"With threshold {custom_threshold}, {len(deduped)} frames remain")

Higher thresholds allow more visual variation before frames are considered duplicates—useful for videos with gradual lighting changes or slow pans.

Why 16×16 Grayscale Works

The thumbnail approach trades spatial precision for computational efficiency:

Property	Benefit
16×16 resolution	256 comparisons per frame pair—fast in pure Python
Grayscale	Eliminates color space complexity; luminance captures structural similarity
Raw bytes	No PIL, OpenCV, or numpy dependencies
Mean absolute difference	Simple, interpretable metric with clear threshold semantics

The algorithm specifically targets shot-level deduplication rather than semantic similarity. Two frames from different camera angles may appear similar as thumbnails, but the conservative threshold and chronological comparison prevent incorrect merging.

Key Source Files

Path	Purpose
skills/watch/scripts/frames.py	Core implementation: DEDUP_THUMB, DEDUP_THRESHOLD, _thumb_frames, _frame_delta, dedupe_perceptual, _dedupe_by_deltas
tests/test_dedup.py	Unit tests validating threshold behavior and edge cases
skills/watch/SKILL.md	Skill definition invoking the frame pipeline

Summary

• 16×16 thumbnails are generated via ffmpeg with scale=16:16,format=gray filtering
• Mean absolute difference compares thumbnails on a 0-255 scale with defensive length checking
• Greedy chronological deduplication keeps the first frame of each visually distinct segment
• Configurable threshold (DEDUP_THRESHOLD = 2.0) controls merge aggressiveness
• Pure stdlib implementation requires no external image processing libraries

Frequently Asked Questions

Why 16×16 pixels specifically?

The 16×16 size balances detection sensitivity with computational speed. At 256 bytes per thumbnail, even hour-long videos at 1 fps consume only ~900KB of thumbnail memory. Larger thumbnails improve discrimination but slow comparison; smaller thumbnails miss subtle scene changes. The source code hardcodes this at line 31 as DEDUP_THUMB = 16.

How does the threshold value translate to visual similarity?

A DEDUP_THRESHOLD of 2.0 means the average pixel difference across all 256 grayscale values must be ≤ 2 intensity levels (on 0-255). For context: identical frames score 0.0, a single pixel shifted by 2 levels across the entire image scores 2.0, and completely inverted frames (0↔255) score 255.0. The default is intentionally conservative to preserve legitimate content variations.

Can this algorithm detect duplicate frames out of order?

No. The _dedupe_by_deltas implementation only compares each frame to the most recently kept frame in chronological sequence. This design choice preserves temporal coherence and runs in O(n) time. For non-chronological duplicate detection, a different hashing approach would be required.

What happens if ffmpeg produces malformed thumbnails?

The _frame_delta function includes explicit length validation. If len(thumb_a) != len(thumb_b), it immediately returns float("inf"), ensuring the frames are never merged. This guards against truncated streams, decode errors, or filesystem corruption without crashing the pipeline.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}