# Claude-Video Frame Deduplication Algorithm: Mean Absolute Pixel Difference on 16×16 Thumbnails Explained

> Discover Claude-Video's frame deduplication algorithm. Learn how mean absolute pixel difference on 16x16 thumbnails efficiently removes duplicate frames with a clear threshold.

- Repository: [bradautomates/claude-video](https://github.com/bradautomates/claude-video)
- Tags: deep-dive
- Published: 2026-08-03

---

**Claude-Video uses a three-stage frame deduplication pipeline that compares 16×16 grayscale thumbnails using mean absolute pixel difference, collapsing near-identical frames with a configurable threshold.**

The `bradautomates/claude-video` repository implements an efficient perceptual deduplication system that runs automatically after frame extraction. This article breaks down the complete algorithm as implemented in the source code, from thumbnail generation through the greedy comparison strategy that decides which frames survive.

## Thumbnail Generation with FFmpeg

Every extracted JPEG frame gets reduced to a 256-byte fingerprint before any comparison happens. The `_thumb_frames` helper in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) handles this in a single FFmpeg pass:

```python

# frames.py#L424-L461

def _thumb_frames(jpeg_paths: list[str]) -> list[bytes]:
    """Build 16×16 grayscale thumbnails via raw video stream."""
    # FFmpeg pipeline: decode JPEGs → scale to 16x16 → grayscale raw output

    # Output format: 256 bytes per frame (16 * 16 = 256 pixel intensities)

    ...

```

The constant `DEDUP_THUMB = 16` controls the output dimensions. Using `format=gray` eliminates color channels entirely, producing a raw byte stream that requires zero external image libraries. Each thumbnail contains exactly 256 values in the range 0-255, representing pixel intensity from black to white.

This design prioritizes **speed and simplicity**: a 256-byte array fits in CPU cache, processes in microseconds, and avoids dependencies like PIL or OpenCV.

## Mean Absolute Pixel Difference Calculation

The core similarity metric lives in `_frame_delta` (frames.py#L415-L422). For any two thumbnails, it computes the average absolute deviation across all 256 positions:

```python
def _frame_delta(a: bytes, b: bytes) -> float:
    """Mean absolute per-pixel difference (0-255) between two grayscale thumbnails."""
    if not a or len(a) != len(b):
        return float("inf")                     # mismatched size → never collapse

    return sum(abs(x - y) for x, y in zip(a, b)) / len(a)

```

The return value has direct interpretability:

- **0.0** → identical thumbnails (perfect match)
- **255.0** → maximum opposite (pure black vs. pure white)
- **Typical threshold: 2.0** → allows minor compression artifacts or sub-pixel shifts

The `float("inf")` guard ensures defensive behavior: any malformed or mismatched thumbnail pairs automatically fail the duplicate test rather than crashing the pipeline.

## Greedy Deduplication Algorithm

The `_dedupe_by_deltas` function (frames.py#L779-L801) implements a **single-pass greedy strategy** that preserves chronological order while maximizing compression:

```python
def _dedupe_by_deltas(candidates, thumbs, threshold=DEDUP_THRESHOLD):
    if len(thumbs) != len(candidates) or len(candidates) <= 1:
        return candidates, 0                     # fail-open: keep everything

    
    kept = [candidates[0]]                       # always keep first frame

    last = thumbs[0]                              # reference thumbnail

    dropped = []
    
    for cand, thumb in zip(candidates[1:], thumbs[1:]):
        if _frame_delta(thumb, last) <= threshold:
            dropped.append(cand)                  # too similar → discard

        else:
            kept.append(cand)
            last = thumb                          # new reference point

    
    # Filesystem cleanup and re-indexing of survivors...

    return kept, len(dropped)

```

### Why Greedy Works Here

The algorithm compares each candidate only to the **most recent survivor**, not to all previous frames. This choice reflects how video content actually behaves:

- Static scenes (slides, terminal screens, paused video) generate long runs of near-identical thumbnails → collapsed to single representative
- Hard cuts or camera motion change thumbnails sufficiently → trigger new reference points
- Gradual changes (slow pans, fades) accumulate difference frame-to-frame until threshold breach

The `DEDUP_THRESHOLD` default of **2.0** was chosen empirically: it catches JPEG recompression noise and tiny temporal shifts while preserving genuine content changes.

## Public API and Integration

The `dedupe_perceptual` function orchestrates the complete pipeline. It is invoked automatically after frame extraction unless `--no-dedup` is passed to the CLI.

### Basic Usage

```python
from pathlib import Path
import frames

# Extract frames first

candidates = frames.extract(
    video_path="my_video.mp4",
    out_dir=Path("out_dir"),
    fps=2.0,
    max_frames=100,
)

# Run perceptual deduplication

survivors, dropped = frames.dedupe_perceptual(candidates)

print(f"Removed {dropped} near-duplicate frames")
for f in survivors:
    print(f["index"], f["timestamp_seconds"], f["path"])

```

### Direct Invocation with Custom Threshold

```python

# Useful for testing or fine-tuning sensitivity

survivors, dropped = frames.dedupe_perceptual(candidates, threshold=2.0)

```

## Why 16×16 Resolution?

The thumbnail size represents a deliberate engineering tradeoff:

| Factor | 16×16 Advantage |
|--------|-----------------|
| **Speed** | 256 integer operations per comparison; thousands of frames process in milliseconds |
| **Memory** | Entire thumbnail set for 1000 frames fits in ~250KB |
| **Robustness** | Down-scaling suppresses sensor noise, compression artifacts, and sub-pixel motion |
| **Portability** | Pure standard library; no NumPy, PIL, or OpenCV dependencies |

At 16×16, a solid-color slide and a terminal window with static text produce distinct luminance patterns. Meanwhile, consecutive frames of identical content—common in screen recordings—collapse regardless of minor encoding variations.

## Unit Test Validation

The test suite in [`tests/test_dedup.py`](https://github.com/bradautomates/claude-video/blob/main/tests/test_dedup.py) documents expected behavior through concrete examples:

```python
def test_dedupe_collapses_identical_run(tmp_path: Path):
    # Create 5 identical dummy JPEGs

    cands = [
        {"index": i, "timestamp_seconds": float(i), "path": str(tmp_path / f"frame_{i:04d}.jpg"), "reason": "scene-change"}
        for i in range(5)
    ]
    for c in cands:
        Path(c["path"]).write_bytes(b"x")          # dummy data

    thumbs = [bytes([0] * 256)] * 5                # all thumbnails identical

    survivors, dropped = frames._dedupe_by_deltas(cands, thumbs, threshold=2.0)

    assert dropped == 4
    assert len(survivors) == 1
    assert survivors[0]["index"] == 0

```

This test verifies the **fail-fast property**: given truly identical thumbnails, only the first frame survives and the rest are dropped with correct counting.

## Key Implementation Files

- **[`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py)** — Contains `_thumb_frames`, `_frame_delta`, `_dedupe_by_deltas`, and `dedupe_perceptual`
- **[`tests/test_dedup.py`](https://github.com/bradautomates/claude-video/blob/main/tests/test_dedup.py)** — Unit tests covering threshold boundaries, empty inputs, and mismatch handling
- **[`skills/watch/scripts/watch.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/watch.py)** — CLI entry point that conditionally triggers deduplication

## Summary

- **Thumbnail generation**: FFmpeg produces 16×16 grayscale images (256 bytes each) via `_thumb_frames`
- **Difference metric**: `_frame_delta` computes mean absolute pixel difference on raw byte arrays
- **Deduplication logic**: Greedy single-pass in `_dedupe_by_deltas` compares to last-kept frame using threshold ≤ 2.0
- **Default behavior**: Automatically active; disable with `--no-dedup` flag
- **Performance**: Pure Python standard library, no external dependencies, sub-millisecond per comparison

## Frequently Asked Questions

### What does the mean absolute pixel difference actually measure?

It measures the average luminance change between two downscaled frames. Each of the 256 pixel positions contributes `abs(a - b)`; the sum divided by 256 yields a 0-255 scale where lower values indicate higher visual similarity. This matches human perception better than raw byte equality because it tolerates minor compression noise.

### Why use 16×16 thumbnails instead of full-resolution frames?

Full frames would require orders of magnitude more memory and computation for marginal accuracy gains. At 16×16, the algorithm preserves enough structural information to distinguish scene changes while operating entirely in CPU cache. The down-scaling also acts as a natural low-pass filter that ignores irrelevant high-frequency variation.

### What happens if thumbnails have mismatched sizes?

The `_frame_delta` function returns `float("inf")`, which always exceeds any finite threshold. This fail-open design ensures that corrupted or partial data never causes false deduplication—problematic frames simply pass through unchanged.

### Can I adjust the sensitivity of duplicate detection?

Yes. The `threshold` parameter in `dedupe_perceptual` accepts any positive float. Higher values (e.g., 5.0 or 10.0) collapse more frames including subtle transitions; lower values (e.g., 0.5) preserve nearly everything except perfect duplicates. The default 2.0 balances noise tolerance against content preservation for typical screen recordings.