# How Perceptual Frame Deduplication Works in Claude-Video: A Technical Deep Dive

> Understand how Claude-Video's perceptual frame deduplication algorithm works. Learn about its efficient grayscale thumbnail comparison and greedy filtering for removing near-duplicate frames.

- Repository: [bradautomates/claude-video](https://github.com/bradautomates/claude-video)
- Tags: deep-dive
- Published: 2026-08-10

---

**Claude-Video removes near-duplicate frames by comparing 16×16 grayscale thumbnails using mean absolute pixel difference with a greedy filtering algorithm.**

The `bradautomates/claude-video` repository implements a lightweight, perceptual deduplication system that collapses static or near-static video segments before sending frames to downstream LLMs. This reduces token costs without sacrificing meaningful visual changes. The algorithm runs entirely on compressed JPEG thumbnails, making it fast enough to process in a single FFmpeg pass.

## Generating 16×16 Grayscale Thumbnails

Every deduplication operation begins with `DEDUP_THUMB = 16` — a fixed thumbnail size that balances detail with computational efficiency.

In [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) (lines 24-31), the `_thumb_frames` function generates these thumbnails through a single FFmpeg invocation:

```python
def _thumb_frames(frame_paths: list[Path]) -> list[bytes]:
    """Return list of 16×16 grayscale raw bytes, one per frame."""
    cmd = [
        "ffmpeg",
        "-hide_banner", "-loglevel", "error",
        "-i", "-",
        "-vf", f"scale={DEDUP_THUMB}:{DEDUP_THUMB}:flags=lanczos,format=gray",
        "-f", "image2pipe", "-c:v", "rawvideo", "-pix_fmt", "gray", "-"
    ]
    # Pipelined JPEG → raw byte buffers

```

**Key design choices:**

- **Lanczos scaling** preserves edge sharpness during aggressive downscaling
- **Grayscale conversion** eliminates color information, focusing purely on luminance changes
- **Raw pipe output** avoids disk I/O — thumbnails stay in memory as byte buffers

This pipeline processes frames in batch, returning one `bytes` object per frame where each buffer contains exactly 256 bytes (16 × 16 pixels).

## Computing Frame Similarity with Mean Absolute Difference

The core similarity metric lives in `_frame_delta` (lines 15-22 of [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py)):

```python
def _frame_delta(a: bytes, b: bytes) -> float:
    """Mean absolute per-pixel difference between two thumbnail buffers."""
    if len(a) != len(b):
        return float('inf')  # Mismatched frames never collapse

    diff_sum = sum(abs(a[i] - b[i]) for i in range(len(a)))
    return diff_sum / len(a)

```

**Critical safeguards:**

- **Length mismatch protection** returns infinite distance, preventing accidental deduplication across different video resolutions or corruption
- **Mean absolute error (MAE)** is preferred over MSE or SSIM for speed — with only 256 pixels, structural analysis offers no practical advantage
- **Float division** yields a normalized 0-255 scale where identical frames score 0.0

## Greedy Deduplication Algorithm

The `dedupe_perceptual` function orchestrates the filtering process (lines 63-71), delegating to `_dedupe_by_deltas` (lines 78-89) for the actual selection logic:

```python
def dedupe_perceptual(frames: list[dict], threshold: float = DEDUP_THRESHOLD) -> tuple:
    """Greedy perceptual deduplication with configurable threshold."""
    if len(frames) <= 1:
        return frames, 0
    
    thumbs = _thumb_frames([f["path"] for f in frames])
    keep_mask = _dedupe_by_deltas(thumbs, threshold)
    
    # Apply mask, delete dropped files, re-index survivors

    ...

```

The **greedy selection algorithm** works as follows:

1. **Initialize** with first frame always kept
2. **Iterate** through remaining frames
3. **Compare** current frame's thumbnail delta to **last kept frame** (not previous frame)
4. **Drop** if `delta ≤ DEDUP_THRESHOLD` (default 2.0), **keep** otherwise
5. **Delete** dropped JPEG files and re-index metadata

This "last kept" reference point — rather than comparing to the immediate predecessor — prevents chain-collapse artifacts where small cumulative differences would otherwise accumulate.

## Default Threshold and Tuning

The `DEDUP_THRESHOLD = 2.0` default reflects empirical calibration:

| Threshold | Behavior | Use Case |
|-----------|----------|----------|
| 0.0 | Only exact duplicates removed | Lossless archival |
| 2.0 | **Default** — near-static content collapsed | General LLM processing |
| 5.0+ | Aggressive compression, may miss subtle changes | High-motion content only |

Thresholds are specified in mean pixel difference (0-255 scale), so 2.0 represents less than 1% average luminance change across the thumbnail.

## Integration Across Extraction Engines

Perceptual deduplication applies universally unless disabled with `--no-dedup`. The three frame extraction engines all invoke `dedupe_perceptual`:

- **`extract_scene_or_uniform`** — lines 44-57: Scene detection + uniform sampling
- **`extract_keyframes`** — lines 65-68: FFmpeg keyframe extraction

Metadata tracking (lines 165-170) stores `deduped_count` for pipeline observability:

```python
metadata = {
    "extracted_count": len(raw_frames),
    "deduped_count": len(dropped_frames),
    "final_count": len(surviving_frames)
}

```

## Practical Usage Examples

### Python API

```python
from pathlib import Path
from skills.watch.scripts import frames

# Frame candidates from any extraction engine

candidates = [
    {"path": "frame_0001.jpg", "timestamp_seconds": 0.0},
    {"path": "frame_0002.jpg", "timestamp_seconds": 0.5},
    # ...

]

# Deduplicate with custom threshold (stricter than default)

deduped, dropped = frames.dedupe_perceptual(candidates, threshold=1.5)

print(f"Removed {dropped} frames, kept {len(deduped)}")

```

### CLI Disable Flag

```bash

# Bypass deduplication entirely

python -m skills.watch.scripts.frames video.mp4 out_dir --no-dedup

```

## Performance Characteristics

- **Memory**: O(n) for thumbnail buffers (256 bytes × frame count)
- **Compute**: O(n) delta calculations, each 256 absolute differences
- **Disk**: Only surviving frames retained; dropped JPEGs deleted immediately
- **Bottleneck**: FFmpeg thumbnail generation, not Python comparison loop

For a 10,000-frame video, thumbnail generation dominates at ~2-3 seconds; the deduplication loop completes in milliseconds.

## Summary

- **Thumbnail generation**: Single FFmpeg pass produces 16×16 grayscale images via `DEDUP_THUMB = 16`
- **Similarity metric**: Mean absolute pixel difference in `_frame_delta`, with infinite-distance guard for mismatched buffers
- **Selection algorithm**: Greedy "last kept" comparison in `_dedupe_by_deltas` using `DEDUP_THRESHOLD = 2.0`
- **Universal application**: All three extraction engines (scene, uniform, keyframe) run deduplication unless `--no-dedup` is specified
- **Metadata tracking**: `deduped_count` field records compression achieved

## Frequently Asked Questions

### Why use 16×16 thumbnails instead of full-resolution frames?

**Downscaling to 16×16 captures structural changes while eliminating noise and compression artifacts.** At 256 pixels total, the comparison is fast enough to process thousands of frames per second, yet large enough to distinguish meaningful scene transitions from minor encoding variations. Full-resolution comparison would be orders of magnitude slower with no accuracy benefit for the duplicate-detection task.

### What happens if frame dimensions vary within a video?

**The `_frame_delta` function returns infinite distance for buffer length mismatches**, as implemented on lines 17-18 of [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py). This conservative fallback prevents any deduplication across clips with different resolutions. In practice, this shouldn't occur within a single video file — FFmpeg scaling normalizes dimensions during the thumbnail generation pass.

### Can the threshold be adjusted per-video or per-use-case?

**Yes — the `dedupe_perceptual` function accepts a `threshold` parameter defaulting to `DEDUP_THRESHOLD` (2.0).** Lower values preserve more frames for fine-grained analysis; higher values aggressively compress static content like presentations or security footage. The threshold is not exposed in the top-level CLI, but direct API access allows full customization.

### How does this compare to cryptographic or pixel-perfect deduplication?

**Perceptual deduplication handles JPEG re-encoding, minor color shifts, and compression variation that would defeat hash-based methods.** Identical frames compressed at different quality settings would share perceptual thumbnails but not cryptographic hashes. The MAE metric specifically tolerates small encoding differences while catching meaningful content changes.