# How Frame Deduplication Works in claude-video: 16×16 Grayscale Thumbnail Analysis

> Discover how claude-video uses 16x16 grayscale thumbnail analysis for efficient frame deduplication. Learn how it drops similar frames to save space and processing time.

- Repository: [bradautomates/claude-video](https://github.com/bradautomates/claude-video)
- Tags: internals
- Published: 2026-08-04

---

**Claude-video removes near-duplicate frames by downsampling extracted frames to 16×16 grayscale thumbnails and dropping candidates whose mean absolute pixel difference falls below a threshold of 2.0.**

The frame deduplication process in `bradautomates/claude-video` is a lightweight, **perceptual-delta filter** designed to conserve your final frame budget for visually distinct content. It runs automatically after any frame extraction engine—whether uniform sampling, scene-change detection, or keyframe extraction—comparing compressed grayscale representations rather than full-resolution images to maximize speed.

## Generating 16×16 Grayscale Thumbnails with FFmpeg

The deduplication pipeline begins by transforming each extracted JPEG into a minimal perceptual fingerprint. Two constants control this behavior in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py):

- **`DEDUP_THUMB = 16`** — the thumbnail width and height in pixels (lines 31–38)
- **`DEDUP_THRESHOLD = 2.0`** — the maximum mean absolute difference for frames to be considered duplicates (lines 31–38)

The `_thumb_frames` function batches this operation through a single **FFmpeg** command that reads the JPEG sequence, scales every frame to 16×16, and converts to grayscale:

```python
cmd = [
    "ffmpeg", "-hide_banner", "-loglevel", "error",
    "-start_number", str(int(digits)),
    "-i", pattern,
    "-vf", f"scale={DEDUP_THUMB}:{DEDUP_THUMB},format=gray",
    "-f", "rawvideo", "-"
]

```

This outputs raw grayscale bytes as a `bytes` object per frame, minimizing memory overhead compared to decoding full images (see `_thumb_frames` in [`frames.py`](https://github.com/bradautomates/claude-video/blob/main/frames.py), lines 42–50).

## Computing Perceptual Similarity with Mean Absolute Difference

With thumbnails generated, the `_frame_delta` helper calculates perceptual distance between two frames. It computes the **mean absolute per-pixel difference** across all 256 pixels (range 0–255):

```python
return sum(abs(x - y) for x, y in zip(a, b)) / len(a)

```

If thumbnail byte strings differ in length—indicating a processing failure—the function returns infinity, ensuring mismatched frames are never collapsed (see `_frame_delta` in [`frames.py`](https://github.com/bradautomates/claude-video/blob/main/frames.py), lines 15–21).

## Greedy Deduplication Algorithm

The `dedupe_perceptual` function orchestrates the actual filtering through `_dedupe_by_deltas`. The algorithm uses a **greedy, single-reference strategy**:

1. **Always retain** the first frame as the initial reference
2. For each subsequent candidate, compute `δ = _frame_delta(candidate_thumb, last_kept_thumb)`
3. **Drop** the candidate if `δ ≤ DEDUP_THRESHOLD` (its JPEG file is deleted)
4. **Keep** the candidate and update the reference if `δ > DEDUP_THRESHOLD`

This approach efficiently collapses static bursts into single representatives while preserving visual changes that appear later. The reference updates immediately upon keeping a frame, so gradual transitions are tracked correctly (see `_dedupe_by_deltas` in [`frames.py`](https://github.com/bradautomates/claude-video/blob/main/frames.py), lines 78–100).

## CLI and Programmatic Usage

The deduplication step is **enabled by default** in the main watch script. It can be controlled via command line or used directly in Python code.

### Default Deduplication (Enabled)

```sh
python -m skills.watch.scripts.watch demo.mp4 output_dir

```

The JSON summary includes `deduped_count` showing how many frames were removed.

### Disable Deduplication

```sh
python -m skills.watch.scripts.watch demo.mp4 output_dir --no-dedup

```

All extracted frames are preserved; `deduped_count` will be `0`.

### Programmatic Control

```python
from skills.watch.scripts.frames import dedupe_perceptual, extract
from pathlib import Path

# Extract frames at 1 fps

candidates = extract(
    video_path="demo.mp4",
    out_dir=Path("tmp"),
    fps=1.0,
    max_frames=200,
)

# Remove near-duplicates

unique_frames, dropped = dedupe_perceptual(candidates)
print(f"Kept {len(unique_frames)}, dropped {dropped}")

```

### Custom Threshold (Advanced)

Increase tolerance for more aggressive deduplication:

```python
unique_frames, dropped = dedupe_perceptual(candidates, threshold=5.0)

```

## Integration and Reporting

The deduplication filter is **engine-agnostic**. In [`watch.py`](https://github.com/bradautomates/claude-video/blob/main/watch.py), the driver explicitly invokes it after extraction completes, unless `--no-dedup` was passed (line 212):

```python
if dedup:
    frames, deduped_count = dedupe_perceptual(frames)

```

The final report annotates results with deduplication statistics (lines 294–300), showing candidates and how many near-duplicates were dropped for each extraction engine.

Unit tests in [`tests/test_dedup.py`](https://github.com/bradautomates/claude-video/blob/main/tests/test_dedup.py) verify greedy behavior, threshold boundary conditions, and graceful handling when thumbnail generation produces unexpected output lengths.

## Summary

- **16×16 grayscale thumbnails** provide a 256-byte perceptual fingerprint per frame via single-pass FFmpeg processing
- **Mean absolute difference** with threshold 2.0 determines visual similarity
- **Greedy single-reference algorithm** collapses static sequences while tracking visual changes
- **Automatic by default** with CLI override and full programmatic access
- **Validated by unit tests** covering edge cases and failure modes

## Frequently Asked Questions

### Why 16×16 pixels specifically?

This resolution strikes a balance between **perceptual sensitivity** and **computational efficiency**. At 256 grayscale values, the entire thumbnail fits in cache while still capturing coarse structural differences that matter for video understanding. Smaller sizes miss meaningful changes; larger sizes slow comparison without improving results for typical video content.

### Can I adjust how aggressive the deduplication is?

Yes. Pass a custom `threshold` to `dedupe_perceptual()`—higher values treat more frames as duplicates. The default 2.0 mean absolute difference corresponds to roughly 0.8% average pixel variation. Values above 10.0 risk collapsing genuinely distinct frames, while values below 1.0 preserve subtle changes including compression artifacts.

### What happens if FFmpeg fails to generate thumbnails?

The `_frame_delta` function detects length mismatches and returns **infinite distance**, guaranteeing those frames are never considered duplicates. The algorithm proceeds safely, and the user receives the deduplication count in output. Unit tests specifically verify this failure-safety path.

### Does deduplication run before or after frame extraction engines?

**After**. The filter receives the candidate list from whichever extraction engine ran—uniform sampling, scene-change detection, or keyframe extraction—then applies perceptual filtering. This design lets engines focus on temporal selection while deduplication handles visual redundancy independently.