# How the Frame Delta Algorithm Uses 16×16 Grayscale Thumbnails for Similarity Detection in Claude Video

> Learn how the frame delta algorithm in Claude Video uses 16x16 grayscale thumbnails for efficient similarity detection and duplicate frame removal.

- Repository: [bradautomates/claude-video](https://github.com/bradautomates/claude-video)
- Tags: deep-dive
- Published: 2026-08-10

---

**The frame delta algorithm in `bradautomates/claude-video` converts extracted video frames to 16×16 grayscale thumbnails and compares them using mean absolute pixel difference to detect and remove visually duplicate frames.**

The `claude-video` repository implements a lightweight, deterministic frame deduplication system that operates entirely on low-resolution thumbnails without external image libraries. This approach preserves rapid scene changes while eliminating redundant frames from static segments.

## How 16×16 Thumbnails Are Generated

The thumbnail pipeline uses **ffmpeg** in a single pass to transform full-resolution frames into compact byte arrays ideal for fast comparison.

### The DEDUP_THUMB Constant

The thumbnail dimensions are controlled by a single constant in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py):

```python

# Line 31

DEDUP_THUMB = 16  # 16×16 pixel thumbnails

```

This produces 256-byte arrays (`16 × 16 = 256`) that fit efficiently in memory even for long videos.

### The _thumb_frames Function

The `_thumb_frames` function (lines 24-34, 44-51) executes ffmpeg with specific scaling and format filters:

```python

# ffmpeg scales to 16×16 and converts to grayscale

"scale=16:16,format=gray"

# Raw pixel bytes stream directly to Python

"-f", "rawvideo"

```

Each frame becomes a `bytes` object exactly 256 bytes in length. No image decoding libraries are required—pure stdlib operations handle the comparison.

## The Mean Absolute Difference Metric

Similarity computation lives in `_frame_delta` (lines 15-22), which implements a robust pixel-wise comparison.

### _frame_delta Implementation

The function computes **mean absolute per-pixel difference** on the 0-255 grayscale scale:

```python
def _frame_delta(thumb_a: bytes, thumb_b: bytes) -> float:
    if len(thumb_a) != len(thumb_b):
        return float("inf")  # Safety: never merge mismatched thumbnails

    # Sum of absolute differences, averaged across 256 pixels

    return sum(abs(a - b) for a, b in zip(thumb_a, thumb_b)) / 256

```

The length check serves as a defensive guard against ffmpeg decoding errors or truncated streams.

## Greedy Deduplication with Configurable Threshold

The `dedupe_perceptual` function orchestrates the full deduplication pipeline, while `_dedupe_by_deltas` performs the actual filtering.

### Threshold Configuration

The default tolerance is defined on line 38:

```python
DEDUP_THRESHOLD = 2.0  # Mean absolute difference ≤ 2.0 merges frames

```

This conservative default ensures only essentially identical frames are merged—preserving slide transitions, subtle animations, and camera movement.

### The _dedupe_by_deltas Algorithm

Lines 82-88 implement **chronological greedy deduplication**:

1. Keep the first frame automatically
2. For each subsequent frame, compare its thumbnail to the **last kept** frame
3. If `_frame_delta` ≤ `DEDUP_THRESHOLD`: delete the JPEG and skip
4. Otherwise: keep the frame and update the comparison baseline

This approach maintains temporal ordering and avoids the O(n²) cost of pairwise comparison across all frames.

## Complete Usage Examples

### Running Deduplication Manually

```python
from pathlib import Path
from skills.watch.scripts.frames import extract, dedupe_perceptual

# Extract full-resolution frames at 1 fps

frames = extract(
    video_path="example.mp4",
    out_dir=Path("tmp/frames"),
    fps=1.0,
    resolution=512,
    max_frames=200,
)

# Collapse near-identical frames using 16×16 grayscale thumbnails

deduped_frames, dropped = dedupe_perceptual(frames)

print(f"Kept {len(deduped_frames)} frames, dropped {dropped} duplicates")

```

### Adjusting Similarity Sensitivity

```python
from skills.watch.scripts.frames import dedupe_perceptual, DEDUP_THRESHOLD

# Raise threshold to retain more frames (less aggressive deduplication)

custom_threshold = 5.0
deduped, _ = dedupe_perceptual(frames, threshold=custom_threshold)

print(f"With threshold {custom_threshold}, {len(deduped)} frames remain")

```

Higher thresholds allow more visual variation before frames are considered duplicates—useful for videos with gradual lighting changes or slow pans.

## Why 16×16 Grayscale Works

The thumbnail approach trades spatial precision for computational efficiency:

| Property | Benefit |
|----------|---------|
| 16×16 resolution | 256 comparisons per frame pair—fast in pure Python |
| Grayscale | Eliminates color space complexity; luminance captures structural similarity |
| Raw bytes | No PIL, OpenCV, or numpy dependencies |
| Mean absolute difference | Simple, interpretable metric with clear threshold semantics |

The algorithm specifically targets **shot-level deduplication** rather than semantic similarity. Two frames from different camera angles may appear similar as thumbnails, but the conservative threshold and chronological comparison prevent incorrect merging.

## Key Source Files

| Path | Purpose |
|------|---------|
| [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) | Core implementation: `DEDUP_THUMB`, `DEDUP_THRESHOLD`, `_thumb_frames`, `_frame_delta`, `dedupe_perceptual`, `_dedupe_by_deltas` |
| [`tests/test_dedup.py`](https://github.com/bradautomates/claude-video/blob/main/tests/test_dedup.py) | Unit tests validating threshold behavior and edge cases |
| [`skills/watch/SKILL.md`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/SKILL.md) | Skill definition invoking the frame pipeline |

## Summary

- **16×16 thumbnails** are generated via ffmpeg with `scale=16:16,format=gray` filtering
- **Mean absolute difference** compares thumbnails on a 0-255 scale with defensive length checking
- **Greedy chronological deduplication** keeps the first frame of each visually distinct segment
- **Configurable threshold** (`DEDUP_THRESHOLD = 2.0`) controls merge aggressiveness
- **Pure stdlib implementation** requires no external image processing libraries

## Frequently Asked Questions

### Why 16×16 pixels specifically?

The 16×16 size balances detection sensitivity with computational speed. At 256 bytes per thumbnail, even hour-long videos at 1 fps consume only ~900KB of thumbnail memory. Larger thumbnails improve discrimination but slow comparison; smaller thumbnails miss subtle scene changes. The source code hardcodes this at line 31 as `DEDUP_THUMB = 16`.

### How does the threshold value translate to visual similarity?

A `DEDUP_THRESHOLD` of 2.0 means the average pixel difference across all 256 grayscale values must be ≤ 2 intensity levels (on 0-255). For context: identical frames score 0.0, a single pixel shifted by 2 levels across the entire image scores 2.0, and completely inverted frames (0↔255) score 255.0. The default is intentionally conservative to preserve legitimate content variations.

### Can this algorithm detect duplicate frames out of order?

No. The `_dedupe_by_deltas` implementation only compares each frame to the **most recently kept** frame in chronological sequence. This design choice preserves temporal coherence and runs in O(n) time. For non-chronological duplicate detection, a different hashing approach would be required.

### What happens if ffmpeg produces malformed thumbnails?

The `_frame_delta` function includes explicit length validation. If `len(thumb_a) != len(thumb_b)`, it immediately returns `float("inf")`, ensuring the frames are never merged. This guards against truncated streams, decode errors, or filesystem corruption without crashing the pipeline.