# Near-Duplicate Frame Detection Using Grayscale Thumbnails in Claude-Video

> Discover Claude-Videos near duplicate frame detection algorithm. Learn how 16x16 grayscale thumbnails and mean-absolute-difference efficiently identify and discard redundant frames.

- Repository: [bradautomates/claude-video](https://github.com/bradautomates/claude-video)
- Tags: how-to-guide
- Published: 2026-07-31

---

**Claude-Video eliminates visually redundant frames by converting each JPEG into a 16×16 grayscale thumbnail and comparing consecutive images with a mean-absolute-difference metric, discarding frames with a per-pixel delta of 2.0 or less.**

The `bradautomates/claude-video` repository provides a computationally efficient method for near-duplicate frame detection using grayscale thumbnails to clean video frame sequences before transcription or LLM analysis. This algorithm processes extracted JPEG images through a three-stage pipeline that minimizes memory usage while preserving visual fidelity, ensuring downstream agents receive only information-rich snapshots.

## How the Algorithm Works

### Stage 1: Generating 16×16 Grayscale Thumbnails

The deduplication process begins in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py) where the `_thumb_frames()` function (lines 24-31) constructs an FFmpeg pipeline. This pipeline reads the sequence of candidate JPEG files, scales each image to `DEDUP_THUMB = 16` pixels in both dimensions, forces a gray pixel format to eliminate color channel noise, and outputs raw video data as a sequence of byte buffers.

Each thumbnail occupies exactly 256 bytes (16 × 16 pixels), creating a compact representation that retains sufficient structural information to distinguish between static slides, fade transitions, and repeated screen captures while remaining computationally trivial to process.

### Stage 2: Computing Mean Absolute Per-Pixel Difference

Once thumbnails are generated, the algorithm measures visual similarity using `_frame_delta()` (lines 15-22 in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py)). This function calculates the **mean absolute per-pixel difference** between two thumbnail byte buffers by summing the absolute difference of every corresponding byte and dividing by the total pixel count (256).

The metric returns a value in the range 0–255, where 0 indicates identical frames. If the input buffers differ in length—indicating a processing error or corrupted frame—the function returns `infinity` to prevent accidental merging of dissimilar content.

### Stage 3: Greedy Sequential Deduplication

The public entry point `dedupe_perceptual()` (lines 64-71) orchestrates the final filtering stage by invoking `_dedupe_by_deltas()` (lines 80-88). This implements a **greedy sequential scan** that processes frames in chronological order:

1. The first frame is automatically retained as the initial reference.
2. Each subsequent frame's thumbnail is compared against the *last kept* frame using the mean absolute difference metric.
3. If the delta is **≤ 2.0** (`DEDUP_THRESHOLD`), the frame is considered near-identical and deleted from disk.
4. If the delta exceeds 2.0, the frame is retained and becomes the new reference for subsequent comparisons.

After completion, surviving frames are re-indexed sequentially and deleted JPEG files are permanently unlinked, preventing downstream steps from processing redundant data.

## Configuration Constants and Threshold Tuning

The algorithm relies on two critical constants defined in [`skills/watch/scripts/frames.py`](https://github.com/bradautomates/claude-video/blob/main/skills/watch/scripts/frames.py):

- **`DEDUP_THUMB = 16`**: Determines the thumbnail resolution. The 16×16 size provides enough granularity to detect meaningful visual changes while keeping memory footprint minimal.
- **`DEDUP_THRESHOLD = 2.0`**: Represents a conservative threshold of approximately 0.8% of the maximum pixel intensity (255). This ensures only frames that differ by less than two intensity levels on average are collapsed, preventing the removal of subtle but meaningful changes such as scrolling code or appearing slide bullets.

If FFmpeg fails to generate thumbnails (e.g., due to corrupted input files), `dedupe_perceptual()` gracefully degrades to a no-op, returning all frames unmodified to prevent data loss.

## Code Examples

### Direct Deduplication of Extracted Frames

```python
from pathlib import Path
from skills.watch.scripts.frames import extract, dedupe_perceptual

# Extract candidate frames using uniform sampling

candidates = extract(
    video_path="example.mp4",
    out_dir=Path("tmp/frames"),
    fps=1.0,
    resolution=512,
    max_frames=200,
)

# Remove near-duplicates using grayscale thumbnail comparison

unique_frames, dropped = dedupe_perceptual(candidates)

print(f"Kept {len(unique_frames)} frames, dropped {dropped} near-duplicates.")

```

### Integrated Scene-Engine Workflow

```python
from pathlib import Path
from skills.watch.scripts.frames import extract_scene_or_uniform

frames, meta = extract_scene_or_uniform(
    video_path="lecture.mov",
    out_dir=Path("tmp/out"),
    fps=2.0,
    target_frames=120,
    resolution=512,
    max_frames=100,
    dedup=True,          # Enables grayscale-thumbnail deduplication

)

print(meta)   # Engine type, candidate count, deduped count, etc.

```

## Summary

- **Grayscale thumbnail generation**: `_thumb_frames()` uses FFmpeg to create 16×16 pixel representations, reducing each frame to 256 bytes of luminance data.
- **Deterministic comparison**: `_frame_delta()` computes mean absolute per-pixel difference (0–255 range), returning `inf` for buffer mismatches to guarantee safety.
- **Greedy filtering**: `dedupe_perceptual()` applies a conservative threshold of 2.0 to identify near-identical consecutive frames, mimicking human perception of continuous shots.
- **Automatic cleanup**: The algorithm physically deletes redundant JPEG files and re-indexes survivors, ensuring downstream transcription agents process only unique visual content.
- **Fault tolerance**: If thumbnail generation fails, the system preserves all frames rather than risking data loss.

## Frequently Asked Questions

### What happens if FFmpeg fails during thumbnail generation?

If the `_thumb_frames()` function cannot generate thumbnails due to FFmpeg errors or corrupted JPEG files, `dedupe_perceptual()` automatically becomes a no-op. The function returns all candidate frames unmodified, ensuring that temporary processing failures never result in the loss of potentially unique visual data.

### Why does the algorithm use 16×16 resolution specifically?

The 16×16 dimension (`DEDUP_THUMB = 16`) strikes a balance between computational efficiency and discriminative power. At 256 pixels total, the thumbnails are small enough to process thousands of frames rapidly using simple byte arithmetic, yet large enough to retain structural information needed to distinguish between static slides, fade transitions, and minor UI updates.

### How does the threshold of 2.0 prevent accidental deletion of meaningful frames?

The `DEDUP_THRESHOLD = 2.0` represents less than 1% of the maximum pixel intensity range (0–255). This conservative value ensures that frames must be nearly indistinguishable—differing by an average of less than two intensity levels across all pixels—before being flagged as duplicates. This prevents the removal of subtle but meaningful changes such as cursor movements, text highlighting, or slowly appearing bullet points in presentations.

### Can the algorithm detect duplicate frames that are not consecutive in the video sequence?

No, the current implementation uses a greedy sequential scan that compares each frame only against the *last kept* frame. This design choice reflects the typical structure of video content where duplicates usually appear in contiguous bursts (static shots, freeze frames). Non-consecutive duplicates require global comparison algorithms, which would significantly increase computational complexity and memory requirements beyond the lightweight scope of this tool.