Near-Duplicate Frame Detection Using Grayscale Thumbnails in Claude-Video

Claude-Video eliminates visually redundant frames by converting each JPEG into a 16×16 grayscale thumbnail and comparing consecutive images with a mean-absolute-difference metric, discarding frames with a per-pixel delta of 2.0 or less.

The bradautomates/claude-video repository provides a computationally efficient method for near-duplicate frame detection using grayscale thumbnails to clean video frame sequences before transcription or LLM analysis. This algorithm processes extracted JPEG images through a three-stage pipeline that minimizes memory usage while preserving visual fidelity, ensuring downstream agents receive only information-rich snapshots.

How the Algorithm Works

Stage 1: Generating 16×16 Grayscale Thumbnails

The deduplication process begins in skills/watch/scripts/frames.py where the _thumb_frames() function (lines 24-31) constructs an FFmpeg pipeline. This pipeline reads the sequence of candidate JPEG files, scales each image to DEDUP_THUMB = 16 pixels in both dimensions, forces a gray pixel format to eliminate color channel noise, and outputs raw video data as a sequence of byte buffers.

Each thumbnail occupies exactly 256 bytes (16 × 16 pixels), creating a compact representation that retains sufficient structural information to distinguish between static slides, fade transitions, and repeated screen captures while remaining computationally trivial to process.

Stage 2: Computing Mean Absolute Per-Pixel Difference

Once thumbnails are generated, the algorithm measures visual similarity using _frame_delta() (lines 15-22 in skills/watch/scripts/frames.py). This function calculates the mean absolute per-pixel difference between two thumbnail byte buffers by summing the absolute difference of every corresponding byte and dividing by the total pixel count (256).

The metric returns a value in the range 0–255, where 0 indicates identical frames. If the input buffers differ in length—indicating a processing error or corrupted frame—the function returns infinity to prevent accidental merging of dissimilar content.

Stage 3: Greedy Sequential Deduplication

The public entry point dedupe_perceptual() (lines 64-71) orchestrates the final filtering stage by invoking _dedupe_by_deltas() (lines 80-88). This implements a greedy sequential scan that processes frames in chronological order:

1. The first frame is automatically retained as the initial reference.
2. Each subsequent frame's thumbnail is compared against the last kept frame using the mean absolute difference metric.
3. If the delta is ≤ 2.0 (DEDUP_THRESHOLD), the frame is considered near-identical and deleted from disk.
4. If the delta exceeds 2.0, the frame is retained and becomes the new reference for subsequent comparisons.

After completion, surviving frames are re-indexed sequentially and deleted JPEG files are permanently unlinked, preventing downstream steps from processing redundant data.

Configuration Constants and Threshold Tuning

The algorithm relies on two critical constants defined in skills/watch/scripts/frames.py:

• DEDUP_THUMB = 16: Determines the thumbnail resolution. The 16×16 size provides enough granularity to detect meaningful visual changes while keeping memory footprint minimal.
• DEDUP_THRESHOLD = 2.0: Represents a conservative threshold of approximately 0.8% of the maximum pixel intensity (255). This ensures only frames that differ by less than two intensity levels on average are collapsed, preventing the removal of subtle but meaningful changes such as scrolling code or appearing slide bullets.

If FFmpeg fails to generate thumbnails (e.g., due to corrupted input files), dedupe_perceptual() gracefully degrades to a no-op, returning all frames unmodified to prevent data loss.

Code Examples

Direct Deduplication of Extracted Frames

from pathlib import Path
from skills.watch.scripts.frames import extract, dedupe_perceptual

# Extract candidate frames using uniform sampling

candidates = extract(
    video_path="example.mp4",
    out_dir=Path("tmp/frames"),
    fps=1.0,
    resolution=512,
    max_frames=200,
)

# Remove near-duplicates using grayscale thumbnail comparison

unique_frames, dropped = dedupe_perceptual(candidates)

print(f"Kept {len(unique_frames)} frames, dropped {dropped} near-duplicates.")

Integrated Scene-Engine Workflow

from pathlib import Path
from skills.watch.scripts.frames import extract_scene_or_uniform

frames, meta = extract_scene_or_uniform(
    video_path="lecture.mov",
    out_dir=Path("tmp/out"),
    fps=2.0,
    target_frames=120,
    resolution=512,
    max_frames=100,
    dedup=True,          # Enables grayscale-thumbnail deduplication

)

print(meta)   # Engine type, candidate count, deduped count, etc.

Summary

• Grayscale thumbnail generation: _thumb_frames() uses FFmpeg to create 16×16 pixel representations, reducing each frame to 256 bytes of luminance data.
• Deterministic comparison: _frame_delta() computes mean absolute per-pixel difference (0–255 range), returning inf for buffer mismatches to guarantee safety.
• Greedy filtering: dedupe_perceptual() applies a conservative threshold of 2.0 to identify near-identical consecutive frames, mimicking human perception of continuous shots.
• Automatic cleanup: The algorithm physically deletes redundant JPEG files and re-indexes survivors, ensuring downstream transcription agents process only unique visual content.
• Fault tolerance: If thumbnail generation fails, the system preserves all frames rather than risking data loss.

Frequently Asked Questions

What happens if FFmpeg fails during thumbnail generation?

If the _thumb_frames() function cannot generate thumbnails due to FFmpeg errors or corrupted JPEG files, dedupe_perceptual() automatically becomes a no-op. The function returns all candidate frames unmodified, ensuring that temporary processing failures never result in the loss of potentially unique visual data.

Why does the algorithm use 16×16 resolution specifically?

The 16×16 dimension (DEDUP_THUMB = 16) strikes a balance between computational efficiency and discriminative power. At 256 pixels total, the thumbnails are small enough to process thousands of frames rapidly using simple byte arithmetic, yet large enough to retain structural information needed to distinguish between static slides, fade transitions, and minor UI updates.

How does the threshold of 2.0 prevent accidental deletion of meaningful frames?

The DEDUP_THRESHOLD = 2.0 represents less than 1% of the maximum pixel intensity range (0–255). This conservative value ensures that frames must be nearly indistinguishable—differing by an average of less than two intensity levels across all pixels—before being flagged as duplicates. This prevents the removal of subtle but meaningful changes such as cursor movements, text highlighting, or slowly appearing bullet points in presentations.

Can the algorithm detect duplicate frames that are not consecutive in the video sequence?

No, the current implementation uses a greedy sequential scan that compares each frame only against the last kept frame. This design choice reflects the typical structure of video content where duplicates usually appear in contiguous bursts (static shots, freeze frames). Non-consecutive duplicates require global comparison algorithms, which would significantly increase computational complexity and memory requirements beyond the lightweight scope of this tool.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}