How the bradautomates/claude-video Frame Deduplication Algorithm Works: A Technical Deep Dive

The bradautomates/claude-video frame deduplication algorithm eliminates visually redundant frames by converting extracted JPEGs to 16×16 grayscale thumbnails, computing mean absolute pixel differences, and applying a greedy drop-based filter that preserves only the first representative of each visual segment.

The deduplication system resides in the skills/watch/scripts/frames.py module of the bradautomates/claude-video repository. It provides a lightweight, pure-stdlib approach to perceptual hashing that removes near-duplicate frames from video extractions before they reach downstream language models.

Three-Stage Perceptual Deduplication Pipeline

The algorithm processes frames through three distinct phases, ensuring that only visually distinct shots survive while maintaining deterministic, reproducible results.

Stage 1: Thumbnail Generation with FFmpeg

The routine begins by creating tiny grayscale thumbnails from each extracted JPEG frame. The _thumb_frames helper invokes FFmpeg to downscale every image to a 16 × 16 pixel gray representation using the DEDUP_THUMB = 16 constant.

The raw thumbnail bytes are collected into a list for in-memory comparison. If FFmpeg fails or the byte count does not match the expected number of frames, the system implements a fail-open design: it returns the original frame list unchanged rather than crashing or corrupting data.

# From skills/watch/scripts/frames.py

DEDUP_THUMB = 16  # Size of grayscale thumbnail (pixels per side)

def _thumb_frames(frame_paths):
    # FFmpeg downscale to 16x16 grayscale, returns raw bytes

    # Fail-open: returns None on error, causing dedupe_perceptual 

    # to return original frames

    pass

Stage 2: Per-Pixel Mean Absolute Difference

For any candidate pair of thumbnails, the _frame_delta function calculates the average of |a - b| across all corresponding pixels. This mean absolute difference provides a normalized distance metric where higher values indicate greater visual disparity.

The implementation treats mismatched byte lengths as infinite distance, ensuring that corrupted frames never get collapsed into valid sequences.

def _frame_delta(thumb_a, thumb_b):
    """Calculate mean absolute difference between two thumbnails."""
    if len(thumb_a) != len(thumb_b):
        return float('inf')  # Never dedupe corrupted frames

    return sum(abs(a - b) for a, b in zip(thumb_a, thumb_b)) / len(thumb_a)

Stage 3: Greedy Temporal Filtering

The core deduplication logic lives in _dedupe_by_deltas, which implements a greedy keep-first strategy. The function walks the chronological candidate list and compares each frame against the last kept frame rather than against all previous frames.

• The first frame is always retained as the baseline.
• For each subsequent candidate, the algorithm compares its thumbnail delta to the last kept thumbnail.
• If the delta is ≤ DEDUP_THRESHOLD (default 2.0), the candidate is considered a near-duplicate, deleted from disk, and skipped.
• If the delta exceeds the threshold, the candidate becomes the new last and is retained.

This approach ensures that visually similar sequences do not produce chains of marginally different frames; only the first representative of each visual segment survives.

def _dedupe_by_deltas(frames_with_thumbs, threshold=2.0):
    survivors = []
    dropped = 0
    last = None
    
    for frame in frames_with_thumbs:
        if last is None:
            survivors.append(frame)
            last = frame
        else:
            delta = _frame_delta(frame['thumb'], last['thumb'])
            if delta <= threshold:
                # Drop near-duplicate

                Path(frame['path']).unlink()
                dropped += 1
            else:
                survivors.append(frame)
                last = frame
    return survivors, dropped

Public API and Integration

The public entry point dedupe_perceptual located in skills/watch/scripts/frames.py orchestrates the entire pipeline. It prepares thumbnails via _thumb_frames and delegates filtering to _dedupe_by_deltas.

The function returns a tuple (survivors, dropped_count) and is invoked by every extraction engine—scene-based, uniform sampling, and keyframe detection—unless the user specifies the --no-dedup CLI flag.

def dedupe_perceptual(candidate_frames, dedup=True):
    """
    Main entry point for frame deduplication.
    
    Returns:
        tuple: (surviving_frames, dropped_count)
    """
    if not dedup or not candidate_frames:
        return candidate_frames, 0
    
    thumbs = _thumb_frames([f['path'] for f in candidate_frames])
    if thumbs is None:
        return candidate_frames, 0  # Fail-open

    
    frames_with_thumbs = [
        {**f, 'thumb': t} for f, t in zip(candidate_frames, thumbs)
    ]
    return _dedupe_by_deltas(frames_with_thumbs, DEDUP_THRESHOLD)

Configuration Parameters

The algorithm exposes two primary tunable constants:

• DEDUP_THUMB (default: 16): Defines the pixel dimensions of the grayscale thumbnail. Larger values increase precision but require more memory and computation.
• DEDUP_THRESHOLD (default: 2.0): Sets the maximum mean per-pixel difference that qualifies frames as duplicates. Lower values are stricter (fewer drops), while higher values aggressively collapse frames.

The --no-dedup CLI flag disables the entire routine, passing dedup=False to dedupe_perceptual and bypassing thumbnail generation entirely.

Practical Usage Examples

Run perceptual deduplication programmatically on extracted frame candidates:

from skills.watch.scripts import frames

candidates = [
    {"index": 0, "timestamp_seconds": 0.0, "path": "frame_0000.jpg"},
    {"index": 1, "timestamp_seconds": 1.0, "path": "frame_0001.jpg"},
    {"index": 2, "timestamp_seconds": 2.0, "path": "frame_0002.jpg"},
]

kept_frames, dropped_count = frames.dedupe_perceptual(candidates)
print(f"Kept {len(kept_frames)} frames, dropped {dropped_count} near-duplicates")

Disable deduplication via the command line when processing video:

python -m skills.watch.scripts.frames video.mp4 output_dir --fps 4 --no-dedup

Summary

• The bradautomates/claude-video frame deduplication algorithm lives in skills/watch/scripts/frames.py and uses a three-stage perceptual hashing approach.
• Stage 1 generates 16×16 grayscale thumbnails using FFmpeg, implementing a fail-open policy that returns original frames on processing errors.
• Stage 2 computes mean absolute pixel differences via _frame_delta, treating corrupted data as infinitely distant to prevent invalid collapses.
• Stage 3 applies a greedy keep-first filter (_dedupe_by_deltas) that compares each candidate only against the last retained frame, dropping duplicates with delta ≤ 2.0.
• The public dedupe_perceptual function serves all extraction engines and can be disabled via the --no-dedup CLI flag for raw frame preservation.

Frequently Asked Questions

What makes the bradautomates/claude-video deduplication algorithm perceptual rather than cryptographic?

The algorithm uses perceptual hashing because it compares visual content via downsampled grayscale thumbnails rather than comparing cryptographic file hashes. Two frames with identical pixel content but different metadata would have the same cryptographic hash but different perceptual scores if their visual content diverged. Conversely, two visually similar frames with slightly different compression artifacts will have different MD5 hashes but perceptual deltas below the 2.0 threshold, allowing the algorithm to identify them as duplicates based on human visual perception.

Why does the algorithm use a greedy keep-first strategy instead of clustering?

The greedy approach in _dedupe_by_deltas compares each candidate only against the last kept frame rather than against all previous frames or using k-means clustering. This design choice ensures temporal coherence: if frame A is kept and frames B and C are both similar to A but distinct from each other, the greedy algorithm drops B (similar to A) and keeps C (distinct from A). This prevents chains of marginally different frames from accumulating while guaranteeing that the first representative of each visual segment always survives, which is optimal for video summarization tasks.

What happens when FFmpeg fails during thumbnail generation?

The _thumb_frames function implements a fail-open safety mechanism. If FFmpeg returns a non-zero exit code or produces an unexpected byte count, the function returns None, causing dedupe_perceptual to immediately return the original frame list with zero drops. This ensures that extraction pipeline failures never result in data loss; the system degrades gracefully by presenting all frames rather than crashing or returning partial results.

How does the DEDUP_THRESHOLD parameter affect frame retention?

The DEDUP_THRESHOLD constant (default 2.0) defines the maximum mean absolute pixel difference that still qualifies as a duplicate. With a threshold of 2.0, a frame whose thumbnail pixels differ from the last kept frame by an average of 2.0 intensity values or less is deleted. Lowering this value to 1.0 or 0.5 makes the algorithm stricter, preserving more frames with subtle differences, while raising it to 5.0 or 10.0 aggressively collapses frames and may eliminate intentional gradual transitions or fades.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}