How Frame Deduplication Works in bradautomates/claude-video: A 3-Stage Algorithm

Frame deduplication uses a perceptual hashing pipeline that converts extracted frames to 16×16 grayscale thumbnails, computes mean absolute pixel differences, and applies a greedy filter to drop near-duplicates while preserving only visually distinct shots.

The bradautomates/claude-video repository provides a lightweight, pure-standard-library video processing toolkit designed to extract meaningful frames for language model consumption. At its core, the frame deduplication algorithm lives in skills/watch/scripts/frames.py and prevents redundant visual data from overwhelming downstream AI systems by removing perceptually similar frames while guaranteeing scene diversity through a deterministic three-stage process.

Stage 1: Thumbnail Generation with FFmpeg

The deduplication process begins in the _thumb_frames helper function, which converts each extracted JPEG into a tiny grayscale representation.

• Each frame is down-scaled to a 16×16 pixel grayscale image using the constant DEDUP_THUMB = 16
• FFmpeg handles the resizing operation, outputting raw byte arrays for each thumbnail
• The raw bytes of all thumbnails are collected into a list for in-memory processing

This design keeps the algorithm fail-open: if FFmpeg exits with an error or the resulting byte count does not match the number of input frames, the routine immediately returns the original frame list unchanged rather than crashing or corrupting data.

Stage 2: Per-Pixel Delta Calculation

Once thumbnails are generated, the algorithm calculates visual similarity using the _frame_delta function.

For any two thumbnail byte arrays, the function computes the mean absolute difference across all pixels:

# Conceptual implementation from skills/watch/scripts/frames.py

def _frame_delta(a, b):
    if len(a) != len(b):
        return float('inf')  # Corrupted frames are treated as maximally different

    return sum(abs(x - y) for x, y in zip(a, b)) / len(a)

This approach treats mismatched lengths as infinite distance, ensuring that corrupted or truncated frames are never collapsed into their neighbors. The calculation remains lightweight by operating on raw byte values without requiring heavy image processing libraries.

Stage 3: Greedy Filtering by Visual Similarity

The final stage occurs in _dedupe_by_deltas, which implements a greedy chronological filter to determine which frames survive:

1. Always keep the first frame as the initial reference
2. Compare each candidate to the last kept frame (not the immediately preceding frame)
3. Drop duplicates: If the mean absolute difference is ≤ DEDUP_THRESHOLD (default 2.0), the candidate is deleted and skipped
4. Keep distinct frames: If the difference exceeds the threshold, the candidate becomes the new reference and survives

This "keep-first, drop-near-duplicates" strategy ensures that a chain of similar frames does not result in multiple retained images—only the first representative of a visual segment survives. The threshold uses an inclusive comparison (≤), meaning frames exactly at the 2.0 difference limit are treated as duplicates.

The public entry point dedupe_perceptual orchestrates these stages and returns a tuple (survivors, dropped_count), which extraction engines use to report statistics. Dropped JPEG files are physically deleted from disk, and surviving frames are re-indexed to maintain sequential order.

Key Parameters and Configuration

Parameter	Default	Description
DEDUP_THUMB	16	Pixel dimensions of the grayscale thumbnail (16×16)
DEDUP_THRESHOLD	2.0	Maximum mean per-pixel difference that still counts as a duplicate
--no-dedup flag	False	CLI flag to bypass deduplication entirely (sets dedup=False)

Implementation Examples

Run deduplication programmatically on a list of frame candidates:

from skills.watch.scripts import frames

candidates = [
    {"index": 0, "timestamp_seconds": 0.0, "path": "frame_0000.jpg"},
    {"index": 1, "timestamp_seconds": 1.0, "path": "frame_0001.jpg"},
    {"index": 2, "timestamp_seconds": 2.0, "path": "frame_0002.jpg"},
]

# Returns (survivors, dropped_count)

kept_frames, dropped = frames.dedupe_perceptual(candidates)
print(f"Retained {len(kept_frames)} frames, removed {dropped} near-duplicates")

Disable deduplication via the command line when running the extraction pipeline:

python -m skills.watch.scripts.frames video.mp4 output_dir --fps 4 --no-dedup

Summary

• Frame deduplication in bradautomates/claude-video operates entirely within skills/watch/scripts/frames.py using pure Python standard library functions.
• The algorithm converts frames to 16×16 grayscale thumbnails to minimize computational overhead while preserving luma-based visual distinctions.
• Mean absolute difference calculations identify perceptually similar frames, with corrupted data treated as infinitely different to prevent false collapses.
• A greedy filtering mechanism compares each candidate against the last kept frame, ensuring only the first representative of similar visual sequences survives.
• The default threshold of 2.0 and inclusive comparison (≤) provide deterministic, conservative deduplication suitable for presentation slides and static scene detection.
• Fail-open behavior ensures that FFmpeg failures result in the original frame set being returned unchanged rather than data loss.

Frequently Asked Questions

What happens if FFmpeg fails during thumbnail generation?

According to the source code in skills/watch/scripts/frames.py, the deduplication routine implements a fail-open design. If the FFmpeg subprocess returns a non-zero exit code or the resulting thumbnail byte count does not match the expected number of frames, the function immediately returns the original frame list unchanged. This prevents crashes or data corruption when processing malformed video files.

Why does the algorithm compare against the last kept frame instead of the previous frame?

The greedy comparison against the last kept frame (implemented in _dedupe_by_deltas) prevents a chain of similar frames from all being retained. If the algorithm compared only against the immediately preceding frame, a slow transition or fade might result in every intermediate frame being kept because each differs slightly from its predecessor. By comparing against the last survivor, the algorithm ensures that only the first representative of a visual segment is retained, collapsing entire similar sequences into a single frame.

How does the algorithm handle corrupted or mismatched frame data?

The _frame_delta function treats any length mismatch between two thumbnail byte arrays as infinite distance. This means corrupted frames, truncated downloads, or metadata errors cause the frame to be considered maximally different from its neighbors, guaranteeing they are never collapsed as duplicates. This defensive coding ensures visual integrity even when input data is unreliable.

Can I adjust the similarity threshold for stricter or looser deduplication?

Yes, by modifying the DEDUP_THRESHOLD constant in skills/watch/scripts/frames.py (default 2.0). Lower values (e.g., 1.0) require pixels to be nearly identical to trigger deduplication, resulting in fewer drops and more frames retained. Higher values (e.g., 5.0) treat visually similar but not identical frames as duplicates, aggressively reducing the frame count. The threshold represents the mean absolute difference per pixel on a 0-255 scale.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}