How Claude Video's Frame Deduplication Algorithm Works: Perceptual Hashing for Video Processing

Claude Video removes near-duplicate frames by downscaling images to 16×16 grayscale thumbnails and calculating mean absolute differences against the last kept frame, preserving only frames that exceed a configurable similarity threshold.

The bradautomates/claude-video repository implements an efficient perceptual deduplication system that preprocesses video content before the frame budget cap is applied. This Claude Video frame deduplication algorithm ensures that only visually distinct frames are sent to the Claude API, significantly reducing token costs for videos containing static segments, slide decks, or paused recordings.

Core Algorithm Components

The deduplication pass operates as a lightweight preprocessing step that trades minimal CPU overhead for substantial reductions in redundant data.

Thumbnail Generation and Normalization

Every extracted frame undergoes aggressive downscaling to enable rapid comparison. In skills/watch/scripts/frames.py, the thumb_image helper (around line 440) generates a square thumbnail of DEDUP_THUMB pixels—defaulting to 16×16—and converts the image to 8-bit grayscale. This normalization drastically reduces the data footprint while preserving the essential visual "shape" required for similarity detection.

Per-Pixel Distance Calculation

For each candidate frame, the algorithm computes the mean absolute difference between the current frame's thumbnail and the thumbnail of the last kept frame. This produces a simple scalar distance value that quantifies visual dissimilarity without requiring complex perceptual hashing algorithms.

Greedy Last-Kept Comparison Logic

The core selection logic resides in the private function _dedupe_by_deltas (defined at line 479 of skills/watch/scripts/frames.py). This function implements a greedy filtering strategy:

• A frame is kept only if its distance exceeds the configurable threshold (default 2.0)
• Comparison is always performed against the last kept frame rather than the immediate predecessor
• This design collapses entire runs of identical or near-identical frames into a single representative image

The function returns two values: a list of surviving frame timestamps and the total count of dropped frames.

Pipeline Integration and CLI Control

The deduplication step integrates seamlessly with the main extraction workflow while remaining optional for specialized use cases.

Invocation and Flags

The deduplication pass is invoked by the main extraction pipeline through the sequence extract_frames → dedupe_perceptual. Control flags are parsed in skills/watch/scripts/watch.py (see lines 212 and 224), where the --no-dedup flag allows users to bypass deduplication entirely when every sampled frame is required for fine-grained motion analysis.

Metadata Tracking

After processing, the pipeline records the effectiveness of the deduplication pass by adding deduped_count to the output metadata (around line 549 of watch.py). This metric allows users to verify how many redundant frames were eliminated before the frame budget was applied.

Practical Usage Examples

Run the watch skill with default deduplication enabled (recommended for most content):

python -m skills.watch.scripts.watch https://www.youtube.com/watch?v=example

Disable deduplication when processing content requiring frame-perfect analysis:

python -m skills.watch.scripts.watch https://www.youtube.com/watch?v=example --no-dedup

Call the deduplication routine directly in custom pipelines:

from skills.watch.scripts import frames

# candidates: list of (timestamp, path) tuples from the extractor

# thumbs: list of 16×16 greyscale thumbnails for those frames

survivors, dropped = frames._dedupe_by_deltas(
    candidates, 
    thumbs, 
    threshold=2.0
)

print(f"Kept {len(survivors)} frames, dropped {dropped} near-duplicates")

Performance Impact and Budget Optimization

Because the Claude Video frame deduplication algorithm runs before the --max-frames cap is enforced, the remaining budget is spent exclusively on distinct visual content. This architecture delivers particular value for:

• Screen recordings with static UI elements
• Slide deck presentations with extended pauses on individual slides
• Paused or buffering video segments where content remains unchanged

The computational cost is negligible—a single downscale operation and mean-difference calculation per frame—while the savings in API tokens and processing time can be substantial for content with high temporal redundancy.

Summary

• Thumbnail preprocessing: Frames are downscaled to 16×16 grayscale to enable rapid comparison while preserving visual structure.
• Similarity metric: Mean absolute difference between thumbnails determines frame uniqueness.
• Greedy filtering: The _dedupe_by_deltas function in skills/watch/scripts/frames.py keeps frames only when they differ significantly from the last kept frame (threshold default 2.0).
• Optional execution: The --no-dedup flag in watch.py allows bypassing the filter when complete frame sequences are required.
• Budget efficiency: Deduplication occurs before the --max-frames limit, ensuring the frame budget consumes only visually distinct content.

Frequently Asked Questions

What is the default similarity threshold for frame deduplication?

The default threshold is 2.0, representing the mean absolute difference between 16×16 grayscale thumbnails. You can configure this value when calling frames._dedupe_by_deltas() directly, though the CLI uses this default consistently.

How do I disable frame deduplication when processing a video?

Pass the --no-dedup flag when invoking the watch skill. This flag is parsed in skills/watch/scripts/watch.py (lines 212 and 224) and causes the pipeline to skip the dedupe_perceptual step entirely, preserving every sampled frame for analysis.

Why does the algorithm compare against only the last kept frame instead of all previous frames?

Comparing against the last kept frame (rather than the immediate predecessor or all previous frames) creates a greedy collapsing behavior. When a sequence of near-identical frames appears, this approach selects the first distinct frame and drops all subsequent similar frames until a significant change occurs. This is more computationally efficient than pairwise comparison against all history while still effectively eliminating redundant static sequences.

Does deduplication affect the maximum frame budget calculation?

Yes, deduplication runs before the --max-frames budget cap is applied. Because duplicate frames are removed prior to budget enforcement, the final frame count sent to Claude represents only visually distinct content. This means a video with 1000 extracted frames but only 50 distinct visual states will consume only 50 frames from your budget after deduplication.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}