How Claude Video's Frame Deduplication Algorithm Works: Implementation Guide

Claude Video removes near-duplicate frames before applying the frame-budget cap by generating 16×16 greyscale thumbnails, computing mean absolute differences, and greedily filtering against the last kept frame using a configurable threshold.

The Claude Video frame deduplication algorithm is a lightweight perceptual filter implemented in the bradautomates/claude-video repository that eliminates redundant visual data before sending frames to Claude. This preprocessing step ensures that static screen recordings, paused video segments, and slide decks do not waste the user-specified frame budget on identical content. By comparing downscaled thumbnails rather than full-resolution images, the system trades minimal CPU overhead for significant cost savings and improved relevance in downstream AI analysis.

The Perceptual Deduplication Pipeline

The deduplication pass operates as a three-stage pipeline that processes extracted frames before the final budget cap is applied.

Generating 16×16 Greyscale Thumbnails

Every extracted frame is downscaled to a square of DEDUP_THUMB pixels (default 16 × 16) and converted to 8-bit greyscale. This aggressive reduction preserves the visual "shape" of the image while drastically reducing the data that must be compared. In skills/watch/scripts/frames.py, the thumb_image helper implements this transformation around line 440, creating a compact representation suitable for fast pixel-wise comparison.

Computing Mean Absolute Difference

For each candidate frame, the algorithm calculates the mean absolute difference between its thumbnail and the thumbnail of the last kept frame. This scalar distance metric reflects the visual similarity between two frames, where a value of zero indicates identical pixel data. The calculation compares corresponding pixels in the two 16×16 greyscale arrays, summing the absolute differences and dividing by the total pixel count.

Greedy Last-Kept Comparison Logic

The frame survives the filter only if its distance exceeds a configurable threshold (default 2.0). Because the comparison is always against the last kept frame rather than the immediately preceding frame, the algorithm collapses entire runs of identical or near-identical content into a single representative image. This greedy approach is implemented in the private function _dedupe_by_deltas defined at line 479 of skills/watch/scripts/frames.py, which returns a tuple containing the list of surviving frame timestamps and the count of dropped frames.

Source Code Implementation Details

The deduplication logic is tightly integrated into the frame extraction pipeline while remaining bypassable for specialized use cases.

Core Dedup Logic in frames.py

The dedupe_perceptual function (exposed through extract_frames) orchestrates the thumbnail generation and filtering process. The critical private helper _dedupe_by_deltas performs the actual greedy filtering loop, maintaining a reference to the last accepted thumbnail and comparing each subsequent candidate against it. When the mean absolute difference falls below the threshold, the candidate is discarded and the drop counter increments.

CLI Integration and Metadata Tracking

The entry point in skills/watch/scripts/watch.py exposes deduplication control through the --no-dedup flag. Lines 212 and 224 handle this argument, conditionally skipping the dedupe_perceptual call when the user requires every sampled frame. After processing, the system records the effectiveness of the filter in the output metadata; around line 549, the deduped_count field is populated to indicate how many redundant frames were removed from the stream.

Budget Application Order

Deduplication occurs before the --max-frames budget cap is enforced. This sequencing ensures that the frame quota is spent exclusively on distinct visual content rather than wasted on redundant data. For video containing long static segments, this ordering can reduce API costs and improve context window utilization by ensuring Claude receives only visually unique information.

CLI and Programmatic Usage

You can control the deduplication behavior through command-line arguments or by invoking the internal API directly.

Enabling Default Deduplication

Run the watch skill with default settings to automatically remove near-duplicates:

python -m skills.watch.scripts.watch https://www.youtube.com/watch?v=example

Disabling Deduplication for Motion Analysis

Pass the --no-dedup flag when processing video requiring frame-perfect granularity, such as fine-grained motion analysis or high-speed recordings where every micro-change matters:

python -m skills.watch.scripts.watch https://www.youtube.com/watch?v=example --no-dedup

Direct API Access

For custom pipelines, import the deduplication routine directly from frames.py:

from skills.watch.scripts import frames

# candidates: list of (timestamp, path) tuples from the extractor

# thumbs: list of 16×16 greyscale thumbnails corresponding to candidates

survivors, dropped = frames._dedupe_by_deltas(candidates, thumbs, threshold=2.0)

print(f"Kept {len(survivors)} frames, dropped {dropped} near-duplicates")

Frame Budget Optimization

The deduplication algorithm directly impacts cost efficiency by ensuring the --max-frames limit applies only to perceptually unique content. In skills/watch/scripts/watch.py, the pipeline architecture guarantees that the dedupe_perceptual step completes before the budget enforcement logic executes. This design is particularly effective for screen recordings containing static UI elements, presentation slides with minimal animation, or paused video segments, where the algorithm commonly achieves compression ratios exceeding 10:1 without loss of semantic information.

Summary

• Thumbnail preprocessing: All frames are downscaled to 16×16 greyscale images in skills/watch/scripts/frames.py to enable fast comparison.
• Distance metric: The algorithm uses mean absolute difference between thumbnails, with a default threshold of 2.0 determining deduplication boundaries.
• Greedy filtering: The _dedupe_by_deltas function implements last-kept comparison logic at line 479, collapsing runs of duplicates into single representative frames.
• CLI control: The --no-dedup flag in watch.py (lines 212 and 224) allows users to preserve every extracted frame when necessary.
• Budget efficiency: Deduplication occurs before the --max-frames cap, ensuring Claude receives only distinct visual content and maximizing the value of each API call.

Frequently Asked Questions

What is the default threshold for frame deduplication in Claude Video?

The default threshold is 2.0, representing the mean absolute difference between 16×16 greyscale thumbnails. Frames with a distance equal to or less than this value are considered near-duplicates and discarded, while frames exceeding this threshold are kept as visually distinct content.

How does the algorithm handle gradual transitions or slow motion?

Because the algorithm compares each frame against the last kept frame rather than the immediate predecessor, gradual transitions typically survive the filter until the cumulative visual change exceeds the threshold. Slow motion video will drop frames that are perceptually identical within the 16×16 greyscale space but retain frames where the mean absolute difference crosses the 2.0 boundary, preserving the temporal progression without redundant samples.

Can I adjust the thumbnail size for deduplication?

The thumbnail size is controlled by the DEDUP_THUMB constant in skills/watch/scripts/frames.py, which defaults to 16 pixels. While the source code allows modification of this value, changing it requires editing the constant definition and affects the granularity of the mean absolute difference calculation—larger thumbnails increase CPU overhead but may catch subtle visual differences that 16×16 sampling misses.

Does disabling deduplication affect the frame budget calculation?

Disabling deduplication with --no-dedup changes which frames are eligible for the budget cap but does not change the cap itself. When deduplication is active, the --max-frames limit applies only to the surviving unique frames; when disabled, the limit applies to every sampled frame extracted from the video, potentially including hundreds of identical or near-identical images that would otherwise be filtered out.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}