How Claude Video Performs Frame Deduplication to Remove Similar Frames

Claude Video removes near-duplicate frames by converting images to 16×16 greyscale thumbnails and computing the mean absolute difference against the last kept frame, discarding any frame with a distance below the configurable threshold of 2.0.

Claude Video is an open-source video analysis tool that extracts frames for processing by AI models. To maximize efficiency and ensure the --max-frames budget is spent only on visually distinct content, the repository implements a lightweight frame deduplication algorithm that runs as a pre-processing step before the frame cap is applied.

How the Deduplication Algorithm Works

The deduplication pass operates as a greedy, single-pass filter that compares each new frame against the most recently kept frame. This approach collapses runs of identical or near-identical frames—common in screen recordings, paused video, or slide decks—into a single representative image.

Thumbnail Generation and Downsampling

Every extracted frame is first converted into a tiny 16×16 greyscale thumbnail defined by the DEDUP_THUMB constant. This down-scaling drastically reduces the data volume for comparison while preserving the essential visual "shape" of the image.

The thumbnail generation is handled by the thumb_image helper function located in skills/watch/scripts/frames.py around line 440. By converting to 8-bit greyscale during this step, the algorithm minimizes CPU overhead while maintaining sufficient fidelity for perceptual comparison.

Per-Pixel Distance Calculation

For each candidate frame, the algorithm computes the mean absolute difference between the current frame's thumbnail and the thumbnail of the last kept frame. This produces a simple scalar distance value that quantifies visual similarity.

The comparison is strictly against the last kept frame rather than all previous frames. This greedy approach ensures that gradual transitions are preserved while static sequences are aggressively compressed.

Threshold-Based Filtering

A frame is retained only if its computed distance exceeds the configurable threshold parameter, which defaults to 2.0. Frames falling below this threshold are discarded immediately.

This logic is encapsulated in the private function _dedupe_by_deltas, defined at line 479 of skills/watch/scripts/frames.py. The function accepts the candidate frame list, their corresponding thumbnails, and the threshold value, returning a tuple containing the list of surviving frame timestamps and the count of dropped frames.

Source Code Implementation

The deduplication pipeline is integrated into the main extraction workflow through the dedupe_perceptual function, called by extract_frames during processing.

Main Pipeline Integration

When the watch.py CLI entry point processes a video, it checks for the --no-dedup flag at lines 212 and 224. If deduplication is enabled (the default), the pipeline invokes the deduplication routine and records the results. The final count of removed frames is stored in the output metadata as deduped_count around line 549 of watch.py.

Direct API Usage

For custom pipelines, you can invoke the deduplication routine directly:

from skills.watch.scripts import frames

# candidates: list of (timestamp, path) tuples from the extractor

# thumbs: list of 16×16 greyscale thumbnails for those frames

survivors, dropped = frames._dedupe_by_deltas(
    candidates, 
    thumbs, 
    threshold=2.0
)

print(f"Kept {len(survivors)} frames, dropped {dropped} near-duplicates")

CLI Configuration and Usage

The deduplication behavior can be controlled via command-line flags when running the watch skill.

Running with default deduplication (recommended for most content):

python -m skills.watch.scripts.watch https://www.youtube.com/watch?v=example

Disabling deduplication for fine-grained motion analysis:

python -m skills.watch.scripts.watch https://www.youtube.com/watch?v=example --no-dedup

When --no-dedup is specified, the pipeline skips the _dedupe_by_deltas call entirely, preserving every sampled frame regardless of visual similarity.

Impact on Frame Budget and Performance

Because deduplication runs before the --max-frames cap is applied, the algorithm ensures that the frame budget is allocated exclusively to distinct visual content. This significantly improves cost efficiency for downstream Claude API calls by eliminating redundant images.

The performance overhead is minimal: the algorithm trades a small amount of CPU time for down-scaling and mean-difference calculations against a substantial reduction in redundant frame processing and API token consumption.

Summary

• Down-sampling: Frames are converted to 16×16 greyscale thumbnails in skills/watch/scripts/frames.py to enable fast comparison.
• Greedy filtering: The _dedupe_by_deltas function (line 479) compares each frame only against the last kept frame using mean absolute difference.
• Threshold control: The default threshold of 2.0 filters near-duplicates; this value is configurable when calling the function directly.
• CLI toggle: Use --no-dedup to bypass the filter; otherwise, deduplication runs automatically and reports dropped counts via deduped_count in the metadata.
• Budget optimization: Deduplication occurs before the --max-frames limit, ensuring distinct content receives priority.

Frequently Asked Questions

What is the default threshold for frame deduplication in Claude Video?

The default threshold is 2.0, defined as the mean absolute difference between 16×16 greyscale thumbnails. Frames with a distance below this value from the last kept frame are discarded. You can override this when calling _dedupe_by_deltas programmatically by passing a different threshold parameter.

Where is the frame deduplication logic implemented in the source code?

The core algorithm resides in skills/watch/scripts/frames.py at line 479 within the _dedupe_by_deltas function. The thumbnail generation helper thumb_image is located around line 440 in the same file. The CLI integration and --no-dedup flag handling are implemented in skills/watch/scripts/watch.py at lines 212 and 224.

How does disabling deduplication affect the frame budget?

When you pass the --no-dedup flag, the pipeline skips the _dedupe_by_deltas call entirely. This means identical or near-identical frames consume slots in your --max-frames budget, potentially reducing the diversity of visual content sent to Claude and increasing API costs without improving analysis quality.

Why does the algorithm compare against only the last kept frame rather than all previous frames?

The greedy "last kept" comparison ensures that sequences of similar frames collapse to a single representative image while preserving gradual visual transitions. Comparing against all previous frames would be computationally expensive (O(n²)) and could inadvertently remove legitimately distinct frames that happen to resemble earlier content in the video.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}