How claude-video Implements Frame Deduplication Using Mean Absolute Difference

claude-video removes visually duplicate frames before sending video to the LLM by comparing 16×16 grayscale thumbnails using mean absolute pixel difference, keeping only frames that differ by more than 2.0 average intensity units.

The claude-video repository optimizes video analysis workflows by eliminating redundant visual information before it reaches the language model. This frame deduplication mechanism using mean absolute difference ensures that static scenes, fade transitions, and duplicate keyframes do not waste tokens or context window space. The implementation relies on lightweight image processing operations that remain fast and dependency-free by leveraging only Python's standard library alongside ffmpeg.

The Three-Stage Deduplication Pipeline

The deduplication logic resides in skills/watch/scripts/frames.py and executes in three distinct phases to minimize computational overhead while preserving visual distinctiveness.

Stage 1: Generating 16×16 Grayscale Thumbnails

Each extracted JPEG is down-scaled to a 16 × 16 pixel grayscale image defined by the constant DEDUP_THUMB = 16 (declared at lines 31-38). This aggressive reduction keeps the comparison operation fast and memory-efficient while retaining enough structural detail to distinguish between flat slides, fade transitions, and static screens.

The thumbnail generation occurs in _thumb_frames (lines 24-33), which invokes a single ffmpeg pass to read raw grayscale data directly into memory without creating intermediate files.

Stage 2: Calculating Mean Absolute Pixel Difference

Similarity measurement happens in _frame_delta (lines 15-22). This function computes the mean absolute difference between corresponding pixels of two thumbnail arrays:

# Conceptual implementation from frames.py lines 15-22

def _frame_delta(thumb_a, thumb_b):
    # Both are 16x16 uint8 arrays

    diff = np.abs(thumb_a.astype(float) - thumb_b.astype(float))
    return diff.mean()

If the returned delta is ≤ DEDUP_THRESHOLD (default 2.0), the frames are classified as "near-identical" and marked for removal. This threshold is defined alongside DEDUP_THUMB at the top of the file (lines 31-38).

Stage 3: Greedy Chronological Filtering

The _dedupe_by_deltas function (lines 78-100) implements a greedy drop-while-keeping-last algorithm:

1. Keep the first frame as the initial reference.
2. Compare each subsequent frame's thumbnail to the last kept thumbnail.
3. If the delta exceeds the threshold, keep the frame and update the reference.
4. If the delta is within threshold, delete the JPEG file and skip the frame.

This approach ensures that the final sequence preserves the temporal structure while removing consecutive duplicates or extended static segments. Surviving frames are re-indexed to maintain sequential naming.

Implementation Details in skills/watch/scripts/frames.py

The public entry point dedupe_perceptual orchestrates the workflow by first calling _thumb_frames to generate thumbnails for all candidates, then forwarding them to _dedupe_by_deltas. If fewer than two candidates exist or thumbnail generation fails, the function returns the original set unchanged.

The constants controlling sensitivity are co-located at the module level:

# From skills/watch/scripts/frames.py lines 31-38

DEDUP_THUMB = 16        # Thumbnail dimensions (16x16)

DEDUP_THRESHOLD = 2.0   # Mean absolute difference cutoff

Using the dedupe_perceptual API

You can invoke deduplication programmatically after extracting frames:

from pathlib import Path
from skills.watch.scripts import frames

# Extract candidates using any engine (uniform, scene, keyframe)

candidates = frames.extract(
    video_path="example.mp4",
    out_dir=Path("out"),
    fps=1.0,
    max_frames=200,
)

# Run perceptual deduplication (mean absolute difference, threshold=2.0)

unique_frames, dropped = frames.dedupe_perceptual(candidates)

print(f"Kept {len(unique_frames)} frames, dropped {dropped} near-duplicates")

For command-line usage, the deduplication runs automatically unless disabled:

# CLI invocation (deduplication enabled by default)

python -m skills.watch.scripts.frames example.mp4 out --max-frames 200

# Disable deduplication explicitly

python -m skills.watch.scripts.frames example.mp4 out --max-frames 200 --no-dedup

Adjust the sensitivity by passing a custom threshold:

# Stricter deduplication (keeps fewer frames)

unique_frames, dropped = frames.dedupe_perceptual(candidates, threshold=1.5)

# Looser deduplication (tolerates more noise)

unique_frames, dropped = frames.dedupe_perceptual(candidates, threshold=5.0)

Integration with Video Processing Pipelines

The deduplication step is triggered automatically within every extraction engine—uniform, scene, and keyframe—unless the user passes the --no-dedup flag. After processing, skills/watch/scripts/watch.py exposes the number of removed frames as deduped_count in the metadata output (lines 293-298), allowing you to verify how aggressively the pipeline compressed the video.

Post-deduplication, the remaining frames may undergo further sampling (such as _even_sample) before being encoded for the LLM.

Summary

• claude-video collapses redundant frames using a mean absolute difference metric computed on 16×16 grayscale thumbnails.
• The _frame_delta function (lines 15-22) calculates average pixel differences, while _dedupe_by_deltas (lines 78-100) performs greedy chronological filtering against the most recently kept frame.
• The default DEDUP_THRESHOLD = 2.0 (defined at lines 31-38) balances sensitivity and compression.
• Access the functionality via dedupe_perceptual in skills/watch/scripts/frames.py or disable it with --no-dedup in the CLI.
• The pipeline reports the number of dropped frames as deduped_count in watch.py (lines 293-298).

Frequently Asked Questions

Why does claude-video use mean absolute difference instead of perceptual hashing?

Mean absolute difference provides deterministic, lightweight comparison that requires no additional dependencies beyond numpy and standard library operations. According to the source code in skills/watch/scripts/frames.py, this approach avoids the complexity of cryptographic or perceptual hashing libraries while remaining sufficiently accurate to detect static screens and fade transitions on low-resolution thumbnails.

What happens if thumbnail generation fails during deduplication?

If _thumb_frames (lines 24-33) fails to extract grayscale data from the input JPEGs, dedupe_perceptual returns the original candidate list unchanged and sets dropped to zero. This fail-safe ensures that extraction pipelines continue operating even when processing corrupted or unsupported image formats.

How does the greedy algorithm handle sudden scene changes?

The _dedupe_by_deltas function (lines 78-100) compares each candidate only to the last kept frame rather than the immediate predecessor. This means that after a scene change, the new distinct frame becomes the reference, and subsequent similar frames are compared against this new reference. This prevents false positives when transitioning between two different but static scenes.

Can I adjust the frame deduplication threshold for noisy video content?

Yes. Pass a custom threshold parameter to dedupe_perceptual. Values higher than the default 2.0 (such as 5.0) tolerate more pixel variation, useful for noisy or low-light footage, while values lower than 2.0 enforce stricter deduplication for high-fidelity content. The threshold represents the mean absolute intensity difference across the 16×16 thumbnail grid.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}