How Hydrus Visual Duplicate Detection Works: A Deep Dive into the Duplicate Comparison Algorithm

Hydrus determines visual similarity through a six-stage pipeline that combines early-exit checks, LAB color histogram analysis, and regional edge-map comparisons to assign confidence scores ranging from "near-perfect" to "probably" duplicates.

The duplicate comparison algorithm in the hydrusnetwork/hydrus repository powers one of the platform's most powerful features: automatically identifying visually similar images even when they differ in encoding, compression, or minor cropping. This analysis examines the implementation in hydrus/client/files/images/ClientVisualData.py to explain exactly how the system quantifies visual similarity.

The Six-Stage Visual Similarity Pipeline

According to the Hydrus source code, the duplicate comparison algorithm processes image pairs through a cascading series of tests, each increasing in computational cost but decreasing in false positives.

Stage 1: Early-Exit Sanity Checks

Before any expensive processing begins, FilesAreVisuallySimilarSimple performs rapid disqualification tests. If one image contains an alpha channel while the other does not, the pair is immediately marked as non-duplicates. Similarly, extreme aspect ratio mismatches or resolutions below configurable thresholds trigger immediate rejection. These checks prevent wasted CPU cycles on fundamentally incompatible files.

Stage 2: Global LAB Color Histogram Comparison

The algorithm converts both images to LAB color space and generates normalized histograms via GenerateImageVisualDataNumPy. It then computes a Wasserstein distance between the histograms using GetHistogramNormalisedWassersteinDistance.

• Distance ≈ 0–0.02: Classified as "near-perfect" visual duplicates
• Distance ≈ 0.02–0.05: Yields "almost certainly" or "very probably" confidence levels

This global color comparison catches identical files and those with minor re-encoding artifacts.

Stage 3: Regional Edge-Map Analysis

For more granular detection, the system downscales images and generates edge maps through GenerateEdgeMapNumPy. The FilesAreVisuallySimilarRegionalEdgeMap function splits these maps into a tiled grid using GenerateImageVisualDataTiledNumPy, then computes Wasserstein distances per tile. This regional approach detects subtle differences like slight color shifts, minor crops, or compression variations that global histograms might miss.

Stage 4: Tiled LAB Histogram Comparison

The most discriminating test occurs in FilesAreVisuallySimilarRegionalLabHistograms. By comparing LAB histograms across individual tiles rather than globally, this stage identifies local variations. The algorithm aggregates these regional distances to determine "very probably" and "almost certainly" confidence classifications with high precision.

Stage 5: Confidence Score Mapping

All distance metrics feed into a scoring system that maps results to five confidence constants defined in ClientVisualData.py (lines 221–224):

• NOT: Definitively different images
• NEAR-PERFECT: Identical or virtually identical files
• ALMOST-CERTAINLY: Extremely high confidence duplicates
• VERY-PROBABLY: High confidence with minor variations detected
• PROBABLY: Moderate confidence, may require manual review

The final output is a tuple: (is_duplicate: bool, confidence_constant, human_readable_message).

Stage 6: Integration with Auto-Resolution

The visual duplicate test is exposed through PairComparatorRelativeVisualDuplicates in hydrus/client/duplicates/ClientDuplicatesAutoResolutionComparators.py. This comparator integrates with ClientDuplicatesAutoResolution to enable batch processing and UI-driven duplicate management workflows.

How the Components Fit Together

The architecture separates data generation from comparison logic to enable efficient caching and reuse.

VisualData Generation

The ClientVisualData.GenerateImageVisualDataNumPy function creates lightweight VisualData objects containing:

• Image resolution metadata
• LAB color histograms (lab_histograms)
• Optional alpha channel histogram (alpha_hist)

Tiled Analysis Preparation

For regional comparisons, ClientVisualData.GenerateImageVisualDataTiledNumPy produces VisualDataTiled objects that additionally store tiled edge maps (edge_map) for granular spatial analysis.

Comparison Strategy

• Use FilesAreVisuallySimilarSimple for rapid filtering of obvious duplicates
• Employ FilesAreVisuallySimilarRegional* functions when detecting re-encoded JPEGs, watermarked variants, or slightly cropped versions
• Downstream systems consume the confidence constants to drive auto-resolution decisions

Practical Code Examples

Running Direct Visual Comparison

from hydrus.client.files.images import ClientVisualData

# Generate visual data for both images

vd1 = ClientVisualData.GenerateImageVisualDataNumPy(image_path_1)
vd2 = ClientVisualData.GenerateImageVisualDataNumPy(image_path_2)

# Execute simple comparison

is_dupe, confidence, note = ClientVisualData.FilesAreVisuallySimilarSimple(vd1, vd2)

print(f"Duplicate: {is_dupe}")
print(f"Confidence level: {confidence}")
print(f"Details: {note}")

Using the Auto-Resolution Comparator

from hydrus.client.duplicates import ClientDuplicatesAutoResolutionComparators as CDRC
from hydrus.client.files.images import ClientVisualData

# Initialize comparator with strict threshold

visual_comparator = CDRC.PairComparatorRelativeVisualDuplicates(
    acceptable_confidence=ClientVisualData.VISUAL_DUPLICATES_RESULT_ALMOST_CERTAINLY
)

# Compare two MediaResult objects

result = visual_comparator.ComparePair(media_a, media_b)

if result:
    print("Images classified as visual duplicates")

Executing the Tuning Suite

Hydrus includes a built-in diagnostic tool for validating threshold configurations:

# Access via Hydrus client console

self._RunVisualDuplicatesTuningSuite()

This suite exercises the same code paths used in production comparisons, outputting statistical validation of the current similarity thresholds.

Summary

• Early-exit checks in FilesAreVisuallySimilarSimple immediately reject incompatible pairs based on alpha channels or extreme dimension mismatches
• Global LAB histograms provide fast "near-perfect" detection through Wasserstein distance calculations in GetHistogramNormalisedWassersteinDistance
• Regional edge-map and histogram analysis via FilesAreVisuallySimilarRegionalEdgeMap and FilesAreVisuallySimilarRegionalLabHistograms catches subtle variations like re-encodes and minor crops
• Five-tier confidence system (NOT, NEAR-PERFECT, ALMOST-CERTAINLY, VERY-PROBABLY, PROBABLY) feeds into the auto-resolution engine through PairComparatorRelativeVisualDuplicates
• Core implementation resides in hydrus/client/files/images/ClientVisualData.py with integration points in hydrus/client/duplicates/ClientDuplicatesAutoResolutionComparators.py

Frequently Asked Questions

What file types does the Hydrus duplicate comparison algorithm support?

The algorithm primarily operates on bitmap representations, processing images through NumPy arrays after decoding. While the source code in ClientVisualData.py focuses on the visual analysis pipeline itself, Hydrus converts most common image formats (JPEG, PNG, GIF, WebP, BMP) to comparable arrays before the duplicate comparison executes. Video files and documents require different comparison strategies not covered by the visual similarity pipeline.

How does the Wasserstein distance calculation improve duplicate detection?

Unlike simple Euclidean distance or histogram intersection, the Wasserstein distance (implemented in GetHistogramNormalisedWassersteinDistance) measures the minimum "cost" to transform one histogram into another. This metric is particularly robust against color shifts and gamma corrections because it accounts for the ground distance between color bins, making it superior for detecting re-encoded images that maintain perceptual similarity despite technical differences.

Can users configure the sensitivity thresholds for visual duplicates?

Yes. The confidence constants in ClientVisualData.py (lines 221–224) define the classification boundaries, while PairComparatorRelativeVisualDuplicates accepts an acceptable_confidence parameter during instantiation. Administrators can set thresholds to require "almost certainly" classifications for automatic actions while queueing "very probably" matches for manual review, balancing automation accuracy against processing volume.

Why does the algorithm use both global and regional (tiled) comparisons?

Global histograms provide computationally efficient screening for obvious duplicates, while regional analysis handles the "long tail" of difficult cases. Tiled comparisons in FilesAreVisuallySimilarRegionalLabHistograms detect localized differences—such as cropped watermarks, inserted logos, or corner artifacts—that global statistics would average out and miss. This hybrid approach optimizes for both speed and accuracy across diverse duplicate scenarios.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}