# How Semantica's DuplicateDetector Performs Semantic Entity Resolution at Scale

> Learn how Semantica's DuplicateDetector achieves scalable semantic entity resolution using semantic similarity blocking and union-find grouping for millions of entities.

- Repository: [Semantica /semantica](https://github.com/semantica-agi/semantica)
- Tags: how-to-guide
- Published: 2026-08-09

---

**Semantica resolves duplicate entities by combining semantic similarity scoring, blocking-based pruning, and union-find grouping to handle millions of entities efficiently.**

The `semantica-agi/semantica` repository implements a high-performance duplicate detection system designed for large-scale knowledge graphs. By leveraging multi-factor similarity metrics and intelligent blocking strategies, Semantica's `DuplicateDetector` enables accurate **semantic entity resolution** without the prohibitive costs of naive pairwise comparison.

## Blocking and Prefiltering: Reducing O(N²) Complexity

Before computing expensive semantic similarities, the system applies a **blocking strategy** to eliminate obviously unrelated pairs. In [`semantica/deduplication/similarity_calculator.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/similarity_calculator.py), the `SimilarityCalculator._prefilter_pair` method implements cheap prefix-based filtering that discards candidates based on type mismatch, name-length ratio, and token-overlap thresholds.

This prefiltering step dramatically reduces the search space from O(N²) to a manageable subset, allowing the pipeline to scale to millions of entities without comparing every possible pair.

## Multi-Factor Semantic Similarity Calculation

For candidate pairs that pass the blocking phase, the `SimilarityCalculator` aggregates four weighted similarity components into a composite score. The weights are configurable and automatically normalized.

### String and Embedding Similarity

The calculator measures **string similarity** using Levenshtein distance, Jaro-Winkler similarity, and cosine similarity on token vectors. It also computes **embedding similarity** as the cosine distance between entity vector embeddings, capturing semantic meaning beyond surface-level text.

### Property and Relationship Matching

Beyond names, the system evaluates **property similarity** by comparing matching property keys and values, and **relationship similarity** using Jaccard similarity on relationship sets. This ensures that entities with similar connections and attributes receive higher scores, even if their display names differ slightly.

## Confidence Scoring and Threshold Filtering

After raw similarity calculation, `DuplicateDetector._create_duplicate_candidate` in [`semantica/deduplication/duplicate_detector.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/duplicate_detector.py) applies confidence boosting logic. The system adds boosts for exact name matches, matching properties, and type equality, then caps the final confidence score at 1.0.

Candidates falling below the user-defined `confidence_threshold` are filtered out immediately, ensuring only high-fidelity duplicates proceed to the grouping phase.

## Scalable Processing Modes: Batch and Incremental

The `DuplicateDetector` supports two distinct processing patterns optimized for different data ingestion scenarios.

### Batch Detection with Vectorized Operations

The `detect_duplicates` method utilizes `SimilarityCalculator.batch_calculate_similarity` to compute all pairwise similarities in a single vectorized pass. This approach maximizes throughput when processing static datasets or performing initial knowledge graph construction.

### Incremental Detection for Streaming Data

For streaming ingestion, `incremental_detect` compares new entities only against the existing set, avoiding the full O(N²) scan. This incremental path reuses the same similarity engine while maintaining constant-time complexity relative to the existing graph size.

## Union-Find Grouping and Representative Selection

Duplicate candidates are fed into a **union-find (disjoint-set) algorithm** via `_build_duplicate_groups`. This efficiently merges transitive duplicates—where A matches B and B matches C—into single `DuplicateGroup` objects without requiring direct A-to-C comparisons.

Each group then selects a **representative entity** through `_select_representative`, preferring the record with the richest set of properties and relationships. This ensures the most complete canonical record survives the merge process.

## Integration with the Knowledge Graph Pipeline

The detector serves as the core engine for the `EntityResolver` class in [`semantica/kg/entity_resolver.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/kg/entity_resolver.py). This high-level wrapper plugs the detector into the knowledge-graph construction workflow, automatically deduplicating entities before persistence via the `resolve_entities` method.

## Code Examples

```python

# Basic duplicate detection (batch mode)

from semantica.deduplication import DuplicateDetector

entities = [
    {"id": "1", "name": "Apple Inc.", "type": "Company"},
    {"id": "2", "name": "Apple", "type": "Company"},
    {"id": "3", "name": "Microsoft", "type": "Company"},
]

detector = DuplicateDetector(similarity_threshold=0.8, confidence_threshold=0.7)
candidates = detector.detect_duplicates(entities)        # → List[DuplicateCandidate]

groups = detector.detect_duplicate_groups(entities)    # → List[DuplicateGroup]

```

```python

# Incremental detection for streaming ingestion

new_entities = [{"id": "4", "name": "Apple Corp"}]
existing = entities

incremental = detector.incremental_detect(new_entities, existing)

# Returns only the pairs that cross the similarity/confidence thresholds

```

```python

# Using the higher-level EntityResolver (the typical entry point in a KG pipeline)

from semantica.kg import EntityResolver

resolver = EntityResolver(strategy="semantic", similarity_threshold=0.85)
resolved = resolver.resolve_entities(entities)

# `resolved` contains merged entities with the most complete representation

```

## Summary

- **Blocking strategies** in `SimilarityCalculator._prefilter_pair` eliminate impossible pairs before expensive similarity calculations.
- **Multi-factor similarity** combines string, embedding, property, and relationship metrics with configurable weights.
- **Dual processing modes** support both full-batch vectorized processing and incremental streaming detection.
- **Union-find grouping** efficiently handles transitive duplicate relationships through `_build_duplicate_groups`.
- **Canonical representative selection** preserves the richest entity record via `_select_representative`.

## Frequently Asked Questions

### What blocking strategy does Semantica use to reduce comparison costs?

Semantica implements prefix-based blocking in [`semantica/deduplication/similarity_calculator.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/similarity_calculator.py) through the `_prefilter_pair` method. This strategy filters candidates based on type mismatches, name-length ratios, and token-overlap thresholds, reducing the O(N²) comparison space to a manageable subset before semantic scoring begins.

### How does the SimilarityCalculator combine different similarity metrics?

The `SimilarityCalculator` aggregates four weighted components: string similarity (Levenshtein, Jaro-Winkler, cosine), embedding similarity (cosine on vectors), property similarity (key/value matching), and relationship similarity (Jaccard on relationship sets). These weights are configurable and automatically normalized to produce a final composite score between 0 and 1.

### What is the difference between batch and incremental detection in Semantica?

Batch detection via `detect_duplicates` uses `batch_calculate_similarity` to process all pairwise comparisons in a single vectorized pass, ideal for static datasets. Incremental detection via `incremental_detect` compares only new entities against existing records, avoiding full O(N²) scans and enabling real-time streaming ingestion with constant-time complexity relative to graph size.

### How does Semantica select the canonical representative from a duplicate group?

The `_select_representative` method in [`semantica/deduplication/duplicate_detector.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/duplicate_detector.py) selects the entity with the richest set of properties and relationships as the canonical representative. This ensures that merged `DuplicateGroup` objects retain the most complete and informative record during the entity resolution process.