# How to Implement Entity Deduplication Using Semantic Similarity in Semantica

> Learn to implement entity deduplication using semantic similarity with Semantica. Discover DuplicateDetector, candidate generation, and efficient merging for batch and incremental workflows.

- Repository: [Semantica /semantica](https://github.com/semantica-agi/semantica)
- Tags: how-to-guide
- Published: 2026-08-14

---

**Use Semantica's `DuplicateDetector` to compute pairwise semantic similarities, generate `DuplicateCandidate` records above configurable thresholds, and merge transitive duplicate groups via union-find—supporting both full-batch and incremental detection workflows.**

Entity deduplication using semantic similarity is essential for maintaining clean knowledge graphs, especially when ingesting data from multiple sources with inconsistent naming conventions. The Semantica framework provides a production-ready deduplication engine that combines vector-based similarity with configurable confidence scoring and efficient grouping algorithms. This guide walks through the complete implementation based on the actual source code in `semantica-agi/semantica`.

## Core Deduplication Workflow

The deduplication engine follows a three-stage pipeline implemented across [`semantica/deduplication/duplicate_detector.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/duplicate_detector.py) and [`semantica/deduplication/similarity_calculator.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/similarity_calculator.py).

### Stage 1: Compute Pairwise Similarity

The `SimilarityCalculator` class generates similarity scores for entity pairs using either vector embeddings or string-based methods. To avoid quadratic scaling, the calculator optionally applies **blocking**—partitioning entities by shared attributes (type, country, etc.) to limit comparisons.

Key methods in [`similarity_calculator.py`](https://github.com/semantica-agi/semantica/blob/main/similarity_calculator.py):

- `batch_calculate_similarity()` – vector-based similarity for large batches
- `calculate_similarity()` – single pair comparison
- `calculate_string_similarity()` – edit-distance or fuzzy matching fallback

### Stage 2: Create Duplicate Candidates

When a pair exceeds `similarity_threshold`, the `DuplicateDetector` instantiates a `DuplicateCandidate`. Confidence scores are boosted by auxiliary signals:

- Exact name match
- Shared properties
- Same entity type

### Stage 3: Group Duplicates Transitively

Candidates are merged into groups using `_build_duplicate_groups()`, which implements a **union-find (disjoint set)** algorithm. Each group receives:

- Aggregated confidence score
- A **representative entity**—the member with the richest property/relationship set

## Configuration Parameters

The `DuplicateDetector` in [`duplicate_detector.py`](https://github.com/semantica-agi/semantica/blob/main/duplicate_detector.py) exposes these tunable parameters:

| Parameter | Purpose | Typical Range |
|-----------|---------|---------------|
| `similarity_threshold` | Minimum similarity to consider a pair | `0.7` – `0.9` |
| `confidence_threshold` | Minimum confidence to retain a candidate | `0.6` – `0.8` |
| `min_similarity` | Additional floor after similarity computation | `None` or `[0.0, 1.0]` |
| `top_k_per_entity` | Max candidates each entity can sponsor | `None` or integer |
| `max_results` | Global cap on returned candidates | `None` or integer |
| `sort_by` | Ranking field (`confidence` or `similarity_score`) | `confidence` (default) |

These values are validated against defaults stored in [`semantica/deduplication/config.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/config.py).

## Basic Duplicate Detection

This example demonstrates standard batch processing against a static entity collection:

```python
from semantica.deduplication import DuplicateDetector

# Entities require at minimum "id" and "name" fields

entities = [
    {"id": "1", "name": "Apple Inc.", "properties": {"country": "US"}},
    {"id": "2", "name": "Apple", "properties": {"country": "US"}},
    {"id": "3", "name": "Microsoft", "properties": {"country": "US"}},
]

detector = DuplicateDetector(
    similarity_threshold=0.8,
    confidence_threshold=0.7,
    top_k_per_entity=5,
    max_results=10,
)

candidates = detector.detect_duplicates(entities)
for cand in candidates:
    print(
        f"Duplicate: {cand.entity1['name']} ↔ {cand.entity2['name']}, "
        f"sim={cand.similarity_score:.2f}, conf={cand.confidence:.2f}, "
        f"reasons={cand.reasons}"
    )

```

The `reasons` attribute lists which signals contributed to the confidence boost.

## Creating and Inspecting Duplicate Groups

For downstream merge operations, group candidates transitively:

```python
groups = detector.detect_duplicate_groups(entities)
for grp in groups:
    rep = grp.representative
    print(
        f"Group of {len(grp.entities)} entities, confidence={grp.confidence:.2f}, "
        f"representative={rep['name'] if rep else 'None'}"
    )

```

The `representative` selection logic prioritizes entities with the most complete property sets and relationship connections.

## Incremental Detection for Streaming Workflows

When new entities arrive continuously, avoid reprocessing the entire graph using `incremental_detect()`:

```python
new_entities = [
    {"id": "4", "name": "Apple Corp", "properties": {"country": "US"}},
]

existing_entities = entities  # Previously deduplicated collection

incremental_candidates = detector.incremental_detect(new_entities, existing_entities)
print(f"Found {len(incremental_candidates)} new duplicate candidates.")

```

This method runs in **O(N × M)** time for N new entities against M existing entities—far more efficient than full **O((N+M)²)** pairwise comparison.

## Post-Detection: Merging Strategies

After detection, the deduplication pipeline supports automated entity resolution through components in [`semantica/deduplication/merge_strategy.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/merge_strategy.py) and [`semantica/deduplication/entity_merger.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/entity_merger.py):

- **Merge strategy selection** – choose between overwrite, preserve-richer, or custom field-level merging
- **Entity merger execution** – applies the selected strategy to produce canonical records

The [`registry.py`](https://github.com/semantica-agi/semantica/blob/main/registry.py) file registers these components for dynamic discovery by the broader Semantica framework, enabling plugin-style extension.

## Summary

- **Entity deduplication using semantic similarity** in Semantica centers on the `DuplicateDetector` class with configurable thresholds and confidence scoring.
- The three-stage workflow—similarity computation, candidate generation, and transitive grouping—handles both batch and incremental detection.
- Key source files: [`duplicate_detector.py`](https://github.com/semantica-agi/semantica/blob/main/duplicate_detector.py) (orchestration), [`similarity_calculator.py`](https://github.com/semantica-agi/semantica/blob/main/similarity_calculator.py) (scoring), [`config.py`](https://github.com/semantica-agi/semantica/blob/main/config.py) (defaults), [`merge_strategy.py`](https://github.com/semantica-agi/semantica/blob/main/merge_strategy.py) and [`entity_merger.py`](https://github.com/semantica-agi/semantica/blob/main/entity_merger.py) (resolution), and [`registry.py`](https://github.com/semantica-agi/semantica/blob/main/registry.py) (framework integration).
- Use `incremental_detect()` for streaming data to maintain efficiency at scale.

## Frequently Asked Questions

### What similarity methods does Semantica support for entity deduplication?

Semantica supports both **vector-based similarity** (embeddings computed via `batch_calculate_similarity` or `calculate_similarity`) and **string-based similarity** (fuzzy matching via `calculate_string_similarity`) in [`similarity_calculator.py`](https://github.com/semantica-agi/semantica/blob/main/similarity_calculator.py). The `DuplicateDetector` automatically selects appropriate methods based on available entity features.

### How does the union-find grouping algorithm handle ambiguous duplicate chains?

The `_build_duplicate_groups()` method in [`duplicate_detector.py`](https://github.com/semantica-agi/semantica/blob/main/duplicate_detector.py) treats duplicate detection as a **connected components problem**: if A matches B and B matches C, all three form one group regardless of whether A directly matches C. This transitive closure ensures no duplicate escapes detection through intermediate entities.

### Can I customize how representative entities are selected during grouping?

Yes. The representative selection logic evaluates entity richness—count of properties and relationships—and is designed for override. For custom strategies, extend the `DuplicateDetector` class or modify the grouping pipeline before the `EntityMerger` in [`entity_merger.py`](https://github.com/semantica-agi/semantica/blob/main/entity_merger.py) applies your preferred `MergeStrategy`.

### What performance should I expect for large knowledge graphs?

For N entities, full detection is **O(N²)** without blocking, reduced significantly by attribute-based blocking. The `incremental_detect()` method achieves **O(N × M)** for batch inserts against existing graphs. Configure `top_k_per_entity` and `max_results` to enforce hard memory and latency bounds in production deployments.