# How to Implement Semantic Entity Deduplication in Semantica: A Complete Guide

> Implement semantic entity deduplication in Semantica using the DuplicateDetector class. Configure thresholds and scoring for accurate duplicate identification and grouping.

- Repository: [Semantica /semantica](https://github.com/semantica-agi/semantica)
- Tags: how-to-guide
- Published: 2026-08-15

---

**Use the `DuplicateDetector` class in `semantica.deduplication` with configurable similarity thresholds and multi-factor scoring to identify, rank, and group duplicate entities using string, property, relationship, and embedding-based comparison.**

Semantic entity deduplication in Semantica provides a production-ready pipeline for identifying and merging duplicate records in knowledge graphs and entity datasets. Whether you're cleaning customer data, consolidating research databases, or building a unified semantic layer, the `semantica-agi/semantica` repository offers fine-grained control over every stage of the deduplication process—from similarity calculation to final group formation and merging.

## Core Architecture: Three Integrated Components

The semantic entity deduplication system resides in the **`semantica.deduplication`** package and consists of three tightly-coupled components:

| Component | Responsibility |
|-----------|--------------|
| **`SimilarityCalculator`** ([`similarity_calculator.py`](https://github.com/semantica-agi/semantica/blob/main/similarity_calculator.py)) | Computes multi-factor similarity scores between entity pairs |
| **`DuplicateDetector`** ([`duplicate_detector.py`](https://github.com/semantica-agi/semantica/blob/main/duplicate_detector.py)) | Orchestrates detection, builds candidate lists, and forms duplicate groups |
| **`EntityMerger`** ([`entity_merger.py`](https://github.com/semantica-agi/semantica/blob/main/entity_merger.py)) | Merges duplicate groups into canonical representative entities |

Each component can be configured independently, allowing you to optimize for precision-sensitive or recall-sensitive use cases.

## Step 1: Configure Multi-Factor Similarity Scoring

The `SimilarityCalculator.calculate_similarity` method (lines 31-44 in [`similarity_calculator.py`](https://github.com/semantica-agi/semantica/blob/main/similarity_calculator.py)) aggregates four similarity components into a weighted composite score:

```python

# Weighted aggregation from similarity_calculator.py

weights = {
    "string": self.string_weight,
    "property": self.property_weight,
    "relationship": self.relationship_weight,
    "embedding": self.embedding_weight if embedding_score > 0 else 0.0,
}

# Normalized weights → overall_score = Σ(component × weight)

```

### Similarity Components Explained

- **String similarity** – Choose from **Jaro-Winkler**, **Levenshtein distance**, or **cosine similarity over character bigrams** via `_jaro_winkler_similarity`, `_levenshtein_similarity`, or `_cosine_similarity`
- **Property similarity** – Token-wise string comparison for matching keys, neutral handling of missing values (0.5), full credit for exact matches
- **Relationship similarity** – **Jaccard similarity** over hashable relationship tuples using `_make_hashable`
- **Embedding similarity** – Cosine similarity on pre-computed vector embeddings for deep semantic matching

A fast **prefilter** (`_prefilter_pair`, lines 62-78) eliminates obviously unrelated pairs before expensive multi-factor scoring runs.

## Step 2: Detect Duplicate Candidates with Thresholds and Limits

The `DuplicateDetector.detect_duplicates` method drives the core deduplication flow through three phases:

```python
from semantica.deduplication import DuplicateDetector

detector = DuplicateDetector(
    similarity_threshold=0.8,    # Minimum similarity to consider a pair

    confidence_threshold=0.75,   # Minimum confidence for a candidate

    top_k_per_entity=5,          # Keep up to 5 candidates per entity

    max_results=200,             # Global cap on returned candidates

    sort_by="confidence",        # Rank by confidence or similarity_score

)

entities = [
    {"id": "1", "name": "Apple Inc.", "type": "Company"},
    {"id": "2", "name": "Apple", "type": "Company"},
    {"id": "3", "name": "Microsoft", "type": "Company"},
]

candidates = detector.detect_duplicates(entities)
for cand in candidates:
    print(f"Duplicate pair: {cand.entity1['id']} ↔ {cand.entity2['id']}")
    print(f"  similarity={cand.similarity_score:.2f}, confidence={cand.confidence:.2f}")

```

### Detection Pipeline Internals

1. **Batch similarity computation** – `SimilarityCalculator.batch_calculate_similarity` generates candidate pairs via blocking (`_build_block_indexes`, `_generate_candidate_pairs`) with optional capping (`_cap_candidate_pairs`)
2. **Candidate creation** – `_create_duplicate_candidate` (lines 560-618) evaluates **exact name matches**, **property matches**, and **type matches** to compute a composite **confidence score**
3. **Result limiting** – `_apply_result_limits` (lines 668-710) filters by `min_similarity`, sorts by confidence or similarity, applies `top_k_per_entity` and `max_results` caps

## Step 3: Form Transitive Duplicate Groups with Union-Find

After detecting pairwise candidates, `detect_duplicate_groups` builds transitive clusters using a **union-find algorithm**:

```python
groups = detector.detect_duplicate_groups(entities)
for g in groups:
    ids = [e["id"] for e in g.entities]
    print(f"Group ({len(ids)} entities): {ids}")
    print(f"  representative: {g.representative['id']}")
    print(f"  group confidence: {g.confidence:.2f}")

```

The `_build_duplicate_groups` implementation:

- Maps each entity ID to a group reference
- Merges groups when a candidate bridges two existing groups
- Calculates group confidence from average pairwise similarity, boosted for larger groups (`_calculate_group_confidence`, lines 667-678)
- Selects the **most complete entity** as representative—the one with the richest property/relationship set (`_select_representative`, lines 780-891)

## Step 4: Handle Streaming Data with Incremental Detection

For production pipelines that process data in batches, `incremental_detect` (lines 445-511) avoids recomputing the full O(N²) matrix:

```python
new_entities = [{"id": "4", "name": "Apple Corp.", "type": "Company"}]
existing_entities = entities  # Previously processed corpus

incremental_candidates = detector.incremental_detect(new_entities, existing_entities)
print(f"Found {len(incremental_candidates)} new duplicate candidates.")

```

This compares only the new batch against the existing corpus, reducing computational cost from quadratic to linear in the batch size.

## Step 5: Deduplicate Semantic Relationships

Beyond entities, Semantica supports **relationship deduplication** via `detect_relationship_duplicates`:

```python
relationships = [
    {"subject": "Apple", "predicate": "founded_by", "object": "Steve Jobs"},
    {"subject": "Apple", "predicate": "established_by", "object": "Steve Jobs"},
]

duplicates = detector.detect_relationship_duplicates(
    relationships,
    relationship_dedup_mode="semantic_v2",
    predicate_synonym_map={"founded_by": "established_by"},
    literal_normalization_enabled=True,
)

```

In **semantic_v2 mode** (lines 480-496), the method:

- Normalizes predicates using a **predicate synonym map**
- Applies optional literal normalization
- Hashes the normalized `(subject, predicate, object)` triple for comparison

Legacy mode falls back to simple string similarity on predicates.

## Key Configuration Parameters

| Parameter | Default | Effect |
|-----------|---------|--------|
| `similarity_threshold` | 0.75 | Minimum composite similarity for pair consideration |
| `confidence_threshold` | 0.7 | Minimum confidence score for candidate inclusion |
| `string_weight` | 0.4 | Weight for string similarity component |
| `embedding_weight` | 0.3 | Weight for embedding similarity (0 if no embeddings) |
| `property_weight` | 0.2 | Weight for property overlap similarity |
| `relationship_weight` | 0.1 | Weight for relationship Jaccard similarity |
| `blocking_strategy` | `"standard"` | Candidate pair generation strategy for scalability |

All parameters are exposed through the `DuplicateDetector` constructor and `SimilarityCalculator` configuration dictionary in [`semantica/deduplication/config.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/config.py).

## Summary

- **Semantic entity deduplication in Semantica** combines four similarity signals—string, property, relationship, and embedding—into configurable composite scores ([`semantica/deduplication/similarity_calculator.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/similarity_calculator.py))
- **Threshold-based filtering and ranking** via `DuplicateDetector.detect_duplicates` provides precise control over precision-recall tradeoffs
- **Union-find group formation** builds transitive duplicate clusters with confidence-weighted representatives (`detect_duplicate_groups`)
- **Incremental detection** (`incremental_detect`) enables efficient streaming pipelines without full reprocessing
- **Relationship deduplication** with synonym-aware normalization handles semantic predicate variants
- **Full configurability** of weights, thresholds, and blocking strategies adapts the pipeline to diverse data quality scenarios

## Frequently Asked Questions

### How does Semantica handle missing or incomplete entity properties during deduplication?

The `SimilarityCalculator` treats missing values neutrally by assigning a 0.5 similarity score for property comparisons involving absent keys. This prevents missing data from heavily penalizing otherwise similar entities while still rewarding exact matches with full credit. The neutral handling is implemented in the property similarity calculation within [`similarity_calculator.py`](https://github.com/semantica-agi/semantica/blob/main/similarity_calculator.py).

### What embedding models work with Semantica's embedding similarity component?

Semantica accepts pre-computed embeddings as entity properties and computes cosine similarity between vectors. The system is embedding-model-agnostic—you can supply vectors from sentence-transformers, OpenAI embeddings, domain-specific models, or custom fine-tuned representations. The embedding similarity weight automatically zeroes out when no embeddings are present for either entity in a pair.

### Can I customize the blocking strategy for very large datasets?

Yes. The `batch_calculate_similarity` method exposes blocking through `_build_block_indexes` and `_generate_candidate_pairs`, with optional candidate capping via `_cap_candidate_pairs`. You can configure blocking keys (e.g., phonetic codes, locality-sensitive hashes) to reduce candidate pairs from O(N²) to manageable subsets. For extreme scale, combine blocking with `incremental_detect` to process data in bounded-size batches.

### How does Semantica select which entity becomes the group representative?

The `_select_representative` method (lines 780-891 in [`duplicate_detector.py`](https://github.com/semantica-agi/semantica/blob/main/duplicate_detector.py)) chooses the entity with the most complete property and relationship set as the canonical representative. "Completeness" is measured by counting non-null properties and outgoing relationships, ensuring the representative preserves maximum information from the duplicate group.