# What Algorithms Power Semantic Deduplication and Entity Resolution in Semantica?

> Discover the algorithms behind semantic deduplication and entity resolution in Semantica. Explore vector similarity, fuzzy matching, clustering, and rule-based merging.

- Repository: [Semantica /semantica](https://github.com/semantica-agi/semantica)
- Tags: deep-dive
- Published: 2026-08-20

---

**Semantica uses a multi-stage pipeline combining vector-based cosine similarity, fuzzy string matching, agglomerative hierarchical clustering, and rule-based entity merging for semantic deduplication and entity resolution.**

The `semantica-agi/semantica` repository implements a sophisticated deduplication engine designed for knowledge graphs. This system processes semantic triples (subject-predicate-object) and entity records to identify duplicates across variations in naming, structure, and provenance. The architecture separates similarity computation, clustering detection, and merge resolution into distinct, composable components.

## Core Algorithmic Components

### SimilarityCalculator: Multi-Modal Similarity Scoring

The `SimilarityCalculator` class in [`semantica/deduplication/similarity_calculator.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/similarity_calculator.py) computes pairwise similarity using three complementary techniques:

- **Cosine similarity** on dense embeddings (OpenAI, Cohere, or custom embedding models)
- **Jaccard similarity** on token-level n-grams for surface-level text overlap
- **Levenshtein distance** as a fallback for typos and minor string variations

This multi-modal approach ensures robust matching across semantic (meaning-based) and syntactic (string-based) duplicate types.

### DuplicateDetector: Clustering Algorithms

The `DuplicateDetector` class in [`semantica/deduplication/duplicate_detector.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/duplicate_detector.py) implements two primary clustering strategies:

| Algorithm | Use Case | Implementation Detail |
|-----------|----------|----------------------|
| **Agglomerative hierarchical clustering** (single-link) | General-purpose duplicate grouping | Builds dendrogram from similarity matrix, cuts at threshold |
| **DBSCAN-style density-based clustering** | Threshold-supplied detection | Identifies core samples and expands clusters by density |

The detector supports three operational modes:

- **`detect`** — Returns duplicate pairs without merging
- **`merge`** — Full pipeline: detection → clustering → merging
- **`magic`** (default) — Auto-selects threshold and strategy based on data characteristics

### EntityMerger: Rule-Based Canonicalization

The `EntityMerger` in [`semantica/deduplication/entity_merger.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/entity_merger.py) resolves duplicate groups into canonical entities using configurable strategies:

- **`keep-first`** — Preserve the earliest encountered record
- **`keep-most-recent`** — Select by latest timestamp
- **`custom-merge`** — User-defined resolution function

The merger prioritizes **metadata completeness** (highest provenance confidence, most populated fields) when no explicit strategy overrides apply.

### DeduplicationProvenance: Graph-Based Audit Trail

The `DeduplicationProvenance` class in [`semantica/deduplication/deduplication_provenance.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/deduplication_provenance.py) maintains a **provenance graph** recording which source records contributed to each merged entity. This enables downstream reasoning to consider merge history and supports deterministic re-execution.

## Practical Implementation

### Detecting Duplicate Triples

```python
from semantica.deduplication import DuplicateDetector

detector = DuplicateDetector(
    similarity_threshold=0.85,      # cosine similarity cutoff

    mode="semantic_v2",             # enables token-level + fuzzy fallback

)

duplicate_pairs = detector.detect(triples)   # triples: list of (s, p, o) dicts

print(duplicate_pairs)                     # → [(0, 3), (2, 5), ...]

```

### Merging with Custom Strategy

```python
from semantica.deduplication.entity_merger import EntityMerger

def keep_most_recent(old, new):
    return new if new["timestamp"] > old["timestamp"] else old

merger = EntityMerger(strategy=keep_most_recent)
canonical_entities = merger.merge(duplicate_pairs, triples)

```

### Full Pipeline Execution

```python

# Magic mode: auto-configured detection and merging

merged_graph = detector.merge(triples)

```

## Source File Architecture

| File Path | Responsibility |
|-----------|---------------|
| [`semantica/deduplication/similarity_calculator.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/similarity_calculator.py) | Embedding cosine, Jaccard, Levenshtein similarity metrics |
| [`semantica/deduplication/duplicate_detector.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/duplicate_detector.py) | Hierarchical and DBSCAN clustering; mode orchestration |
| [`semantica/deduplication/entity_merger.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/entity_merger.py) | Canonicalization strategies and conflict resolution |
| [`semantica/deduplication/deduplication_provenance.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/deduplication_provenance.py) | Provenance graph construction and merge audit logging |
| [`semantica/deduplication/registry.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/registry.py) | Plugin registration for custom methods and strategies |

## Summary

- **Vector + string hybrid similarity** handles both semantic and surface-level duplicates via `SimilarityCalculator`
- **Hierarchical and density-based clustering** groups duplicates flexibly through `DuplicateDetector`
- **Configurable merge strategies** resolve conflicts with provenance-aware canonicalization in `EntityMerger`
- **Complete audit trails** via provenance graphs ensure reproducibility and downstream reasoning support

The modular design allows domain-specific tuning—such as custom embedding models for company name variants or predicate synonym mappings—without modifying core algorithmic code.

## Frequently Asked Questions

### How does Semantica handle semantic versus syntactic duplicates?

Semantica addresses both types through layered similarity metrics. **Cosine similarity on embeddings** captures semantic equivalence (e.g., "Apple Inc." and "Apple Computer"), while **Jaccard n-gram overlap** and **Levenshtein distance** catch syntactic variations like typos and abbreviations. The `semantic_v2` mode automatically combines these signals with learned or configured weights.

### What clustering algorithm should I choose for large-scale datasets?

For large datasets, **DBSCAN-style clustering** in `DuplicateDetector` offers O(n log n) approximate performance with index acceleration, compared to O(n²) for full hierarchical clustering. Use DBSCAN when you can define a clear similarity threshold; fallback to agglomerative clustering for smaller datasets or when cluster granularity requires fine control via dendrogram cutting.

### Can I implement custom similarity metrics or merge logic?

Yes. The [`registry.py`](https://github.com/semantica-agi/semantica/blob/main/registry.py) module exposes extension points: register custom similarity functions via `SimilarityCalculator.register_metric()` and merge strategies via `EntityMerger.register_strategy()`. Both accept callable objects following the documented signatures in [`semantica/deduplication/registry.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/registry.py).

### How does provenance tracking affect storage overhead?

The `DeduplicationProvenance` graph stores merge operations as annotated edges between source and canonical entities. Overhead scales with merge density rather than raw entity count—typically 15-30% additional storage for heavily deduplicated knowledge graphs. The implementation uses compressed adjacency lists and optional serialization to Parquet for long-term archival.