# How Semantica Handles Entity Resolution and Deduplication at Scale

> Discover how Semantica scales entity resolution and deduplication using parallel pipelines, similarity scoring, and union-find for massive knowledge graphs. Achieve linear-ith scalability.

- Repository: [Semantica /semantica](https://github.com/semantica-agi/semantica)
- Tags: how-to-guide
- Published: 2026-09-10

---

**Semantica employs a multi-stage, highly-parallel pipeline that combines blocking indexes, multi-factor similarity scoring, and union-find algorithms to resolve and deduplicate entities across massive knowledge graphs with linear-ith scalability.**

Entity resolution and deduplication at scale presents significant computational challenges for knowledge graph systems. The `semantica-agi/semantica` repository implements a sophisticated, configurable framework that processes millions to billions of entities through optimized blocking strategies and parallel similarity computation. This architecture avoids the prohibitive O(n²) complexity of naive pairwise comparison while maintaining high accuracy across diverse entity types.

## The Multi-Stage Pipeline Architecture

Semantica’s deduplication engine orchestrates seven distinct stages to transform raw entity comparisons into consolidated duplicate groups. According to the source code in [`duplicate_detector.py`](https://github.com/semantica-agi/semantica/blob/main/duplicate_detector.py) and [`similarity_calculator.py`](https://github.com/semantica-agi/semantica/blob/main/similarity_calculator.py), the pipeline progresses from rapid pre-filtering through sophisticated similarity analysis to final group consolidation. Each stage is designed to minimize computational overhead while maximizing match quality through configurable thresholds and limits.

The system tracks progress throughout execution using a shared **ProgressTracker** that reports fine-grained status updates—such as per-100% progress and remaining items—without incurring per-entity overhead. This ensures low-latency monitoring even when processing billions of records.

## Pre-Filtering and Blocking Strategies

Before expensive similarity calculations begin, Semantica applies aggressive pre-filtering to eliminate obvious non-matches. In [`duplicate_detector.py`](https://github.com/semantica-agi/semantica/blob/main/duplicate_detector.py) (lines 61-82), the `_prefilter_pair` function implements fast heuristics including type mismatches, name-length ratio checks, and token-overlap thresholds to reject incompatible pairs in microseconds.

For candidate generation, the system employs **blocking indexes** configured via [`similarity_calculator.py`](https://github.com/semantica-agi/semantica/blob/main/similarity_calculator.py) (lines 52-71). Entities are partitioned into blocks using strategies such as:

- **First-letter blocking**: Grouping by initial character
- **Token-prefix blocking**: Shared n-gram prefixes  
- **Phonetic Soundex**: Encoding names by sound representation

By generating candidate pairs only from entities sharing a block, Semantica reduces the comparison space from O(n²) to a manageable subset, enabling linear-ith-like scalability.

## Multi-Factor Similarity Scoring

For each candidate pair passing the blocking stage, the **SimilarityCalculator** (defined in [`similarity_calculator.py`](https://github.com/semantica-agi/semantica/blob/main/similarity_calculator.py), lines 71-86) computes a weighted aggregate across four distinct dimensions:

**String similarity** supports multiple algorithms including Jaro-Winkler, Levenshtein distance, and cosine n-gram similarity, allowing adaptation to different naming conventions and typo patterns.

**Property similarity** performs pairwise comparison of entity attributes with partial-match handling, ensuring that schema variations do not prevent valid matches.

**Relationship similarity** calculates Jaccard similarity across hashed relationship sets, capturing structural equivalence in the knowledge graph.

**Embedding similarity** computes cosine similarity of vector embeddings when present, enabling semantic matching beyond lexical similarity.

Component weights normalize at runtime, allowing flexible tuning via configuration without code changes.

## Confidence Filtering and Short-Circuit Optimization

To prevent wasted computation, the pipeline implements aggressive short-circuiting and filtering mechanisms. As implemented in [`duplicate_detector.py`](https://github.com/semantica-agi/semantica/blob/main/duplicate_detector.py) (lines 86-99), if string similarity falls below a baseline threshold and no embeddings are available, the calculation terminates early.

The `_apply_result_limits` function (lines 760-785) enforces multiple constraint layers:

- Configurable **confidence thresholds** and **min-similarity floors**
- **Per-entity top-k limits** to prevent noisy overload from high-frequency entities  
- **Global caps** on total duplicate candidates to bound memory usage

These limits ensure that even under pathological data distributions, the system maintains predictable resource consumption.

## Union-Find Group Formation

Once individual duplicate pairs are identified, Semantica consolidates them into cohesive groups using a union-find algorithm. The `_build_duplicate_groups` method in [`duplicate_detector.py`](https://github.com/semantica-agi/semantica/blob/main/duplicate_detector.py) (lines 333-385) merges intersecting pairs into **DuplicateGroup** objects.

Group confidence calculation (lines 395-406 in `_calculate_group_confidence`) derives from the average pairwise similarity, augmented by a size-based boost that increases confidence for larger clusters. The system selects a **representative entity** for each group based on the richest property and relationship set, ensuring the canonical version retains maximum information density.

## Incremental Detection for Streaming Workloads

For production environments with continuous data ingestion, Semantica provides `incremental_detect` (lines 49-69 in [`duplicate_detector.py`](https://github.com/semantica-agi/semantica/blob/main/duplicate_detector.py)). This method compares new entities exclusively against an existing knowledge base rather than triggering a full re-run.

The incremental mode reuses the identical similarity engine and respects all configured thresholds and limits, ensuring consistency between batch and streaming processing without sacrificing the performance benefits of the blocking architecture.

## Implementation Examples

The following examples demonstrate practical usage of the deduplication API:

```python

# Basic duplicate detection on a list of entity dicts

from semantica.deduplication import DuplicateDetector

entities = [
    {"id": "1", "name": "Apple Inc.", "type": "company"},
    {"id": "2", "name": "Apple", "type": "company"},
    {"id": "3", "name": "Microsoft", "type": "company"},
]

detector = DuplicateDetector(similarity_threshold=0.8, confidence_threshold=0.7)
candidates = detector.detect_duplicates(entities)
print(f"Found {len(candidates)} duplicate pairs")

# Form groups and pick a representative entity

groups = detector.detect_duplicate_groups(entities)
for g in groups:
    print(f"Group of {len(g.entities)} entities – confidence {g.confidence:.2f}")
    print("Representative:", g.representative)

```

```python

# Incremental detection – new data against an existing knowledge base

new = [{"id": "4", "name": "Apple Corp.", "type": "company"}]
existing = entities
incremental = detector.incremental_detect(new, existing)
print(f"Incremental duplicates: {len(incremental)}")

```

```python

# Custom blocking and weighting

custom_calc = SimilarityCalculator(
    string_weight=0.5,
    property_weight=0.3,
    embedding_weight=0.2,
    config={"prefilter_enabled": True, "candidate_strategy": "blocking_v2"}
)

detector = DuplicateDetector(
    similarity_threshold=0.75,
    confidence_threshold=0.65,
    config={"similarity": {"candidate_strategy": "blocking_v2", "max_candidates_per_entity": 50}}
)

```

## Summary

- **Blocking indexes** in [`similarity_calculator.py`](https://github.com/semantica-agi/semantica/blob/main/similarity_calculator.py) reduce the comparison space from O(n²) to linear-ith complexity using configurable strategies like Soundex and token-prefix blocking.
- **Multi-factor similarity** combines string, property, relationship, and embedding metrics with runtime-normalized weights for flexible matching.
- **Short-circuit logic** in `detect_duplicates` eliminates low-probability pairs early, while `_apply_result_limits` enforces per-entity and global caps to bound resource usage.
- **Union-find consolidation** via `_build_duplicate_groups` merges transitive duplicate relationships into canonical DuplicateGroup objects with confidence scoring.
- **Incremental detection** supports streaming workloads by comparing new entities only against existing knowledge bases without full reprocessing.

## Frequently Asked Questions

### How does Semantica achieve scalability for billions of entities?

Semantica achieves linear-ith scalability through aggressive blocking in [`similarity_calculator.py`](https://github.com/semantica-agi/semantica/blob/main/similarity_calculator.py) (lines 52-71), which partitions entities into comparison blocks using phonetic hashes or token prefixes. This eliminates the O(n²) pairwise explosion by comparing only co-blocked candidates. Combined with pre-filtering in `_prefilter_pair` (lines 61-82 of [`duplicate_detector.py`](https://github.com/semantica-agi/semantica/blob/main/duplicate_detector.py)) and early short-circuiting, the system processes billions of records without quadratic degradation.

### Which similarity algorithms does Semantica support for entity matching?

The framework supports **Jaro-Winkler**, **Levenshtein distance**, and **cosine n-gram similarity** for string comparison, as implemented in [`similarity_calculator.py`](https://github.com/semantica-agi/semantica/blob/main/similarity_calculator.py). For structural matching, it uses **Jaccard similarity** on hashed relationship sets. When vector embeddings are available, the system calculates **cosine similarity** to capture semantic equivalence. These components are weighted and aggregated at runtime.

### What is the difference between `detect_duplicates` and `incremental_detect`?

The `detect_duplicates` method performs full pairwise analysis within a provided dataset, while `incremental_detect` (lines 49-69 of [`duplicate_detector.py`](https://github.com/semantica-agi/semantica/blob/main/duplicate_detector.py)) optimizes for streaming scenarios by evaluating new entities exclusively against an existing knowledge base. Both methods utilize identical similarity engines and respect the same confidence thresholds, but incremental mode avoids reprocessing historical data.

### How does Semantica select representative entities from duplicate groups?

During group formation in `_build_duplicate_groups` (lines 333-385), the system selects the representative entity based on information richness—specifically choosing the entity with the most comprehensive property and relationship sets. Group confidence scores derive from `_calculate_group_confidence` (lines 395-406), which averages pairwise similarities and applies a size-based boost to larger clusters.