# Entity Deduplication in Semantica: Implementing Blocking and Similarity Scoring

> Implement entity deduplication in Semantica using blocking and similarity scoring. Discover how DuplicateDetector orchestrates a unified pipeline for efficient duplicate detection and merging.

- Repository: [Semantica /semantica](https://github.com/semantica-agi/semantica)
- Tags: how-to-guide
- Published: 2026-08-16

---

**Entity deduplication in Semantica combines blocking strategies to reduce comparison space with configurable similarity scoring to detect duplicates, orchestrated through the `DuplicateDetector` class that ties blocking, scoring, and merging into a unified pipeline.**

The `semantica-agi/semantica` library provides a production-ready deduplication system designed for entity reconciliation tasks. Its architecture follows established record-linkage patterns: partition entities into coarse-grained blocks, compute fine-grained similarity within each block, then merge confirmed duplicates with full provenance tracking. This article examines the implementation based on the source code in [`semantica/deduplication/`](https://github.com/semantica-agi/semantica/tree/main/semantica/deduplication).

---

## Blocking Strategies in Semantica

Blocking is the first stage of the deduplication pipeline. The goal is to partition an entity collection into smaller subsets (blocks) where potential duplicates are guaranteed to co-occur, reducing the complexity from O(n²) to O(∑|block|²).

### Simple Blocking

The `SimpleBlocker` class in [[`blocking.py`](https://github.com/semantica-agi/semantica/blob/main/blocking.py)](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/blocking.py) provides deterministic key-based partitioning. Users supply a key function that extracts a coarse attribute from each entity.

```python
from semantica.deduplication.blocking import SimpleBlocker

blocker = SimpleBlocker(key=lambda e: e["name"][0].lower())
blocks = blocker.partition(entities)  # {"a": [...], "b": [...], ...}

```

The blocker returns a dictionary mapping keys to entity lists. Each block contains all entities sharing that key, ensuring no cross-block duplicates are missed while keeping block sizes manageable.

### Advanced Blocking Options

Semantica's blocking module supports multiple strategies:

- **Hash-based blocking** – uses deterministic hashing of normalized identifiers
- **Phonetic blocking** – applies Soundex or Metaphone encodings to group similar-sounding names
- **Composite blocking** – combines multiple keys with hierarchical partitioning

---

## Similarity Scoring Architecture

The [[`similarity_calculator.py`](https://github.com/semantica-agi/semantica/blob/main/similarity_calculator.py)](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/similarity_calculator.py) module implements the `SimilarityCalculator` class, which computes pairwise similarity scores within blocks.

### Multi-Metric Scoring

The calculator accepts a list of scorer components with optional weights, enabling domain-specific similarity definitions:

```python
from semantica.deduplication.similarity_calculator import SimilarityCalculator

scorer = SimilarityCalculator(
    scorers=["jaro_winkler", "cosine"],
    weights=[0.6, 0.4],
)

```

Each scorer returns a float in **[0, 1]**, and the weighted combination produces the final similarity score.

### Available Scorers

| Scorer | Description | Use Case |
|--------|-------------|----------|
| `jaro_winkler` | Edit-distance variant favoring prefix matches | Name variations, typos |
| `levenshtein` | Classic edit distance | Character-level differences |
| `cosine` | TF-IDF cosine similarity over token sets | Descriptive text fields |
| `semantic_v2` | Domain-specific embeddings | Semantic equivalence beyond surface form |

### Pairwise Comparison Loop

For each block, the system performs exhaustive pairwise comparisons:

```python
pair_scores = []
for block in blocks.values():
    for i, e1 in enumerate(block):
        for e2 in block[i + 1:]:
            score = scorer.compare(e1, e2)
            if score >= 0.85:
                pair_scores.append((e1, e2, score))

```

The threshold (here `0.85`) is user-configurable and controls the precision-recall tradeoff.

---

## Entity Merging and Provenance

Confirmed duplicates proceed to the [[`entity_merger.py`](https://github.com/semantica-agi/semantica/blob/main/entity_merger.py)](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/entity_merger.py) module, which resolves conflicts and produces canonical entities.

### Core Merging Logic

```python
from semantica.deduplication.entity_merger import EntityMerger

merger = EntityMerger(provenance=True)
merged = merger.merge(e1, e2)

```

With `provenance=True`, the merger records:
- Original source entity IDs
- Merge timestamp and operation type
- Conflict resolution decisions

This enables downstream auditing and potential rollback operations.

### Detect vs. Merge Modes

The merger supports two operational modes:

- **Detection mode** – returns duplicate groups without modifying entities (dry-run analysis)
- **Apply mode** – performs the merge and emits canonical entities

---

## High-Level Orchestration with DuplicateDetector

The [[`duplicate_detector.py`](https://github.com/semantica-agi/semantica/blob/main/duplicate_detector.py)](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/duplicate_detector.py) module provides the unified `DuplicateDetector` class that orchestrates the entire pipeline.

### Configuration-Based Initialization

```python
from semantica.deduplication import DuplicateDetector

detector = DuplicateDetector(
    blocking_strategy="simple",
    similarity_strategy="semantic_v2",
    min_similarity=0.85,
)

```

### Full Pipeline Execution

```python

# Detect duplicate groups

duplicates = detector.detect(entity_collection)

# Optional: apply merges

canonical_entities = detector.merge(duplicates)

```

The `detect()` method returns a list of duplicate groups, where each group contains entity pairs with their similarity scores. The `merge()` method consumes these groups and produces the final deduplicated output.

---

## CLI Integration

Semantica exposes deduplication through the command line:

```bash
semantica deduplicate \
    --input entities.jsonl \
    --strategy semantic_v2 \
    --blocking simple \
    --min-similarity 0.9 \
    --dry-run

```

The CLI forwards arguments to `DuplicateDetector`, supporting both analysis (`--dry-run`) and live execution modes. See [[`tests/test_cli_commands.py`](https://github.com/semantica-agi/semantica/blob/main/tests/test_cli_commands.py)](https://github.com/semantica-agi/semantica/blob/main/semantica/tests/test_cli_commands.py) for usage examples.

---

## Complete Working Example

Below is a runnable end-to-end example combining all pipeline stages:

```python
from semantica.deduplication import DuplicateDetector
from semantica.deduplication.blocking import SimpleBlocker
from semantica.deduplication.similarity_calculator import SimilarityCalculator
from semantica.deduplication.entity_merger import EntityMerger

# Sample entity collection

entities = [
    {"id": "e1", "name": "Acme Corporation", "domain": "acme.com"},
    {"id": "e2", "name": "ACME Corp", "domain": "acme.com"},
    {"id": "e3", "name": "Global Industries", "domain": "global.com"},
]

# High-level approach

detector = DuplicateDetector(
    blocking_strategy="simple",
    similarity_strategy="jaro_winkler",
    min_similarity=0.8,
)

duplicates = detector.detect(entities)
print(f"Found {len(duplicates)} duplicate groups")

# Low-level component approach

blocker = SimpleBlocker(key=lambda e: e["domain"].split(".")[0])
blocks = blocker.partition(entities)

scorer = SimilarityCalculator(
    scorers=["jaro_winkler", "cosine"],
    weights=[0.7, 0.3],
)

merger = EntityMerger(provenance=True)
merged_entities = []

for block in blocks.values():
    for i, e1 in enumerate(block):
        for e2 in block[i + 1:]:
            if scorer.compare(e1, e2) >= 0.8:
                merged = merger.merge(e1, e2)
                merged_entities.append(merged)

```

---

## Summary

- **Blocking** in Semantica uses the `SimpleBlocker` class with configurable key functions to partition entities into comparison blocks, implemented in [[`blocking.py`](https://github.com/semantica-agi/semantica/blob/main/blocking.py)](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/blocking.py)
- **Similarity scoring** employs the `SimilarityCalculator` with pluggable metrics (Jaro-Winkler, cosine, semantic embeddings) in [[`similarity_calculator.py`](https://github.com/semantica-agi/semantica/blob/main/similarity_calculator.py)](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/similarity_calculator.py)
- **Merging** via `EntityMerger` resolves conflicts with full provenance tracking in [[`entity_merger.py`](https://github.com/semantica-agi/semantica/blob/main/entity_merger.py)](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/entity_merger.py)
- **Orchestration** through `DuplicateDetector` unifies all stages with both programmatic and CLI interfaces in [[`duplicate_detector.py`](https://github.com/semantica-agi/semantica/blob/main/duplicate_detector.py)](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/duplicate_detector.py)

---

## Frequently Asked Questions

### What is entity blocking and why is it necessary?

Entity blocking partitions a dataset into smaller groups based on shared coarse attributes (like first letters, domain names, or phonetic codes). It is necessary because comparing every entity pair in large datasets is computationally infeasible—blocking reduces comparisons from O(n²) to manageable subsets while guaranteeing true duplicates remain in the same block for evaluation.

### How does Semantica compute similarity scores between entities?

Semantica's `SimilarityCalculator` combines multiple token-level and semantic metrics through weighted averaging. Each configured scorer returns a [0, 1] score, and the final similarity is the weighted sum. Users can customize the scorer list, weights, and similarity threshold to match their domain's precision-recall requirements.

### Can I use custom similarity functions in Semantica?

Yes. The `SimilarityCalculator` accepts any object implementing the scorer interface. You can register custom scorers by name in the configuration or pass callable objects directly. The architecture in [[`similarity_calculator.py`](https://github.com/semantica-agi/semantica/blob/main/similarity_calculator.py)](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/similarity_calculator.py) is designed for extensibility without modifying core library code.

### What happens to entity provenance after merging?

When `EntityMerger` operates with `provenance=True`, it preserves the complete history of each merge operation including source entity IDs, timestamps, and conflict resolution decisions. This metadata attaches to the canonical entity, enabling audit trails, downstream lineage tracking, and potential future unmerge operations.