Entity Deduplication in Semantica: Implementing Blocking and Similarity Scoring

Entity deduplication in Semantica combines blocking strategies to reduce comparison space with configurable similarity scoring to detect duplicates, orchestrated through the DuplicateDetector class that ties blocking, scoring, and merging into a unified pipeline.

The semantica-agi/semantica library provides a production-ready deduplication system designed for entity reconciliation tasks. Its architecture follows established record-linkage patterns: partition entities into coarse-grained blocks, compute fine-grained similarity within each block, then merge confirmed duplicates with full provenance tracking. This article examines the implementation based on the source code in semantica/deduplication/.

---

Blocking Strategies in Semantica

Blocking is the first stage of the deduplication pipeline. The goal is to partition an entity collection into smaller subsets (blocks) where potential duplicates are guaranteed to co-occur, reducing the complexity from O(n²) to O(∑|block|²).

Simple Blocking

The SimpleBlocker class in [blocking.py](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/blocking.py) provides deterministic key-based partitioning. Users supply a key function that extracts a coarse attribute from each entity.

from semantica.deduplication.blocking import SimpleBlocker

blocker = SimpleBlocker(key=lambda e: e["name"][0].lower())
blocks = blocker.partition(entities)  # {"a": [...], "b": [...], ...}

The blocker returns a dictionary mapping keys to entity lists. Each block contains all entities sharing that key, ensuring no cross-block duplicates are missed while keeping block sizes manageable.

Advanced Blocking Options

Semantica's blocking module supports multiple strategies:

• Hash-based blocking – uses deterministic hashing of normalized identifiers
• Phonetic blocking – applies Soundex or Metaphone encodings to group similar-sounding names
• Composite blocking – combines multiple keys with hierarchical partitioning

---

Similarity Scoring Architecture

The [similarity_calculator.py](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/similarity_calculator.py) module implements the SimilarityCalculator class, which computes pairwise similarity scores within blocks.

Multi-Metric Scoring

The calculator accepts a list of scorer components with optional weights, enabling domain-specific similarity definitions:

from semantica.deduplication.similarity_calculator import SimilarityCalculator

scorer = SimilarityCalculator(
    scorers=["jaro_winkler", "cosine"],
    weights=[0.6, 0.4],
)

Each scorer returns a float in [0, 1], and the weighted combination produces the final similarity score.

Available Scorers

Scorer	Description	Use Case
jaro_winkler	Edit-distance variant favoring prefix matches	Name variations, typos
levenshtein	Classic edit distance	Character-level differences
cosine	TF-IDF cosine similarity over token sets	Descriptive text fields
semantic_v2	Domain-specific embeddings	Semantic equivalence beyond surface form

Pairwise Comparison Loop

For each block, the system performs exhaustive pairwise comparisons:

pair_scores = []
for block in blocks.values():
    for i, e1 in enumerate(block):
        for e2 in block[i + 1:]:
            score = scorer.compare(e1, e2)
            if score >= 0.85:
                pair_scores.append((e1, e2, score))

The threshold (here 0.85) is user-configurable and controls the precision-recall tradeoff.

---

Entity Merging and Provenance

Confirmed duplicates proceed to the [entity_merger.py](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/entity_merger.py) module, which resolves conflicts and produces canonical entities.

Core Merging Logic

from semantica.deduplication.entity_merger import EntityMerger

merger = EntityMerger(provenance=True)
merged = merger.merge(e1, e2)

With provenance=True, the merger records:

• Original source entity IDs
• Merge timestamp and operation type
• Conflict resolution decisions

This enables downstream auditing and potential rollback operations.

Detect vs. Merge Modes

The merger supports two operational modes:

• Detection mode – returns duplicate groups without modifying entities (dry-run analysis)
• Apply mode – performs the merge and emits canonical entities

---

High-Level Orchestration with DuplicateDetector

The [duplicate_detector.py](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/duplicate_detector.py) module provides the unified DuplicateDetector class that orchestrates the entire pipeline.

Configuration-Based Initialization

from semantica.deduplication import DuplicateDetector

detector = DuplicateDetector(
    blocking_strategy="simple",
    similarity_strategy="semantic_v2",
    min_similarity=0.85,
)

Full Pipeline Execution

# Detect duplicate groups

duplicates = detector.detect(entity_collection)

# Optional: apply merges

canonical_entities = detector.merge(duplicates)

The detect() method returns a list of duplicate groups, where each group contains entity pairs with their similarity scores. The merge() method consumes these groups and produces the final deduplicated output.

---

CLI Integration

Semantica exposes deduplication through the command line:

semantica deduplicate \
    --input entities.jsonl \
    --strategy semantic_v2 \
    --blocking simple \
    --min-similarity 0.9 \
    --dry-run

The CLI forwards arguments to DuplicateDetector, supporting both analysis (--dry-run) and live execution modes. See [tests/test_cli_commands.py](https://github.com/semantica-agi/semantica/blob/main/semantica/tests/test_cli_commands.py) for usage examples.

---

Complete Working Example

Below is a runnable end-to-end example combining all pipeline stages:

from semantica.deduplication import DuplicateDetector
from semantica.deduplication.blocking import SimpleBlocker
from semantica.deduplication.similarity_calculator import SimilarityCalculator
from semantica.deduplication.entity_merger import EntityMerger

# Sample entity collection

entities = [
    {"id": "e1", "name": "Acme Corporation", "domain": "acme.com"},
    {"id": "e2", "name": "ACME Corp", "domain": "acme.com"},
    {"id": "e3", "name": "Global Industries", "domain": "global.com"},
]

# High-level approach

detector = DuplicateDetector(
    blocking_strategy="simple",
    similarity_strategy="jaro_winkler",
    min_similarity=0.8,
)

duplicates = detector.detect(entities)
print(f"Found {len(duplicates)} duplicate groups")

# Low-level component approach

blocker = SimpleBlocker(key=lambda e: e["domain"].split(".")[0])
blocks = blocker.partition(entities)

scorer = SimilarityCalculator(
    scorers=["jaro_winkler", "cosine"],
    weights=[0.7, 0.3],
)

merger = EntityMerger(provenance=True)
merged_entities = []

for block in blocks.values():
    for i, e1 in enumerate(block):
        for e2 in block[i + 1:]:
            if scorer.compare(e1, e2) >= 0.8:
                merged = merger.merge(e1, e2)
                merged_entities.append(merged)

---

Summary

• Blocking in Semantica uses the SimpleBlocker class with configurable key functions to partition entities into comparison blocks, implemented in [blocking.py](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/blocking.py)
• Similarity scoring employs the SimilarityCalculator with pluggable metrics (Jaro-Winkler, cosine, semantic embeddings) in [similarity_calculator.py](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/similarity_calculator.py)
• Merging via EntityMerger resolves conflicts with full provenance tracking in [entity_merger.py](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/entity_merger.py)
• Orchestration through DuplicateDetector unifies all stages with both programmatic and CLI interfaces in [duplicate_detector.py](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/duplicate_detector.py)

---

Frequently Asked Questions

What is entity blocking and why is it necessary?

Entity blocking partitions a dataset into smaller groups based on shared coarse attributes (like first letters, domain names, or phonetic codes). It is necessary because comparing every entity pair in large datasets is computationally infeasible—blocking reduces comparisons from O(n²) to manageable subsets while guaranteeing true duplicates remain in the same block for evaluation.

How does Semantica compute similarity scores between entities?

Semantica's SimilarityCalculator combines multiple token-level and semantic metrics through weighted averaging. Each configured scorer returns a [0, 1] score, and the final similarity is the weighted sum. Users can customize the scorer list, weights, and similarity threshold to match their domain's precision-recall requirements.

Can I use custom similarity functions in Semantica?

Yes. The SimilarityCalculator accepts any object implementing the scorer interface. You can register custom scorers by name in the configuration or pass callable objects directly. The architecture in [similarity_calculator.py](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/similarity_calculator.py) is designed for extensibility without modifying core library code.

What happens to entity provenance after merging?

When EntityMerger operates with provenance=True, it preserves the complete history of each merge operation including source entity IDs, timestamps, and conflict resolution decisions. This metadata attaches to the canonical entity, enabling audit trails, downstream lineage tracking, and potential future unmerge operations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}