How Semantica's DuplicateDetector Performs Semantic Entity Resolution at Scale

Semantica resolves duplicate entities by combining semantic similarity scoring, blocking-based pruning, and union-find grouping to handle millions of entities efficiently.

The semantica-agi/semantica repository implements a high-performance duplicate detection system designed for large-scale knowledge graphs. By leveraging multi-factor similarity metrics and intelligent blocking strategies, Semantica's DuplicateDetector enables accurate semantic entity resolution without the prohibitive costs of naive pairwise comparison.

Blocking and Prefiltering: Reducing O(N²) Complexity

Before computing expensive semantic similarities, the system applies a blocking strategy to eliminate obviously unrelated pairs. In semantica/deduplication/similarity_calculator.py, the SimilarityCalculator._prefilter_pair method implements cheap prefix-based filtering that discards candidates based on type mismatch, name-length ratio, and token-overlap thresholds.

This prefiltering step dramatically reduces the search space from O(N²) to a manageable subset, allowing the pipeline to scale to millions of entities without comparing every possible pair.

Multi-Factor Semantic Similarity Calculation

For candidate pairs that pass the blocking phase, the SimilarityCalculator aggregates four weighted similarity components into a composite score. The weights are configurable and automatically normalized.

String and Embedding Similarity

The calculator measures string similarity using Levenshtein distance, Jaro-Winkler similarity, and cosine similarity on token vectors. It also computes embedding similarity as the cosine distance between entity vector embeddings, capturing semantic meaning beyond surface-level text.

Property and Relationship Matching

Beyond names, the system evaluates property similarity by comparing matching property keys and values, and relationship similarity using Jaccard similarity on relationship sets. This ensures that entities with similar connections and attributes receive higher scores, even if their display names differ slightly.

Confidence Scoring and Threshold Filtering

After raw similarity calculation, DuplicateDetector._create_duplicate_candidate in semantica/deduplication/duplicate_detector.py applies confidence boosting logic. The system adds boosts for exact name matches, matching properties, and type equality, then caps the final confidence score at 1.0.

Candidates falling below the user-defined confidence_threshold are filtered out immediately, ensuring only high-fidelity duplicates proceed to the grouping phase.

Scalable Processing Modes: Batch and Incremental

The DuplicateDetector supports two distinct processing patterns optimized for different data ingestion scenarios.

Batch Detection with Vectorized Operations

The detect_duplicates method utilizes SimilarityCalculator.batch_calculate_similarity to compute all pairwise similarities in a single vectorized pass. This approach maximizes throughput when processing static datasets or performing initial knowledge graph construction.

Incremental Detection for Streaming Data

For streaming ingestion, incremental_detect compares new entities only against the existing set, avoiding the full O(N²) scan. This incremental path reuses the same similarity engine while maintaining constant-time complexity relative to the existing graph size.

Union-Find Grouping and Representative Selection

Duplicate candidates are fed into a union-find (disjoint-set) algorithm via _build_duplicate_groups. This efficiently merges transitive duplicates—where A matches B and B matches C—into single DuplicateGroup objects without requiring direct A-to-C comparisons.

Each group then selects a representative entity through _select_representative, preferring the record with the richest set of properties and relationships. This ensures the most complete canonical record survives the merge process.

Integration with the Knowledge Graph Pipeline

The detector serves as the core engine for the EntityResolver class in semantica/kg/entity_resolver.py. This high-level wrapper plugs the detector into the knowledge-graph construction workflow, automatically deduplicating entities before persistence via the resolve_entities method.

Code Examples

# Basic duplicate detection (batch mode)

from semantica.deduplication import DuplicateDetector

entities = [
    {"id": "1", "name": "Apple Inc.", "type": "Company"},
    {"id": "2", "name": "Apple", "type": "Company"},
    {"id": "3", "name": "Microsoft", "type": "Company"},
]

detector = DuplicateDetector(similarity_threshold=0.8, confidence_threshold=0.7)
candidates = detector.detect_duplicates(entities)        # → List[DuplicateCandidate]

groups = detector.detect_duplicate_groups(entities)    # → List[DuplicateGroup]

# Incremental detection for streaming ingestion

new_entities = [{"id": "4", "name": "Apple Corp"}]
existing = entities

incremental = detector.incremental_detect(new_entities, existing)

# Returns only the pairs that cross the similarity/confidence thresholds

# Using the higher-level EntityResolver (the typical entry point in a KG pipeline)

from semantica.kg import EntityResolver

resolver = EntityResolver(strategy="semantic", similarity_threshold=0.85)
resolved = resolver.resolve_entities(entities)

# `resolved` contains merged entities with the most complete representation

Summary

• Blocking strategies in SimilarityCalculator._prefilter_pair eliminate impossible pairs before expensive similarity calculations.
• Multi-factor similarity combines string, embedding, property, and relationship metrics with configurable weights.
• Dual processing modes support both full-batch vectorized processing and incremental streaming detection.
• Union-find grouping efficiently handles transitive duplicate relationships through _build_duplicate_groups.
• Canonical representative selection preserves the richest entity record via _select_representative.

Frequently Asked Questions

What blocking strategy does Semantica use to reduce comparison costs?

Semantica implements prefix-based blocking in semantica/deduplication/similarity_calculator.py through the _prefilter_pair method. This strategy filters candidates based on type mismatches, name-length ratios, and token-overlap thresholds, reducing the O(N²) comparison space to a manageable subset before semantic scoring begins.

How does the SimilarityCalculator combine different similarity metrics?

The SimilarityCalculator aggregates four weighted components: string similarity (Levenshtein, Jaro-Winkler, cosine), embedding similarity (cosine on vectors), property similarity (key/value matching), and relationship similarity (Jaccard on relationship sets). These weights are configurable and automatically normalized to produce a final composite score between 0 and 1.

What is the difference between batch and incremental detection in Semantica?

Batch detection via detect_duplicates uses batch_calculate_similarity to process all pairwise comparisons in a single vectorized pass, ideal for static datasets. Incremental detection via incremental_detect compares only new entities against existing records, avoiding full O(N²) scans and enabling real-time streaming ingestion with constant-time complexity relative to graph size.

How does Semantica select the canonical representative from a duplicate group?

The _select_representative method in semantica/deduplication/duplicate_detector.py selects the entity with the richest set of properties and relationships as the canonical representative. This ensures that merged DuplicateGroup objects retain the most complete and informative record during the entity resolution process.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}