How to Implement Entity Deduplication Using Semantic Similarity in Semantica

Use Semantica's DuplicateDetector to compute pairwise semantic similarities, generate DuplicateCandidate records above configurable thresholds, and merge transitive duplicate groups via union-find—supporting both full-batch and incremental detection workflows.

Entity deduplication using semantic similarity is essential for maintaining clean knowledge graphs, especially when ingesting data from multiple sources with inconsistent naming conventions. The Semantica framework provides a production-ready deduplication engine that combines vector-based similarity with configurable confidence scoring and efficient grouping algorithms. This guide walks through the complete implementation based on the actual source code in semantica-agi/semantica.

Core Deduplication Workflow

The deduplication engine follows a three-stage pipeline implemented across semantica/deduplication/duplicate_detector.py and semantica/deduplication/similarity_calculator.py.

Stage 1: Compute Pairwise Similarity

The SimilarityCalculator class generates similarity scores for entity pairs using either vector embeddings or string-based methods. To avoid quadratic scaling, the calculator optionally applies blocking—partitioning entities by shared attributes (type, country, etc.) to limit comparisons.

Key methods in similarity_calculator.py:

• batch_calculate_similarity() – vector-based similarity for large batches
• calculate_similarity() – single pair comparison
• calculate_string_similarity() – edit-distance or fuzzy matching fallback

Stage 2: Create Duplicate Candidates

When a pair exceeds similarity_threshold, the DuplicateDetector instantiates a DuplicateCandidate. Confidence scores are boosted by auxiliary signals:

• Exact name match
• Shared properties
• Same entity type

Stage 3: Group Duplicates Transitively

Candidates are merged into groups using _build_duplicate_groups(), which implements a union-find (disjoint set) algorithm. Each group receives:

• Aggregated confidence score
• A representative entity—the member with the richest property/relationship set

Configuration Parameters

The DuplicateDetector in duplicate_detector.py exposes these tunable parameters:

Parameter	Purpose	Typical Range
similarity_threshold	Minimum similarity to consider a pair	0.7 – 0.9
confidence_threshold	Minimum confidence to retain a candidate	0.6 – 0.8
min_similarity	Additional floor after similarity computation	None or [0.0, 1.0]
top_k_per_entity	Max candidates each entity can sponsor	None or integer
max_results	Global cap on returned candidates	None or integer
sort_by	Ranking field (confidence or similarity_score)	confidence (default)

These values are validated against defaults stored in semantica/deduplication/config.py.

Basic Duplicate Detection

This example demonstrates standard batch processing against a static entity collection:

from semantica.deduplication import DuplicateDetector

# Entities require at minimum "id" and "name" fields

entities = [
    {"id": "1", "name": "Apple Inc.", "properties": {"country": "US"}},
    {"id": "2", "name": "Apple", "properties": {"country": "US"}},
    {"id": "3", "name": "Microsoft", "properties": {"country": "US"}},
]

detector = DuplicateDetector(
    similarity_threshold=0.8,
    confidence_threshold=0.7,
    top_k_per_entity=5,
    max_results=10,
)

candidates = detector.detect_duplicates(entities)
for cand in candidates:
    print(
        f"Duplicate: {cand.entity1['name']} ↔ {cand.entity2['name']}, "
        f"sim={cand.similarity_score:.2f}, conf={cand.confidence:.2f}, "
        f"reasons={cand.reasons}"
    )

The reasons attribute lists which signals contributed to the confidence boost.

Creating and Inspecting Duplicate Groups

For downstream merge operations, group candidates transitively:

groups = detector.detect_duplicate_groups(entities)
for grp in groups:
    rep = grp.representative
    print(
        f"Group of {len(grp.entities)} entities, confidence={grp.confidence:.2f}, "
        f"representative={rep['name'] if rep else 'None'}"
    )

The representative selection logic prioritizes entities with the most complete property sets and relationship connections.

Incremental Detection for Streaming Workflows

When new entities arrive continuously, avoid reprocessing the entire graph using incremental_detect():

new_entities = [
    {"id": "4", "name": "Apple Corp", "properties": {"country": "US"}},
]

existing_entities = entities  # Previously deduplicated collection

incremental_candidates = detector.incremental_detect(new_entities, existing_entities)
print(f"Found {len(incremental_candidates)} new duplicate candidates.")

This method runs in O(N × M) time for N new entities against M existing entities—far more efficient than full O((N+M)²) pairwise comparison.

Post-Detection: Merging Strategies

After detection, the deduplication pipeline supports automated entity resolution through components in semantica/deduplication/merge_strategy.py and semantica/deduplication/entity_merger.py:

• Merge strategy selection – choose between overwrite, preserve-richer, or custom field-level merging
• Entity merger execution – applies the selected strategy to produce canonical records

The registry.py file registers these components for dynamic discovery by the broader Semantica framework, enabling plugin-style extension.

Summary

• Entity deduplication using semantic similarity in Semantica centers on the DuplicateDetector class with configurable thresholds and confidence scoring.
• The three-stage workflow—similarity computation, candidate generation, and transitive grouping—handles both batch and incremental detection.
• Key source files: duplicate_detector.py (orchestration), similarity_calculator.py (scoring), config.py (defaults), merge_strategy.py and entity_merger.py (resolution), and registry.py (framework integration).
• Use incremental_detect() for streaming data to maintain efficiency at scale.

Frequently Asked Questions

What similarity methods does Semantica support for entity deduplication?

Semantica supports both vector-based similarity (embeddings computed via batch_calculate_similarity or calculate_similarity) and string-based similarity (fuzzy matching via calculate_string_similarity) in similarity_calculator.py. The DuplicateDetector automatically selects appropriate methods based on available entity features.

How does the union-find grouping algorithm handle ambiguous duplicate chains?

The _build_duplicate_groups() method in duplicate_detector.py treats duplicate detection as a connected components problem: if A matches B and B matches C, all three form one group regardless of whether A directly matches C. This transitive closure ensures no duplicate escapes detection through intermediate entities.

Can I customize how representative entities are selected during grouping?

Yes. The representative selection logic evaluates entity richness—count of properties and relationships—and is designed for override. For custom strategies, extend the DuplicateDetector class or modify the grouping pipeline before the EntityMerger in entity_merger.py applies your preferred MergeStrategy.

What performance should I expect for large knowledge graphs?

For N entities, full detection is O(N²) without blocking, reduced significantly by attribute-based blocking. The incremental_detect() method achieves O(N × M) for batch inserts against existing graphs. Configure top_k_per_entity and max_results to enforce hard memory and latency bounds in production deployments.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}