How to Implement Semantic Entity Deduplication in Semantica: A Complete Guide

Use the DuplicateDetector class in semantica.deduplication with configurable similarity thresholds and multi-factor scoring to identify, rank, and group duplicate entities using string, property, relationship, and embedding-based comparison.

Semantic entity deduplication in Semantica provides a production-ready pipeline for identifying and merging duplicate records in knowledge graphs and entity datasets. Whether you're cleaning customer data, consolidating research databases, or building a unified semantic layer, the semantica-agi/semantica repository offers fine-grained control over every stage of the deduplication process—from similarity calculation to final group formation and merging.

Core Architecture: Three Integrated Components

The semantic entity deduplication system resides in the semantica.deduplication package and consists of three tightly-coupled components:

Component	Responsibility
SimilarityCalculator (similarity_calculator.py)	Computes multi-factor similarity scores between entity pairs
DuplicateDetector (duplicate_detector.py)	Orchestrates detection, builds candidate lists, and forms duplicate groups
EntityMerger (entity_merger.py)	Merges duplicate groups into canonical representative entities

Each component can be configured independently, allowing you to optimize for precision-sensitive or recall-sensitive use cases.

Step 1: Configure Multi-Factor Similarity Scoring

The SimilarityCalculator.calculate_similarity method (lines 31-44 in similarity_calculator.py) aggregates four similarity components into a weighted composite score:

# Weighted aggregation from similarity_calculator.py

weights = {
    "string": self.string_weight,
    "property": self.property_weight,
    "relationship": self.relationship_weight,
    "embedding": self.embedding_weight if embedding_score > 0 else 0.0,
}

# Normalized weights → overall_score = Σ(component × weight)

Similarity Components Explained

• String similarity – Choose from Jaro-Winkler, Levenshtein distance, or cosine similarity over character bigrams via _jaro_winkler_similarity, _levenshtein_similarity, or _cosine_similarity
• Property similarity – Token-wise string comparison for matching keys, neutral handling of missing values (0.5), full credit for exact matches
• Relationship similarity – Jaccard similarity over hashable relationship tuples using _make_hashable
• Embedding similarity – Cosine similarity on pre-computed vector embeddings for deep semantic matching

A fast prefilter (_prefilter_pair, lines 62-78) eliminates obviously unrelated pairs before expensive multi-factor scoring runs.

Step 2: Detect Duplicate Candidates with Thresholds and Limits

The DuplicateDetector.detect_duplicates method drives the core deduplication flow through three phases:

from semantica.deduplication import DuplicateDetector

detector = DuplicateDetector(
    similarity_threshold=0.8,    # Minimum similarity to consider a pair

    confidence_threshold=0.75,   # Minimum confidence for a candidate

    top_k_per_entity=5,          # Keep up to 5 candidates per entity

    max_results=200,             # Global cap on returned candidates

    sort_by="confidence",        # Rank by confidence or similarity_score

)

entities = [
    {"id": "1", "name": "Apple Inc.", "type": "Company"},
    {"id": "2", "name": "Apple", "type": "Company"},
    {"id": "3", "name": "Microsoft", "type": "Company"},
]

candidates = detector.detect_duplicates(entities)
for cand in candidates:
    print(f"Duplicate pair: {cand.entity1['id']} ↔ {cand.entity2['id']}")
    print(f"  similarity={cand.similarity_score:.2f}, confidence={cand.confidence:.2f}")

Detection Pipeline Internals

1. Batch similarity computation – SimilarityCalculator.batch_calculate_similarity generates candidate pairs via blocking (_build_block_indexes, _generate_candidate_pairs) with optional capping (_cap_candidate_pairs)
2. Candidate creation – _create_duplicate_candidate (lines 560-618) evaluates exact name matches, property matches, and type matches to compute a composite confidence score
3. Result limiting – _apply_result_limits (lines 668-710) filters by min_similarity, sorts by confidence or similarity, applies top_k_per_entity and max_results caps

Step 3: Form Transitive Duplicate Groups with Union-Find

After detecting pairwise candidates, detect_duplicate_groups builds transitive clusters using a union-find algorithm:

groups = detector.detect_duplicate_groups(entities)
for g in groups:
    ids = [e["id"] for e in g.entities]
    print(f"Group ({len(ids)} entities): {ids}")
    print(f"  representative: {g.representative['id']}")
    print(f"  group confidence: {g.confidence:.2f}")

The _build_duplicate_groups implementation:

• Maps each entity ID to a group reference
• Merges groups when a candidate bridges two existing groups
• Calculates group confidence from average pairwise similarity, boosted for larger groups (_calculate_group_confidence, lines 667-678)
• Selects the most complete entity as representative—the one with the richest property/relationship set (_select_representative, lines 780-891)

Step 4: Handle Streaming Data with Incremental Detection

For production pipelines that process data in batches, incremental_detect (lines 445-511) avoids recomputing the full O(N²) matrix:

new_entities = [{"id": "4", "name": "Apple Corp.", "type": "Company"}]
existing_entities = entities  # Previously processed corpus

incremental_candidates = detector.incremental_detect(new_entities, existing_entities)
print(f"Found {len(incremental_candidates)} new duplicate candidates.")

This compares only the new batch against the existing corpus, reducing computational cost from quadratic to linear in the batch size.

Step 5: Deduplicate Semantic Relationships

Beyond entities, Semantica supports relationship deduplication via detect_relationship_duplicates:

relationships = [
    {"subject": "Apple", "predicate": "founded_by", "object": "Steve Jobs"},
    {"subject": "Apple", "predicate": "established_by", "object": "Steve Jobs"},
]

duplicates = detector.detect_relationship_duplicates(
    relationships,
    relationship_dedup_mode="semantic_v2",
    predicate_synonym_map={"founded_by": "established_by"},
    literal_normalization_enabled=True,
)

In semantic_v2 mode (lines 480-496), the method:

• Normalizes predicates using a predicate synonym map
• Applies optional literal normalization
• Hashes the normalized (subject, predicate, object) triple for comparison

Legacy mode falls back to simple string similarity on predicates.

Key Configuration Parameters

Parameter	Default	Effect
similarity_threshold	0.75	Minimum composite similarity for pair consideration
confidence_threshold	0.7	Minimum confidence score for candidate inclusion
string_weight	0.4	Weight for string similarity component
embedding_weight	0.3	Weight for embedding similarity (0 if no embeddings)
property_weight	0.2	Weight for property overlap similarity
relationship_weight	0.1	Weight for relationship Jaccard similarity
blocking_strategy	"standard"	Candidate pair generation strategy for scalability

All parameters are exposed through the DuplicateDetector constructor and SimilarityCalculator configuration dictionary in semantica/deduplication/config.py.

Summary

• Semantic entity deduplication in Semantica combines four similarity signals—string, property, relationship, and embedding—into configurable composite scores (semantica/deduplication/similarity_calculator.py)
• Threshold-based filtering and ranking via DuplicateDetector.detect_duplicates provides precise control over precision-recall tradeoffs
• Union-find group formation builds transitive duplicate clusters with confidence-weighted representatives (detect_duplicate_groups)
• Incremental detection (incremental_detect) enables efficient streaming pipelines without full reprocessing
• Relationship deduplication with synonym-aware normalization handles semantic predicate variants
• Full configurability of weights, thresholds, and blocking strategies adapts the pipeline to diverse data quality scenarios

Frequently Asked Questions

How does Semantica handle missing or incomplete entity properties during deduplication?

The SimilarityCalculator treats missing values neutrally by assigning a 0.5 similarity score for property comparisons involving absent keys. This prevents missing data from heavily penalizing otherwise similar entities while still rewarding exact matches with full credit. The neutral handling is implemented in the property similarity calculation within similarity_calculator.py.

What embedding models work with Semantica's embedding similarity component?

Semantica accepts pre-computed embeddings as entity properties and computes cosine similarity between vectors. The system is embedding-model-agnostic—you can supply vectors from sentence-transformers, OpenAI embeddings, domain-specific models, or custom fine-tuned representations. The embedding similarity weight automatically zeroes out when no embeddings are present for either entity in a pair.

Can I customize the blocking strategy for very large datasets?

Yes. The batch_calculate_similarity method exposes blocking through _build_block_indexes and _generate_candidate_pairs, with optional candidate capping via _cap_candidate_pairs. You can configure blocking keys (e.g., phonetic codes, locality-sensitive hashes) to reduce candidate pairs from O(N²) to manageable subsets. For extreme scale, combine blocking with incremental_detect to process data in bounded-size batches.

How does Semantica select which entity becomes the group representative?

The _select_representative method (lines 780-891 in duplicate_detector.py) chooses the entity with the most complete property and relationship set as the canonical representative. "Completeness" is measured by counting non-null properties and outgoing relationships, ensuring the representative preserves maximum information from the duplicate group.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}