What Algorithms Power Semantic Deduplication and Entity Resolution in Semantica?

Semantica uses a multi-stage pipeline combining vector-based cosine similarity, fuzzy string matching, agglomerative hierarchical clustering, and rule-based entity merging for semantic deduplication and entity resolution.

The semantica-agi/semantica repository implements a sophisticated deduplication engine designed for knowledge graphs. This system processes semantic triples (subject-predicate-object) and entity records to identify duplicates across variations in naming, structure, and provenance. The architecture separates similarity computation, clustering detection, and merge resolution into distinct, composable components.

Core Algorithmic Components

SimilarityCalculator: Multi-Modal Similarity Scoring

The SimilarityCalculator class in semantica/deduplication/similarity_calculator.py computes pairwise similarity using three complementary techniques:

• Cosine similarity on dense embeddings (OpenAI, Cohere, or custom embedding models)
• Jaccard similarity on token-level n-grams for surface-level text overlap
• Levenshtein distance as a fallback for typos and minor string variations

This multi-modal approach ensures robust matching across semantic (meaning-based) and syntactic (string-based) duplicate types.

DuplicateDetector: Clustering Algorithms

The DuplicateDetector class in semantica/deduplication/duplicate_detector.py implements two primary clustering strategies:

Algorithm	Use Case	Implementation Detail
Agglomerative hierarchical clustering (single-link)	General-purpose duplicate grouping	Builds dendrogram from similarity matrix, cuts at threshold
DBSCAN-style density-based clustering	Threshold-supplied detection	Identifies core samples and expands clusters by density

The detector supports three operational modes:

• detect — Returns duplicate pairs without merging
• merge — Full pipeline: detection → clustering → merging
• magic (default) — Auto-selects threshold and strategy based on data characteristics

EntityMerger: Rule-Based Canonicalization

The EntityMerger in semantica/deduplication/entity_merger.py resolves duplicate groups into canonical entities using configurable strategies:

• keep-first — Preserve the earliest encountered record
• keep-most-recent — Select by latest timestamp
• custom-merge — User-defined resolution function

The merger prioritizes metadata completeness (highest provenance confidence, most populated fields) when no explicit strategy overrides apply.

DeduplicationProvenance: Graph-Based Audit Trail

The DeduplicationProvenance class in semantica/deduplication/deduplication_provenance.py maintains a provenance graph recording which source records contributed to each merged entity. This enables downstream reasoning to consider merge history and supports deterministic re-execution.

Practical Implementation

Detecting Duplicate Triples

from semantica.deduplication import DuplicateDetector

detector = DuplicateDetector(
    similarity_threshold=0.85,      # cosine similarity cutoff

    mode="semantic_v2",             # enables token-level + fuzzy fallback

)

duplicate_pairs = detector.detect(triples)   # triples: list of (s, p, o) dicts

print(duplicate_pairs)                     # → [(0, 3), (2, 5), ...]

Merging with Custom Strategy

from semantica.deduplication.entity_merger import EntityMerger

def keep_most_recent(old, new):
    return new if new["timestamp"] > old["timestamp"] else old

merger = EntityMerger(strategy=keep_most_recent)
canonical_entities = merger.merge(duplicate_pairs, triples)

Full Pipeline Execution

# Magic mode: auto-configured detection and merging

merged_graph = detector.merge(triples)

Source File Architecture

File Path	Responsibility
semantica/deduplication/similarity_calculator.py	Embedding cosine, Jaccard, Levenshtein similarity metrics
semantica/deduplication/duplicate_detector.py	Hierarchical and DBSCAN clustering; mode orchestration
semantica/deduplication/entity_merger.py	Canonicalization strategies and conflict resolution
semantica/deduplication/deduplication_provenance.py	Provenance graph construction and merge audit logging
semantica/deduplication/registry.py	Plugin registration for custom methods and strategies

Summary

• Vector + string hybrid similarity handles both semantic and surface-level duplicates via SimilarityCalculator
• Hierarchical and density-based clustering groups duplicates flexibly through DuplicateDetector
• Configurable merge strategies resolve conflicts with provenance-aware canonicalization in EntityMerger
• Complete audit trails via provenance graphs ensure reproducibility and downstream reasoning support

The modular design allows domain-specific tuning—such as custom embedding models for company name variants or predicate synonym mappings—without modifying core algorithmic code.

Frequently Asked Questions

How does Semantica handle semantic versus syntactic duplicates?

Semantica addresses both types through layered similarity metrics. Cosine similarity on embeddings captures semantic equivalence (e.g., "Apple Inc." and "Apple Computer"), while Jaccard n-gram overlap and Levenshtein distance catch syntactic variations like typos and abbreviations. The semantic_v2 mode automatically combines these signals with learned or configured weights.

What clustering algorithm should I choose for large-scale datasets?

For large datasets, DBSCAN-style clustering in DuplicateDetector offers O(n log n) approximate performance with index acceleration, compared to O(n²) for full hierarchical clustering. Use DBSCAN when you can define a clear similarity threshold; fallback to agglomerative clustering for smaller datasets or when cluster granularity requires fine control via dendrogram cutting.

Can I implement custom similarity metrics or merge logic?

Yes. The registry.py module exposes extension points: register custom similarity functions via SimilarityCalculator.register_metric() and merge strategies via EntityMerger.register_strategy(). Both accept callable objects following the documented signatures in semantica/deduplication/registry.py.

How does provenance tracking affect storage overhead?

The DeduplicationProvenance graph stores merge operations as annotated edges between source and canonical entities. Overhead scales with merge density rather than raw entity count—typically 15-30% additional storage for heavily deduplicated knowledge graphs. The implementation uses compressed adjacency lists and optional serialization to Parquet for long-term archival.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}