How Semantica Handles Entity Resolution and Deduplication at Scale

Semantica employs a multi-stage, highly-parallel pipeline that combines blocking indexes, multi-factor similarity scoring, and union-find algorithms to resolve and deduplicate entities across massive knowledge graphs with linear-ith scalability.

Entity resolution and deduplication at scale presents significant computational challenges for knowledge graph systems. The semantica-agi/semantica repository implements a sophisticated, configurable framework that processes millions to billions of entities through optimized blocking strategies and parallel similarity computation. This architecture avoids the prohibitive O(n²) complexity of naive pairwise comparison while maintaining high accuracy across diverse entity types.

The Multi-Stage Pipeline Architecture

Semantica’s deduplication engine orchestrates seven distinct stages to transform raw entity comparisons into consolidated duplicate groups. According to the source code in duplicate_detector.py and similarity_calculator.py, the pipeline progresses from rapid pre-filtering through sophisticated similarity analysis to final group consolidation. Each stage is designed to minimize computational overhead while maximizing match quality through configurable thresholds and limits.

The system tracks progress throughout execution using a shared ProgressTracker that reports fine-grained status updates—such as per-100% progress and remaining items—without incurring per-entity overhead. This ensures low-latency monitoring even when processing billions of records.

Pre-Filtering and Blocking Strategies

Before expensive similarity calculations begin, Semantica applies aggressive pre-filtering to eliminate obvious non-matches. In duplicate_detector.py (lines 61-82), the _prefilter_pair function implements fast heuristics including type mismatches, name-length ratio checks, and token-overlap thresholds to reject incompatible pairs in microseconds.

For candidate generation, the system employs blocking indexes configured via similarity_calculator.py (lines 52-71). Entities are partitioned into blocks using strategies such as:

• First-letter blocking: Grouping by initial character
• Token-prefix blocking: Shared n-gram prefixes
• Phonetic Soundex: Encoding names by sound representation

By generating candidate pairs only from entities sharing a block, Semantica reduces the comparison space from O(n²) to a manageable subset, enabling linear-ith-like scalability.

Multi-Factor Similarity Scoring

For each candidate pair passing the blocking stage, the SimilarityCalculator (defined in similarity_calculator.py, lines 71-86) computes a weighted aggregate across four distinct dimensions:

String similarity supports multiple algorithms including Jaro-Winkler, Levenshtein distance, and cosine n-gram similarity, allowing adaptation to different naming conventions and typo patterns.

Property similarity performs pairwise comparison of entity attributes with partial-match handling, ensuring that schema variations do not prevent valid matches.

Relationship similarity calculates Jaccard similarity across hashed relationship sets, capturing structural equivalence in the knowledge graph.

Embedding similarity computes cosine similarity of vector embeddings when present, enabling semantic matching beyond lexical similarity.

Component weights normalize at runtime, allowing flexible tuning via configuration without code changes.

Confidence Filtering and Short-Circuit Optimization

To prevent wasted computation, the pipeline implements aggressive short-circuiting and filtering mechanisms. As implemented in duplicate_detector.py (lines 86-99), if string similarity falls below a baseline threshold and no embeddings are available, the calculation terminates early.

The _apply_result_limits function (lines 760-785) enforces multiple constraint layers:

• Configurable confidence thresholds and min-similarity floors
• Per-entity top-k limits to prevent noisy overload from high-frequency entities
• Global caps on total duplicate candidates to bound memory usage

These limits ensure that even under pathological data distributions, the system maintains predictable resource consumption.

Union-Find Group Formation

Once individual duplicate pairs are identified, Semantica consolidates them into cohesive groups using a union-find algorithm. The _build_duplicate_groups method in duplicate_detector.py (lines 333-385) merges intersecting pairs into DuplicateGroup objects.

Group confidence calculation (lines 395-406 in _calculate_group_confidence) derives from the average pairwise similarity, augmented by a size-based boost that increases confidence for larger clusters. The system selects a representative entity for each group based on the richest property and relationship set, ensuring the canonical version retains maximum information density.

Incremental Detection for Streaming Workloads

For production environments with continuous data ingestion, Semantica provides incremental_detect (lines 49-69 in duplicate_detector.py). This method compares new entities exclusively against an existing knowledge base rather than triggering a full re-run.

The incremental mode reuses the identical similarity engine and respects all configured thresholds and limits, ensuring consistency between batch and streaming processing without sacrificing the performance benefits of the blocking architecture.

Implementation Examples

The following examples demonstrate practical usage of the deduplication API:

# Basic duplicate detection on a list of entity dicts

from semantica.deduplication import DuplicateDetector

entities = [
    {"id": "1", "name": "Apple Inc.", "type": "company"},
    {"id": "2", "name": "Apple", "type": "company"},
    {"id": "3", "name": "Microsoft", "type": "company"},
]

detector = DuplicateDetector(similarity_threshold=0.8, confidence_threshold=0.7)
candidates = detector.detect_duplicates(entities)
print(f"Found {len(candidates)} duplicate pairs")

# Form groups and pick a representative entity

groups = detector.detect_duplicate_groups(entities)
for g in groups:
    print(f"Group of {len(g.entities)} entities – confidence {g.confidence:.2f}")
    print("Representative:", g.representative)

# Incremental detection – new data against an existing knowledge base

new = [{"id": "4", "name": "Apple Corp.", "type": "company"}]
existing = entities
incremental = detector.incremental_detect(new, existing)
print(f"Incremental duplicates: {len(incremental)}")

# Custom blocking and weighting

custom_calc = SimilarityCalculator(
    string_weight=0.5,
    property_weight=0.3,
    embedding_weight=0.2,
    config={"prefilter_enabled": True, "candidate_strategy": "blocking_v2"}
)

detector = DuplicateDetector(
    similarity_threshold=0.75,
    confidence_threshold=0.65,
    config={"similarity": {"candidate_strategy": "blocking_v2", "max_candidates_per_entity": 50}}
)

Summary

• Blocking indexes in similarity_calculator.py reduce the comparison space from O(n²) to linear-ith complexity using configurable strategies like Soundex and token-prefix blocking.
• Multi-factor similarity combines string, property, relationship, and embedding metrics with runtime-normalized weights for flexible matching.
• Short-circuit logic in detect_duplicates eliminates low-probability pairs early, while _apply_result_limits enforces per-entity and global caps to bound resource usage.
• Union-find consolidation via _build_duplicate_groups merges transitive duplicate relationships into canonical DuplicateGroup objects with confidence scoring.
• Incremental detection supports streaming workloads by comparing new entities only against existing knowledge bases without full reprocessing.

Frequently Asked Questions

How does Semantica achieve scalability for billions of entities?

Semantica achieves linear-ith scalability through aggressive blocking in similarity_calculator.py (lines 52-71), which partitions entities into comparison blocks using phonetic hashes or token prefixes. This eliminates the O(n²) pairwise explosion by comparing only co-blocked candidates. Combined with pre-filtering in _prefilter_pair (lines 61-82 of duplicate_detector.py) and early short-circuiting, the system processes billions of records without quadratic degradation.

Which similarity algorithms does Semantica support for entity matching?

The framework supports Jaro-Winkler, Levenshtein distance, and cosine n-gram similarity for string comparison, as implemented in similarity_calculator.py. For structural matching, it uses Jaccard similarity on hashed relationship sets. When vector embeddings are available, the system calculates cosine similarity to capture semantic equivalence. These components are weighted and aggregated at runtime.

What is the difference between detect_duplicates and incremental_detect?

The detect_duplicates method performs full pairwise analysis within a provided dataset, while incremental_detect (lines 49-69 of duplicate_detector.py) optimizes for streaming scenarios by evaluating new entities exclusively against an existing knowledge base. Both methods utilize identical similarity engines and respect the same confidence thresholds, but incremental mode avoids reprocessing historical data.

How does Semantica select representative entities from duplicate groups?

During group formation in _build_duplicate_groups (lines 333-385), the system selects the representative entity based on information richness—specifically choosing the entity with the most comprehensive property and relationship sets. Group confidence scores derive from _calculate_group_confidence (lines 395-406), which averages pairwise similarities and applies a size-based boost to larger clusters.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}