Entity Resolution at Scale in semantica.deduplication: Architecture and Implementation

semantica.deduplication implements a six-stage pipeline that uses vector-store embeddings, approximate nearest-neighbor search, and embarrassingly parallel clustering to resolve duplicate entities in datasets ranging from thousands to billions of records.

The semantica-agi/semantica library provides a production-grade entity resolution system through its deduplication module. This Python package transforms noisy, duplicate-laden datasets into canonical entity graphs using a loosely-coupled, scalable architecture that integrates with multiple vector stores.

The Six-Stage Entity Resolution Pipeline

1. Feature Extraction

In semantica/deduplication/similarity_calculator.py, the pipeline converts each record into an embedding vector. The system supports text, structured attributes, and optionally multimodal inputs using the vector store’s configured encoder. Embeddings are computed lazily and cached in the vector store, allowing reuse across multiple deduplication runs without recomputation.

2. Candidate Generation

The duplicate_detector.py module performs fast approximate nearest-neighbor (ANN) searches via FAISS, Qdrant, Pinecone, or other backends defined in semantica/vector_store/. For each record, it retrieves a candidate set of potential duplicates by querying the vector index and filtering results to those exceeding a configurable similarity threshold. This sub-linear lookup process can be parallelized across data shards.

3. Clustering

Once candidates are identified, cluster_builder.py and methods.py group them into clusters using graph-based algorithms like ConnectedComponents or Agglomerative clustering. Operating on the sparse candidate graph rather than the full dataset, these methods utilize the parallelism_manager in semantica/pipeline/parallelism_manager.py to distribute computation across multiple CPU cores or distributed workers.

4. Merge-Strategy Selection

The merge_strategy.py file defines pure-function strategies including KeepFirst, PreferMostRecent, and CustomResolver. These strategies determine which attributes survive the merge when collapsing a cluster into a single canonical entity. Because strategies are stateless pure functions, they can be applied independently to each cluster, enabling embarrassingly parallel execution.

5. Entity Merging

The entity_merger.py module executes the selected strategy to create a canonical entity while recording comprehensive provenance. It writes merge metadata—including original IDs, timestamps, and confidence scores—via deduplication_provenance.py, creating an auditable trail of how each canonical record was formed.

6. Registry and Persistence

Finally, registry.py registers the canonical entities in the graph, updates the vector store embeddings, and persists the ID mapping to a deduplication index. The registry abstracts underlying storage systems (SQL, NoSQL, or in-memory), allowing the same code path to operate in batch jobs or streaming services.

Scalability Mechanisms

Vector-Store Backend Abstraction

The deduplication layer remains agnostic to specific vector databases. By interacting through the semantica/vector_store/ interface, the system supports FAISS, Qdrant, Pinecone, Weaviate, and Milvus. Switching backends requires only configuration changes, enabling teams to select infrastructure that matches their latency and volume requirements.

Parallelism Management

semantica/pipeline/parallelism_manager.py orchestrates multi-process execution across the detector, clustering, and merging stages. The manager automatically partitions the candidate graph and balances workloads across available compute resources, whether on a single multi-core machine or a distributed cluster.

Lazy Provenance Recording

Instead of writing individual audit records for every merge operation, deduplication_provenance.py batches provenance updates and writes them in bulk. This dramatically reduces I/O overhead when processing high-volume datasets.

Chunked Processing via Resource Scheduling

The pipeline accepts data streams through the resource scheduler (semantica/pipeline/resource_scheduler.py). This design enables out-of-core processing of datasets that exceed available memory, resolving each chunk independently before merging results into the global identity map.

Configurable Similarity Thresholds

The config.py file allows per-attribute tuning of similarity cutoffs. By aggressively pruning the candidate space for high-cardinality fields, the system reduces the size of the candidate graph and subsequent clustering workload.

Implementation Example

The following example demonstrates a complete deduplication workflow using a pandas DataFrame:

import pandas as pd
from semantica.deduplication.registry import DeduplicationRegistry
from semantica.deduplication.methods import ConnectedComponents
from semantica.vector_store.faiss_store import FaissVectorStore
from semantica.deduplication.merge_strategy import KeepFirst

# 1️⃣ Load raw data

raw_df = pd.read_csv("data/raw_entities.csv")

# 2️⃣ Initialise vector store (FAISS in this example)

vector_store = FaissVectorStore(dim=768)        # embeddings are 768‑dimensional

# 3️⃣ Register the dataset – this creates embeddings and stores them

registry = DeduplicationRegistry(vector_store=vector_store)
registry.index_documents(raw_df, text_column="description")

# 4️⃣ Detect duplicate candidates

detector = registry.duplicate_detector
candidate_graph = detector.build_candidate_graph(threshold=0.85)

# 5️⃣ Cluster candidates

clusterer = ConnectedComponents(similarity_threshold=0.85)
clusters = clusterer.cluster(candidate_graph)

# 6️⃣ Resolve each cluster

merger = registry.entity_merger
for cluster in clusters:
    canonical = merger.merge_cluster(
        cluster,
        strategy=KeepFirst(),          # simple “first‑seen wins” strategy

        provenance=True                # keep full provenance

    )
    print(f"✔️  Merged {len(cluster)} records → {canonical.id}")

# 7️⃣ Persist the resolved entities back to the vector store

registry.persist_canonical_entities()

Step-by-step explanation:

• Step 2: Creates a FAISS index where embeddings are inserted lazily during document indexing.
• Step 4: build_candidate_graph performs cosine similarity nearest-neighbor searches, retaining only pairs exceeding the 0.85 threshold.
• Step 5: ConnectedComponents treats the candidate pairs as an undirected graph and extracts connected components as duplicate clusters.
• Step 6: merge_cluster applies the KeepFirst strategy, preserving the first record's attributes while recording full provenance metadata.
• Step 7: Writes resolved canonical vectors back to the index, replacing noisy originals.

Key Source Files

• semantica/deduplication/entity_merger.py – Core logic for applying merge strategies and writing provenance.
• semantica/deduplication/duplicate_detector.py – Fast nearest-neighbor search and candidate graph construction.
• semantica/deduplication/cluster_builder.py – Orchestrates clustering algorithms over the candidate graph.
• semantica/deduplication/merge_strategy.py – Definitions of built-in merge policies.
• semantica/deduplication/methods.py – Implementations of ConnectedComponents and Agglomerative clustering.
• semantica/deduplication/registry.py – High-level API wiring together vector-store, detector, clusterer, and merger.
• semantica/deduplication/deduplication_provenance.py – Provenance model recording source IDs, timestamps, and confidence scores.
• semantica/pipeline/parallelism_manager.py – Multi-process and distributed execution management.
• semantica/vector_store/ – Backend adapters for FAISS, Qdrant, Pinecone, and other vector databases.

Summary

• semantica.deduplication uses a six-stage pipeline: feature extraction, candidate generation, clustering, strategy selection, merging, and persistence.
• The system achieves scale through vector-store abstraction, parallel processing via parallelism_manager.py, and lazy provenance batching.
• Clustering algorithms operate on sparse candidate graphs rather than full record sets, enabling billion-record scalability.
• Merge strategies are implemented as pure functions in merge_strategy.py, allowing embarrassingly parallel execution across clusters.
• Full audit trails are maintained through deduplication_provenance.py with batched I/O for performance.

Frequently Asked Questions

How does semantica.deduplication handle billions of records?

The system combines chunked processing via the resource scheduler with vector-store backends that provide sub-linear approximate nearest-neighbor search. The candidate graph remains sparse relative to the total dataset size, and the parallelism_manager.py distributes both clustering and merging across distributed workers.

Which vector databases are supported?

The semantica/vector_store/ package provides adapters for FAISS, Qdrant, Pinecone, Weaviate, and Milvus. The deduplication logic is backend-agnostic, allowing configuration-driven switching without code changes.

How is merge provenance tracked?

The deduplication_provenance.py module records original entity IDs, merge timestamps, similarity scores, and strategy used for every canonical entity. These records are batched and written in bulk to minimize I/O overhead while maintaining full auditability.

Can I implement custom merge strategies?

Yes. The merge_strategy.py interface accepts custom implementations as long as they conform to the pure-function contract. A custom resolver receives the cluster of duplicate records and returns a single canonical entity, with access to all attributes and metadata for sophisticated conflict resolution.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}