# Entity Resolution at Scale in semantica.deduplication: Architecture and Implementation

> Discover how semantica.deduplication achieves entity resolution at scale using vector-store embeddings, ANN search, and parallel clustering for massive datasets. Learn the architecture and implementation.

- Repository: [Semantica /semantica](https://github.com/semantica-agi/semantica)
- Tags: architecture
- Published: 2026-08-21

---

**semantica.deduplication implements a six-stage pipeline that uses vector-store embeddings, approximate nearest-neighbor search, and embarrassingly parallel clustering to resolve duplicate entities in datasets ranging from thousands to billions of records.**

The `semantica-agi/semantica` library provides a production-grade entity resolution system through its `deduplication` module. This Python package transforms noisy, duplicate-laden datasets into canonical entity graphs using a loosely-coupled, scalable architecture that integrates with multiple vector stores.

## The Six-Stage Entity Resolution Pipeline

### 1. Feature Extraction

In [`semantica/deduplication/similarity_calculator.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/similarity_calculator.py), the pipeline converts each record into an embedding vector. The system supports text, structured attributes, and optionally multimodal inputs using the vector store’s configured encoder. **Embeddings are computed lazily and cached** in the vector store, allowing reuse across multiple deduplication runs without recomputation.

### 2. Candidate Generation

The [`duplicate_detector.py`](https://github.com/semantica-agi/semantica/blob/main/duplicate_detector.py) module performs fast approximate nearest-neighbor (ANN) searches via FAISS, Qdrant, Pinecone, or other backends defined in `semantica/vector_store/`. For each record, it retrieves a candidate set of potential duplicates by querying the vector index and filtering results to those exceeding a configurable similarity threshold. This sub-linear lookup process can be parallelized across data shards.

### 3. Clustering

Once candidates are identified, [`cluster_builder.py`](https://github.com/semantica-agi/semantica/blob/main/cluster_builder.py) and [`methods.py`](https://github.com/semantica-agi/semantica/blob/main/methods.py) group them into clusters using graph-based algorithms like **ConnectedComponents** or **Agglomerative** clustering. Operating on the sparse candidate graph rather than the full dataset, these methods utilize the `parallelism_manager` in [`semantica/pipeline/parallelism_manager.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/pipeline/parallelism_manager.py) to distribute computation across multiple CPU cores or distributed workers.

### 4. Merge-Strategy Selection

The [`merge_strategy.py`](https://github.com/semantica-agi/semantica/blob/main/merge_strategy.py) file defines pure-function strategies including **KeepFirst**, **PreferMostRecent**, and **CustomResolver**. These strategies determine which attributes survive the merge when collapsing a cluster into a single canonical entity. Because strategies are stateless pure functions, they can be applied independently to each cluster, enabling embarrassingly parallel execution.

### 5. Entity Merging

The [`entity_merger.py`](https://github.com/semantica-agi/semantica/blob/main/entity_merger.py) module executes the selected strategy to create a canonical entity while recording comprehensive provenance. It writes merge metadata—including original IDs, timestamps, and confidence scores—via [`deduplication_provenance.py`](https://github.com/semantica-agi/semantica/blob/main/deduplication_provenance.py), creating an auditable trail of how each canonical record was formed.

### 6. Registry and Persistence

Finally, [`registry.py`](https://github.com/semantica-agi/semantica/blob/main/registry.py) registers the canonical entities in the graph, updates the vector store embeddings, and persists the ID mapping to a deduplication index. The registry abstracts underlying storage systems (SQL, NoSQL, or in-memory), allowing the same code path to operate in batch jobs or streaming services.

## Scalability Mechanisms

### Vector-Store Backend Abstraction

The deduplication layer remains agnostic to specific vector databases. By interacting through the `semantica/vector_store/` interface, the system supports FAISS, Qdrant, Pinecone, Weaviate, and Milvus. Switching backends requires only configuration changes, enabling teams to select infrastructure that matches their latency and volume requirements.

### Parallelism Management

[`semantica/pipeline/parallelism_manager.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/pipeline/parallelism_manager.py) orchestrates multi-process execution across the detector, clustering, and merging stages. The manager automatically partitions the candidate graph and balances workloads across available compute resources, whether on a single multi-core machine or a distributed cluster.

### Lazy Provenance Recording

Instead of writing individual audit records for every merge operation, [`deduplication_provenance.py`](https://github.com/semantica-agi/semantica/blob/main/deduplication_provenance.py) batches provenance updates and writes them in bulk. This dramatically reduces I/O overhead when processing high-volume datasets.

### Chunked Processing via Resource Scheduling

The pipeline accepts data streams through the **resource scheduler** ([`semantica/pipeline/resource_scheduler.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/pipeline/resource_scheduler.py)). This design enables out-of-core processing of datasets that exceed available memory, resolving each chunk independently before merging results into the global identity map.

### Configurable Similarity Thresholds

The [`config.py`](https://github.com/semantica-agi/semantica/blob/main/config.py) file allows per-attribute tuning of similarity cutoffs. By aggressively pruning the candidate space for high-cardinality fields, the system reduces the size of the candidate graph and subsequent clustering workload.

## Implementation Example

The following example demonstrates a complete deduplication workflow using a pandas DataFrame:

```python
import pandas as pd
from semantica.deduplication.registry import DeduplicationRegistry
from semantica.deduplication.methods import ConnectedComponents
from semantica.vector_store.faiss_store import FaissVectorStore
from semantica.deduplication.merge_strategy import KeepFirst

# 1️⃣ Load raw data

raw_df = pd.read_csv("data/raw_entities.csv")

# 2️⃣ Initialise vector store (FAISS in this example)

vector_store = FaissVectorStore(dim=768)        # embeddings are 768‑dimensional

# 3️⃣ Register the dataset – this creates embeddings and stores them

registry = DeduplicationRegistry(vector_store=vector_store)
registry.index_documents(raw_df, text_column="description")

# 4️⃣ Detect duplicate candidates

detector = registry.duplicate_detector
candidate_graph = detector.build_candidate_graph(threshold=0.85)

# 5️⃣ Cluster candidates

clusterer = ConnectedComponents(similarity_threshold=0.85)
clusters = clusterer.cluster(candidate_graph)

# 6️⃣ Resolve each cluster

merger = registry.entity_merger
for cluster in clusters:
    canonical = merger.merge_cluster(
        cluster,
        strategy=KeepFirst(),          # simple “first‑seen wins” strategy

        provenance=True                # keep full provenance

    )
    print(f"✔️  Merged {len(cluster)} records → {canonical.id}")

# 7️⃣ Persist the resolved entities back to the vector store

registry.persist_canonical_entities()

```

**Step-by-step explanation:**

- **Step 2:** Creates a FAISS index where embeddings are inserted lazily during document indexing.
- **Step 4:** `build_candidate_graph` performs cosine similarity nearest-neighbor searches, retaining only pairs exceeding the `0.85` threshold.
- **Step 5:** `ConnectedComponents` treats the candidate pairs as an undirected graph and extracts connected components as duplicate clusters.
- **Step 6:** `merge_cluster` applies the **KeepFirst** strategy, preserving the first record's attributes while recording full provenance metadata.
- **Step 7:** Writes resolved canonical vectors back to the index, replacing noisy originals.

## Key Source Files

- **[`semantica/deduplication/entity_merger.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/entity_merger.py)** – Core logic for applying merge strategies and writing provenance.
- **[`semantica/deduplication/duplicate_detector.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/duplicate_detector.py)** – Fast nearest-neighbor search and candidate graph construction.
- **[`semantica/deduplication/cluster_builder.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/cluster_builder.py)** – Orchestrates clustering algorithms over the candidate graph.
- **[`semantica/deduplication/merge_strategy.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/merge_strategy.py)** – Definitions of built-in merge policies.
- **[`semantica/deduplication/methods.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/methods.py)** – Implementations of **ConnectedComponents** and **Agglomerative** clustering.
- **[`semantica/deduplication/registry.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/registry.py)** – High-level API wiring together vector-store, detector, clusterer, and merger.
- **[`semantica/deduplication/deduplication_provenance.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/deduplication_provenance.py)** – Provenance model recording source IDs, timestamps, and confidence scores.
- **[`semantica/pipeline/parallelism_manager.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/pipeline/parallelism_manager.py)** – Multi-process and distributed execution management.
- **`semantica/vector_store/`** – Backend adapters for FAISS, Qdrant, Pinecone, and other vector databases.

## Summary

- semantica.deduplication uses a six-stage pipeline: feature extraction, candidate generation, clustering, strategy selection, merging, and persistence.
- The system achieves scale through vector-store abstraction, parallel processing via [`parallelism_manager.py`](https://github.com/semantica-agi/semantica/blob/main/parallelism_manager.py), and lazy provenance batching.
- Clustering algorithms operate on sparse candidate graphs rather than full record sets, enabling billion-record scalability.
- Merge strategies are implemented as pure functions in [`merge_strategy.py`](https://github.com/semantica-agi/semantica/blob/main/merge_strategy.py), allowing embarrassingly parallel execution across clusters.
- Full audit trails are maintained through [`deduplication_provenance.py`](https://github.com/semantica-agi/semantica/blob/main/deduplication_provenance.py) with batched I/O for performance.

## Frequently Asked Questions

### How does semantica.deduplication handle billions of records?

The system combines chunked processing via the resource scheduler with vector-store backends that provide sub-linear approximate nearest-neighbor search. The candidate graph remains sparse relative to the total dataset size, and the [`parallelism_manager.py`](https://github.com/semantica-agi/semantica/blob/main/parallelism_manager.py) distributes both clustering and merging across distributed workers.

### Which vector databases are supported?

The `semantica/vector_store/` package provides adapters for FAISS, Qdrant, Pinecone, Weaviate, and Milvus. The deduplication logic is backend-agnostic, allowing configuration-driven switching without code changes.

### How is merge provenance tracked?

The [`deduplication_provenance.py`](https://github.com/semantica-agi/semantica/blob/main/deduplication_provenance.py) module records original entity IDs, merge timestamps, similarity scores, and strategy used for every canonical entity. These records are batched and written in bulk to minimize I/O overhead while maintaining full auditability.

### Can I implement custom merge strategies?

Yes. The [`merge_strategy.py`](https://github.com/semantica-agi/semantica/blob/main/merge_strategy.py) interface accepts custom implementations as long as they conform to the pure-function contract. A custom resolver receives the cluster of duplicate records and returns a single canonical entity, with access to all attributes and metadata for sophisticated conflict resolution.