# How to Perform Entity Resolution and Semantic Deduplication at Scale with Semantica

> Learn to perform entity resolution and semantic deduplication at scale with Semantica. Effortlessly resolve millions of entity mentions into a knowledge graph by embedding, clustering, and merging duplicates.

- Repository: [Semantica /semantica](https://github.com/semantica-agi/semantica)
- Tags: how-to-guide
- Published: 2026-08-20

---

**Semantica resolves millions of entity mentions into a canonical knowledge graph by embedding each mention in a vector store, clustering nearest neighbors via `EntityResolver`, and merging duplicates through configurable conflict strategies.**

Entity resolution and semantic deduplication at scale are core capabilities of the `semantica-agi/semantica` open-source framework. The pipeline transforms raw text into structured knowledge by linking synonymous mentions and collapsing redundant nodes without losing relational context. Every stage is designed to operate across distributed vector back-ends and parallel workers.

## Architecture for Scalable Entity Resolution and Deduplication

The framework separates extraction, embedding, resolution, and deduplication into discrete layers that communicate through a unified vector-store API.

- **`SemanticExtractor`** ([`semantica/semantic_extract/extractor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/semantic_extract/extractor.py)) detects typed entity mentions and attributes from unstructured text.
- **Vector Store Layer** ([`semantica/vector_store/methods.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/vector_store/methods.py) and `semantica/vector_store/*_store.py`) persists dense embeddings and executes fast nearest-neighbor search via FAISS, Qdrant, Pinecone, or Milvus.
- **`EntityResolver`** ([`semantica/kg/entity_resolver.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/kg/entity_resolver.py)) clusters embeddings that exceed a similarity threshold and returns canonical groups.
- **`ConflictDetector`** ([`semantica/normalize/conflict_detector.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/normalize/conflict_detector.py)) and **`DeduplicationDetector`** ([`semantica/normalize/deduplication.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/normalize/deduplication.py)) merge duplicate nodes and resolve attribute collisions.
- **`ParallelismManager`** ([`semantica/pipeline/parallelism_manager.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/pipeline/parallelism_manager.py)) distributes chunks of work across processes or remote executors.

## The Entity Resolution Pipeline

### Extract and Embed Mentions

After `SemanticExtractor` produces a list of typed mentions, each entity is converted into a dense vector through the vector store's `add_embeddings()` method. The store returns a high-dimensional representation that captures semantic meaning.

### Similarity Search with Hybrid Scoring

`EntityResolver` queries the vector store for the *k* nearest neighbors of every new embedding. In [`semantica/vector_store/hybrid_similarity.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/vector_store/hybrid_similarity.py), the `HybridSimilarity` helper combines cosine similarity with graph-based path scores when a knowledge graph already exists. This ensures structural context augments pure vector closeness.

### Clustering and Canonical Assignment

When a neighbor's similarity exceeds the configurable `similarity_cutoff` parameter inside `EntityResolver`, the resolver groups the mentions into a single cluster. The cluster is then passed to `GraphBuilder` ([`semantica/kg/graph_builder.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/kg/graph_builder.py)), which selects a canonical node and prepares edge rewiring.

### Parallel Execution

The `ParallelismManager` splits incoming entity batches into independent chunks. Each chunk is processed by a separate worker running `EntityResolver.resolve_entities()`, eliminating single-process bottlenecks and enabling horizontal scaling.

## Semantic Deduplication at Scale

Deduplication is triggered automatically when `GraphBuilder` is initialized with `resolve_conflicts=True`.

### Detect Duplicate Candidates

`DeduplicationDetector` scans the graph for nodes that share the same type and exhibit high embedding similarity. It reuses the hybrid similarity metric from the resolver, so both lexical and structural cues inform the detection phase.

### Resolve Attribute Conflicts

For each duplicate group, `ConflictDetector` applies one of three strategies defined in [`semantica/normalize/conflict_detector.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/normalize/conflict_detector.py):

- **keep-first** — Preserve the earliest created node and discard subsequent duplicates.
- **merge** — Union attribute dictionaries, preferring non-null scalars and concatenating list values.
- **LLM-guided** — Invoke a language model to synthesize a consolidated representation when attribute semantics are ambiguous.

### Rewire Graph Edges

Once a canonical node is chosen, all inbound and outbound edges of the removed duplicates are re-attached to the survivor. This step preserves graph connectivity and is coordinated through `GraphBuilder` before the final flush to persistent storage.

## Configuring the Pipeline for Production Workloads

The `PipelineBuilder` in [`semantica/pipeline/pipeline_builder.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/pipeline/pipeline_builder.py) exposes constructor flags that tune throughput and matching strictness.

```python
from semantica.pipeline import PipelineBuilder

pipeline = PipelineBuilder(
    merge_entities=True,
    resolve_conflicts=True,
    similarity_cutoff=0.85,
    deduplication_strategy="merge",
    parallelism=8,
)

graph = pipeline.build_from_dir("/data/documents")

```

Setting `parallelism=8` creates a worker pool that processes entity batches concurrently, while `similarity_cutoff=0.85` raises the bar for declaring two mentions identical.

## Practical Code Examples

### Resolve Entities with FAISS

```python
from semantica.kg import EntityResolver
from semantica.vector_store import FaissStore

store = FaissStore(dimension=768)

entities = [
    {"name": "Apple Inc.", "type": "Organization"},
    {"name": "Apple", "type": "Organization"},
    {"name": "AAPL", "type": "Ticker"},
]

resolver = EntityResolver(
    vector_store=store,
    similarity_cutoff=0.90,
)

resolved = resolver.resolve_entities(entities)
print(resolved)

# [{'canonical_id': 0, 'cluster': [0, 1]}, {'canonical_id': 2, 'cluster': [2]}]

```

This example uses the in-memory `FaissStore` for local development. `EntityResolver.resolve_entities()` returns cluster mappings that indicate which input indices refer to the same real-world object.

### Build a Deduplicated Knowledge Graph

```python
from semantica.pipeline import PipelineBuilder

builder = PipelineBuilder(
    resolve_conflicts=True,
    deduplication_strategy="merge",
    parallelism=4,
)

graph = builder.build_from_files([
    "reports/q1_2024.txt",
    "reports/q2_2024.txt",
])

print(f"Nodes: {len(graph.nodes)}, Edges: {len(graph.edges)}")

```

Here, `PipelineBuilder` orchestrates extraction, embedding, resolution, and deduplication. Enabling `resolve_conflicts` invokes `ConflictDetector` automatically after the initial graph is constructed.

### Scale with a Distributed Vector Store

```python
from semantica.vector_store import PineconeStore
from semantica.kg import EntityResolver

pinecone = PineconeStore(
    index_name="enterprise-kg",
    api_key="YOUR_PINECONE_KEY",
)

resolver = EntityResolver(
    vector_store=pinecone,
    similarity_cutoff=0.87,
    use_hybrid=True,
)

resolved = resolver.resolve_entities(large_entity_list)

```

`PineconeStore` provides `O(log n)` approximate nearest-neighbor search across billions of vectors. Setting `use_hybrid=True` activates the hybrid similarity path so that graph structure and embedding proximity are weighed jointly.

## Summary

- **Entity resolution** in Semantica relies on vector embeddings and nearest-neighbor search inside `EntityResolver` ([`semantica/kg/entity_resolver.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/kg/entity_resolver.py)).
- **Semantic deduplication** is handled by `DeduplicationDetector` and `ConflictDetector`, which merge attributes and rewire edges without breaking graph connectivity.
- The pipeline scales horizontally through `ParallelismManager` and supports multiple vector back-ends, including FAISS, Qdrant, Pinecone, and Milvus.
- Production workloads are tuned via `PipelineBuilder` parameters such as `similarity_cutoff`, `deduplication_strategy`, and `parallelism`.

## Frequently Asked Questions

### What is the default similarity threshold for entity resolution in Semantica?

The `EntityResolver` class exposes a `similarity_cutoff` parameter that controls how close two embeddings must be to form a cluster. While the exact default value depends on the instantiated configuration, typical deployments initialize it around `0.85` and override it through `PipelineBuilder` for stricter or looser matching.

### How does Semantica handle conflicting attributes when merging duplicate nodes?

`ConflictDetector` ([`semantica/normalize/conflict_detector.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/normalize/conflict_detector.py)) supports three strategies: `keep-first`, `merge`, and `LLM-guided`. The `merge` strategy unions dictionaries and concatenates lists, while `LLM-guided` delegates ambiguous cases to a language model for a synthesized canonical value.

### Can the entity resolution pipeline run on multiple CPU cores or remote workers?

Yes. `ParallelismManager` ([`semantica/pipeline/parallelism_manager.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/pipeline/parallelism_manager.py)) partitions entity batches and distributes them across a process pool or remote executors. Both `EntityResolver.resolve_entities()` and the downstream deduplication stages are designed to be embarrassingly parallel.

### Which vector store back-ends support the hybrid similarity search used by EntityResolver?

The hybrid similarity path in [`semantica/vector_store/hybrid_similarity.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/vector_store/hybrid_similarity.py) is back-end agnostic at the API level. It is validated against `FaissStore`, `PineconeStore`, `QdrantStore`, and `MilvusStore` (`semantica/vector_store/*_store.py`), provided the store implements the standard `search` and `add_embeddings` interface.