How to Perform Entity Resolution and Semantic Deduplication at Scale with Semantica

Semantica resolves millions of entity mentions into a canonical knowledge graph by embedding each mention in a vector store, clustering nearest neighbors via EntityResolver, and merging duplicates through configurable conflict strategies.

Entity resolution and semantic deduplication at scale are core capabilities of the semantica-agi/semantica open-source framework. The pipeline transforms raw text into structured knowledge by linking synonymous mentions and collapsing redundant nodes without losing relational context. Every stage is designed to operate across distributed vector back-ends and parallel workers.

Architecture for Scalable Entity Resolution and Deduplication

The framework separates extraction, embedding, resolution, and deduplication into discrete layers that communicate through a unified vector-store API.

• SemanticExtractor (semantica/semantic_extract/extractor.py) detects typed entity mentions and attributes from unstructured text.
• Vector Store Layer (semantica/vector_store/methods.py and semantica/vector_store/*_store.py) persists dense embeddings and executes fast nearest-neighbor search via FAISS, Qdrant, Pinecone, or Milvus.
• EntityResolver (semantica/kg/entity_resolver.py) clusters embeddings that exceed a similarity threshold and returns canonical groups.
• ConflictDetector (semantica/normalize/conflict_detector.py) and DeduplicationDetector (semantica/normalize/deduplication.py) merge duplicate nodes and resolve attribute collisions.
• ParallelismManager (semantica/pipeline/parallelism_manager.py) distributes chunks of work across processes or remote executors.

The Entity Resolution Pipeline

Extract and Embed Mentions

After SemanticExtractor produces a list of typed mentions, each entity is converted into a dense vector through the vector store's add_embeddings() method. The store returns a high-dimensional representation that captures semantic meaning.

Similarity Search with Hybrid Scoring

EntityResolver queries the vector store for the k nearest neighbors of every new embedding. In semantica/vector_store/hybrid_similarity.py, the HybridSimilarity helper combines cosine similarity with graph-based path scores when a knowledge graph already exists. This ensures structural context augments pure vector closeness.

Clustering and Canonical Assignment

When a neighbor's similarity exceeds the configurable similarity_cutoff parameter inside EntityResolver, the resolver groups the mentions into a single cluster. The cluster is then passed to GraphBuilder (semantica/kg/graph_builder.py), which selects a canonical node and prepares edge rewiring.

Parallel Execution

The ParallelismManager splits incoming entity batches into independent chunks. Each chunk is processed by a separate worker running EntityResolver.resolve_entities(), eliminating single-process bottlenecks and enabling horizontal scaling.

Semantic Deduplication at Scale

Deduplication is triggered automatically when GraphBuilder is initialized with resolve_conflicts=True.

Detect Duplicate Candidates

DeduplicationDetector scans the graph for nodes that share the same type and exhibit high embedding similarity. It reuses the hybrid similarity metric from the resolver, so both lexical and structural cues inform the detection phase.

Resolve Attribute Conflicts

For each duplicate group, ConflictDetector applies one of three strategies defined in semantica/normalize/conflict_detector.py:

• keep-first — Preserve the earliest created node and discard subsequent duplicates.
• merge — Union attribute dictionaries, preferring non-null scalars and concatenating list values.
• LLM-guided — Invoke a language model to synthesize a consolidated representation when attribute semantics are ambiguous.

Rewire Graph Edges

Once a canonical node is chosen, all inbound and outbound edges of the removed duplicates are re-attached to the survivor. This step preserves graph connectivity and is coordinated through GraphBuilder before the final flush to persistent storage.

Configuring the Pipeline for Production Workloads

The PipelineBuilder in semantica/pipeline/pipeline_builder.py exposes constructor flags that tune throughput and matching strictness.

from semantica.pipeline import PipelineBuilder

pipeline = PipelineBuilder(
    merge_entities=True,
    resolve_conflicts=True,
    similarity_cutoff=0.85,
    deduplication_strategy="merge",
    parallelism=8,
)

graph = pipeline.build_from_dir("/data/documents")

Setting parallelism=8 creates a worker pool that processes entity batches concurrently, while similarity_cutoff=0.85 raises the bar for declaring two mentions identical.

Practical Code Examples

Resolve Entities with FAISS

from semantica.kg import EntityResolver
from semantica.vector_store import FaissStore

store = FaissStore(dimension=768)

entities = [
    {"name": "Apple Inc.", "type": "Organization"},
    {"name": "Apple", "type": "Organization"},
    {"name": "AAPL", "type": "Ticker"},
]

resolver = EntityResolver(
    vector_store=store,
    similarity_cutoff=0.90,
)

resolved = resolver.resolve_entities(entities)
print(resolved)

# [{'canonical_id': 0, 'cluster': [0, 1]}, {'canonical_id': 2, 'cluster': [2]}]

This example uses the in-memory FaissStore for local development. EntityResolver.resolve_entities() returns cluster mappings that indicate which input indices refer to the same real-world object.

Build a Deduplicated Knowledge Graph

from semantica.pipeline import PipelineBuilder

builder = PipelineBuilder(
    resolve_conflicts=True,
    deduplication_strategy="merge",
    parallelism=4,
)

graph = builder.build_from_files([
    "reports/q1_2024.txt",
    "reports/q2_2024.txt",
])

print(f"Nodes: {len(graph.nodes)}, Edges: {len(graph.edges)}")

Here, PipelineBuilder orchestrates extraction, embedding, resolution, and deduplication. Enabling resolve_conflicts invokes ConflictDetector automatically after the initial graph is constructed.

Scale with a Distributed Vector Store

from semantica.vector_store import PineconeStore
from semantica.kg import EntityResolver

pinecone = PineconeStore(
    index_name="enterprise-kg",
    api_key="YOUR_PINECONE_KEY",
)

resolver = EntityResolver(
    vector_store=pinecone,
    similarity_cutoff=0.87,
    use_hybrid=True,
)

resolved = resolver.resolve_entities(large_entity_list)

PineconeStore provides O(log n) approximate nearest-neighbor search across billions of vectors. Setting use_hybrid=True activates the hybrid similarity path so that graph structure and embedding proximity are weighed jointly.

Summary

• Entity resolution in Semantica relies on vector embeddings and nearest-neighbor search inside EntityResolver (semantica/kg/entity_resolver.py).
• Semantic deduplication is handled by DeduplicationDetector and ConflictDetector, which merge attributes and rewire edges without breaking graph connectivity.
• The pipeline scales horizontally through ParallelismManager and supports multiple vector back-ends, including FAISS, Qdrant, Pinecone, and Milvus.
• Production workloads are tuned via PipelineBuilder parameters such as similarity_cutoff, deduplication_strategy, and parallelism.

Frequently Asked Questions

What is the default similarity threshold for entity resolution in Semantica?

The EntityResolver class exposes a similarity_cutoff parameter that controls how close two embeddings must be to form a cluster. While the exact default value depends on the instantiated configuration, typical deployments initialize it around 0.85 and override it through PipelineBuilder for stricter or looser matching.

How does Semantica handle conflicting attributes when merging duplicate nodes?

ConflictDetector (semantica/normalize/conflict_detector.py) supports three strategies: keep-first, merge, and LLM-guided. The merge strategy unions dictionaries and concatenates lists, while LLM-guided delegates ambiguous cases to a language model for a synthesized canonical value.

Can the entity resolution pipeline run on multiple CPU cores or remote workers?

Yes. ParallelismManager (semantica/pipeline/parallelism_manager.py) partitions entity batches and distributes them across a process pool or remote executors. Both EntityResolver.resolve_entities() and the downstream deduplication stages are designed to be embarrassingly parallel.

Which vector store back-ends support the hybrid similarity search used by EntityResolver?

The hybrid similarity path in semantica/vector_store/hybrid_similarity.py is back-end agnostic at the API level. It is validated against FaissStore, PineconeStore, QdrantStore, and MilvusStore (semantica/vector_store/*_store.py), provided the store implements the standard search and add_embeddings interface.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}