# How Semantica Performs Conflict Detection and Deduplication on Ingested Data

> Learn how Semantica performs conflict detection and deduplication on ingested data using its three-stage vector pipeline and reasoning layer. Prevent contradictions and ensure data integrity.

- Repository: [Semantica /semantica](https://github.com/semantica-agi/semantica)
- Tags: how-to-guide
- Published: 2026-09-04

---

**Semantica detects conflicts and deduplicates ingested data through a three-stage vector-based pipeline—embedding entities, clustering by cosine similarity, and merging duplicates—while the reasoning layer hashes and deduplicates rules to prevent contradictory inference paths.**

The `semantica-agi/semantica` repository implements a robust **conflict detection and deduplication** system designed for knowledge graph ingestion. When raw RDF-style triplets enter the system, Semantica coordinates between its deduplication package and reasoning engine to eliminate redundant entities and prevent conflicting inference rules. This article examines the exact implementation files and methods that power this process.

## The Three-Stage Deduplication Pipeline

Semantica’s deduplication logic resides in the `semantica/deduplication/` package and executes in three distinct phases before data reaches the graph store.

### Vector Embedding Generation

Every entity—whether a subject, object, or predicate—is transformed into a dense vector representation using the configured embedding provider. This occurs during the initialization of the `DuplicateDetector` class in [`semantica/deduplication/duplicate_detector.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/duplicate_detector.py).

The `__init__` method loads the embedding provider from [`semantica/deduplication/embedding_provider.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/embedding_provider.py), which interfaces with vector stores such as Weaviate or Qdrant. Converting raw textual entities into embeddings enables mathematical similarity comparisons across the entire dataset.

### Similarity Calculation and Clustering

Once embedded, Semantica calculates pairwise **cosine similarity** between all entities. The core logic lives in [`semantica/deduplication/methods.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/methods.py), specifically within the `calculate_similarity` and `detect_duplicate_groups` functions.

Entities exceeding a configurable similarity threshold (defaulting to 0.85 in typical configurations) are grouped into duplicate clusters. The system supports multiple clustering strategies controlled by the `strategy` parameter:

- **Single-link clustering** (default): Groups entities if any pair meets the threshold, creating broader clusters.
- **Complete-link clustering**: Only groups entities where all pairs exceed the threshold, producing tighter, more conservative clusters.

The `build_clusters` function in [`semantica/deduplication/methods.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/methods.py) implements these algorithms, returning duplicate groups ready for merging.

### Entity Merging and Canonicalization

The final stage resolves duplicate clusters into single canonical entities. The `EntityMerger` class in [`semantica/deduplication/entity_merger.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/entity_merger.py) handles this through its `merge_duplicates` method.

For each cluster, the merger:

1. Constructs a canonical representation by aggregating metadata from all duplicate entities.
2. Updates the underlying graph store to point all references to the canonical entity.
3. Records provenance data tracking which triples were collapsed, enabling downstream audit trails.

This process ensures that ingested knowledge graphs remain compact and semantically consistent without losing historical context.

## Rule-Level Conflict Detection

Beyond entity deduplication, Semantica prevents logical conflicts during inference. The reasoning engine, implemented in [`semantica/reasoning/reasoner.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/reasoning/reasoner.py), deduplicates identical rules before adding them to the Rete network.

The `Reasoner.add_rule` method hashes the rule’s conditions and conclusions to generate a unique fingerprint. When a new rule arrives, the engine checks this hash against existing rules. If a match exists, the new rule is silently ignored, preventing redundant or contradictory inference paths. This idempotent behavior is verified in [`tests/reasoning/test_reasoner.py`](https://github.com/semantica-agi/semantica/blob/main/tests/reasoning/test_reasoner.py), which confirms that adding the same rule twice results in a single stored rule while distinct rules with different conditions are preserved separately.

## Using the Conflict Detection API

### Python Implementation

Integrate deduplication directly into your ingestion pipeline using the `DuplicateDetector` class:

```python
from semantica.deduplication import DuplicateDetector

# Initialize with vector store and similarity threshold

detector = DuplicateDetector(
    vector_store=my_vector_store,
    similarity_threshold=0.85,
    strategy="single_link"  # or "complete_link"

)

# Detect duplicate groups from ingested triplets

duplicate_groups = detector.detect_duplicate_groups(triplets)

# Execute merge and update graph store

detector.merge_duplicates(duplicate_groups)

```

To prevent rule conflicts in the reasoning layer:

```python
from semantica.reasoning import Reasoner

reasoner = Reasoner()

# Duplicate rules are automatically ignored based on content hash

reasoner.add_rule(
    conditions=[("person", "hasAge", "?age"), ("?age", ">", "18")],
    conclusion=("person", "isAdult", "true")
)

# Second call with identical parameters is deduplicated automatically

reasoner.add_rule(
    conditions=[("person", "hasAge", "?age"), ("?age", ">", "18")],
    conclusion=("person", "isAdult", "true")
)

```

### Command-Line Interface

Semantica exposes deduplication operations through its CLI for operational workflows:

```bash

# Preview potential merges without modifying data

semantica deduplicate --dry-run --strategy single_link

# Execute merge and generate JSON report

semantica deduplicate --action merge --output ./deduplication-report.json

# Adjust similarity threshold for stricter matching

semantica deduplicate --similarity-threshold 0.92 --action merge

```

## Summary

- **Three-stage pipeline**: Semantica deduplicates via embedding ([`embedding_provider.py`](https://github.com/semantica-agi/semantica/blob/main/embedding_provider.py)), cosine similarity clustering ([`methods.py`](https://github.com/semantica-agi/semantica/blob/main/methods.py)), and canonical merging ([`entity_merger.py`](https://github.com/semantica-agi/semantica/blob/main/entity_merger.py)).
- **Configurable clustering**: Choose between single-link and complete-link strategies in `detect_duplicate_groups` based on precision requirements.
- **Hash-based rule deduplication**: The `Reasoner.add_rule` method in [`semantica/reasoning/reasoner.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/reasoning/reasoner.py) prevents conflicting inference by hashing rule conditions and conclusions.
- **Provenance tracking**: The `EntityMerger` records merge history, ensuring traceability from raw ingestion to canonical graph state.
- **Dual interfaces**: Both Python API and CLI support dry-run capabilities and threshold tuning for production data pipelines.

## Frequently Asked Questions

### How does Semantica calculate similarity between entities?

Semantica converts entities to dense vectors using the embedding provider, then computes **cosine similarity** between vector pairs. The `calculate_similarity` function in [`semantica/deduplication/methods.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/methods.py) performs this calculation, comparing embeddings against a configurable threshold (typically 0.85) to determine potential duplicates.

### What clustering strategies does Semantica support for deduplication?

The system supports **single-link** and **complete-link** clustering algorithms, controlled via the `strategy` parameter in [`semantica/deduplication/methods.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/methods.py). Single-link clustering (default) groups entities if any pairwise similarity exceeds the threshold, while complete-link requires all pairs within a cluster to meet the threshold, yielding more conservative groupings.

### How does the reasoner prevent duplicate inference rules?

Before adding a rule to the Rete network, `Reasoner.add_rule` in [`semantica/reasoning/reasoner.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/reasoning/reasoner.py) generates a hash from the rule’s conditions and conclusions. If this hash matches an existing rule, the new addition is discarded. This ensures idempotent rule insertion and prevents contradictory or redundant inference paths from entering the knowledge base.

### Can the similarity threshold be adjusted for different data types?

Yes. The `DuplicateDetector` class accepts a `similarity_threshold` parameter during initialization, allowing pipelines to tune sensitivity per dataset. Higher thresholds (e.g., 0.95) reduce false positives for messy data, while lower thresholds (e.g., 0.80) catch more potential duplicates in clean, standardized vocabularies.