# How Semantica Handles Conflict Detection and Deduplication: A Deep Dive into the Open-Source Knowledge Graph Pipeline

> Explore how Semantica's open-source knowledge graph pipeline detects conflicts and deduplicates entities using dedicated ConflictDetector and DuplicateDetector classes.

- Repository: [Semantica /semantica](https://github.com/semantica-agi/semantica)
- Tags: deep-dive
- Published: 2026-09-05

---

**Semantica detects conflicts and performs deduplication through two complementary subsystems: the `ConflictDetector` class for identifying contradictory values across sources, and the `DuplicateDetector` class for identifying and merging duplicate entities using configurable similarity strategies.**

The semantica-agi/semantica repository provides a comprehensive framework for maintaining data quality in knowledge graphs. Its **conflict detection** and **deduplication** mechanisms work together to ensure that information integrated from heterogeneous sources remains consistent, auditable, and free of redundant records. Understanding how Semantica handles conflict detection and deduplication is essential for building reliable AI systems that consume structured knowledge.

## Conflict Detection Architecture

The conflict detection subsystem resides in [`semantica/conflicts/conflict_detector.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/conflicts/conflict_detector.py) and centers on the `ConflictDetector` class. This component identifies when the same real-world entity carries contradictory values across different source documents.

### Entity Grouping and Identification

All incoming entities are first grouped by a stable identifier—either `id` or `entity_id`. This grouping associates multiple representations of the same real-world object, even when they originate from separate documents or data streams.

### Value Comparison Workflow

For each target property (such as `name` or `type`), the detector collects distinct values supplied by each source. When more than one unique value exists for the same property on the same entity, the system creates a `Conflict` dataclass instance.

The detection workflow leverages the generic progress-tracking utilities in `utils.progress_tracker` to report status. This scales from per-item updates on small datasets to periodic updates on large-scale ingestion jobs.

### Source Provenance and Severity Scoring

When `track_provenance=True` (the default), every value links to a `SourceReference` object via the `SourceTracker`. This captures document origin, page numbers, extraction confidence, and other metadata for compliance and auditing.

Two private helper methods assign risk metrics:
- `_calculate_severity()` → assigns low, medium, high, or critical levels based on field importance and source divergence
- `_calculate_conflict_confidence()` → produces a numeric score considering source count and value variance

Detected conflicts trigger `self.logger.warning` calls and accumulate in `self.detected_conflicts`. The class also exposes `get_conflict_report()` for downstream consumption.

### Extended Conflict Dimensions

Beyond simple value mismatches, the detector examines:
- **Relationship conflicts** via `detect_relationship_conflicts()`
- Type mismatches between sources
- Temporal inconsistencies in date-stamped data
- Logical rule violations against domain constraints

## Deduplication Pipeline

The deduplication subsystem in `semantica/deduplication/*` handles identifying and resolving duplicate entity records. The `DuplicateDetector` class orchestrates a three-stage pipeline.

### Stage 1: Similarity Calculation

The `calculate_similarity` function in [`semantica/deduplication/methods.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/methods.py) applies configurable metrics to candidate entity pairs. Available metrics include:
- **Lexical similarity** — string-based comparisons
- **Semantic similarity** — embedding-based comparisons
- **Fuzzy matching** — tolerance for typographical variation

### Stage 2: Duplicate Detection

The `detect_duplicates` function in the same file generates duplicate pairs or clusters based on thresholding. The function supports "magic" strategies such as `semantic_v2`, which incorporates synonym maps and domain-specific heuristics for improved recall.

Configuration example:

```python
from semantica.deduplication import DuplicateDetector

dup_detector = DuplicateDetector(
    strategy="semantic_v2",
    min_similarity=0.75
)
duplicate_pairs = dup_detector.detect_duplicates(entities)

```

### Stage 3: Entity Merging

When consolidation is required, the `EntityMerger` class in [`semantica/deduplication/entity_merger.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/entity_merger.py) performs the merge. It preserves:
- Source provenance from all original entities
- Metadata tags and extraction confidence scores
- The most confident values for each property

The merger also updates the underlying graph store to maintain referential integrity across relationships.

### CLI Integration and Dry-Run Mode

The deduplication subsystem exposes a CLI command (`semantica deduplicate`) defined in [`semantica/cli.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/cli.py). This supports:

```bash

# Preview duplicates without modifying data

semantica deduplicate --dry-run --output json

# Execute merge with specific strategy

semantica deduplicate --action merge --strategy semantic_v2

```

## Interaction Between Conflict Detection and Deduplication

Both subsystems share foundational infrastructure that enables seamless coordination:

- **Common entity identifier model** — conflicts detected pre-deduplication remain valid after merging
- **Shared `SourceTracker`** — provenance entries merge automatically when entities consolidate
- **Conflict retention** — the merger preserves previously detected conflicts and reattributes them to the consolidated entity

This design allows flexible pipeline arrangements. Users can run conflict detection before deduplication to surface high-severity contradictions for manual resolution, or run deduplication first to reduce noise in subsequent conflict analysis.

## Practical Implementation Examples

### Conflict Detection

```python
from semantica.conflicts import ConflictDetector

detector = ConflictDetector(
    track_provenance=True,
    auto_resolve_low=False
)

conflicts = detector.detect_value_conflicts(
    entities,
    property_name="name"
)

for c in conflicts:
    print(f"{c.entity_id}: {c.conflicting_values}")
    print(f"  Severity: {c.severity}, Confidence: {c.confidence}")

```

### Deduplication with Merge

```python
from semantica.deduplication import DuplicateDetector

dup_detector = DuplicateDetector(
    strategy="semantic_v2",
    min_similarity=0.75
)

# Detection

duplicate_pairs = dup_detector.detect_duplicates(entities)

# Dry-run to preview

preview = dup_detector.merge_duplicates(
    duplicate_pairs,
    dry_run=True
)
print(f"Would merge {len(preview)} entities")

# Execute merge

merged_entities = dup_detector.merge_duplicates(
    duplicate_pairs,
    dry_run=False
)

```

## Key Source Files and Responsibilities

| Component | Source Path | Primary Role |
|-----------|-------------|--------------|
| Conflict detection core | [`semantica/conflicts/conflict_detector.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/conflicts/conflict_detector.py) | Detects value, type, relationship, temporal, and logical conflicts; manages severity scoring and provenance |
| Similarity methods | [`semantica/deduplication/methods.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/methods.py) | Implements `calculate_similarity` and `detect_duplicates` with strategy handling |
| Entity merging | [`semantica/deduplication/entity_merger.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/entity_merger.py) | Consolidates duplicates via `EntityMerger` with provenance preservation |
| Public deduplication API | [`semantica/deduplication/__init__.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/__init__.py) | Exports `DuplicateDetector` for programmatic and CLI use |
| CLI command routing | [`semantica/cli.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/cli.py) | Wires deduplication to command-line with dry-run, output format, and strategy options |

## Summary

- Semantica's **conflict detection** uses the `ConflictDetector` class in [`semantica/conflicts/conflict_detector.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/conflicts/conflict_detector.py) to identify contradictory values, types, and relationships across sources while tracking provenance and severity
- **Deduplication** employs the `DuplicateDetector` class exposed from [`semantica/deduplication/__init__.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/__init__.py), coordinating similarity calculation, duplicate pair detection, and entity merging
- The **three-stage deduplication pipeline**—similarity scoring, duplicate detection, and entity merging—supports configurable strategies including the `semantic_v2` heuristic with synonym awareness
- Both subsystems share the **entity identifier model and `SourceTracker`** provenance infrastructure, enabling coherent operation before or after one another
- **Dry-run modes and structured reporting** support safe, auditable data quality workflows at scale

## Frequently Asked Questions

### What triggers a conflict in Semantica's conflict detection system?

A conflict triggers when the `ConflictDetector` observes multiple distinct values for the same property on entities sharing a common identifier (`id` or `entity_id`). The detector groups entities by this stable identifier, collects values per source, and creates a `Conflict` instance whenever uniqueness violations occur. Severity and confidence scores then characterize the divergence for prioritization.

### How does Semantica choose which values to keep when merging duplicates?

The `EntityMerger` class in [`semantica/deduplication/entity_merger.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/entity_merger.py) applies a confidence-preservation strategy: it retains the most confident value for each property while preserving all source provenance in the merged record. Metadata and extraction confidence from all constituent entities survive the merge, ensuring no source attribution is lost during consolidation.

### Can I run conflict detection and deduplication independently?

Yes. The subsystems are decoupled by design. You can execute `ConflictDetector.detect_value_conflicts()` without any deduplication step, or run `DuplicateDetector.detect_duplicates()` and `merge_duplicates()` without conflict analysis. Many deployments run conflict detection first to surface critical contradictions for manual review, then deduplicate to clean remaining redundancy.

### What similarity strategies does Semantica's deduplication support?

The `DuplicateDetector` accepts a `strategy` parameter that routes to implementations in [`semantica/deduplication/methods.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/deduplication/methods.py). The base implementation supports lexical, semantic, and fuzzy metrics. The `semantic_v2` strategy extends this with synonym map resolution and domain-specific heuristics for improved matching on semantically equivalent but superficially different entities.