How Semantica Handles Conflict Detection and Deduplication: A Deep Dive into the Open-Source Knowledge Graph Pipeline

Semantica detects conflicts and performs deduplication through two complementary subsystems: the ConflictDetector class for identifying contradictory values across sources, and the DuplicateDetector class for identifying and merging duplicate entities using configurable similarity strategies.

The semantica-agi/semantica repository provides a comprehensive framework for maintaining data quality in knowledge graphs. Its conflict detection and deduplication mechanisms work together to ensure that information integrated from heterogeneous sources remains consistent, auditable, and free of redundant records. Understanding how Semantica handles conflict detection and deduplication is essential for building reliable AI systems that consume structured knowledge.

Conflict Detection Architecture

The conflict detection subsystem resides in semantica/conflicts/conflict_detector.py and centers on the ConflictDetector class. This component identifies when the same real-world entity carries contradictory values across different source documents.

Entity Grouping and Identification

All incoming entities are first grouped by a stable identifier—either id or entity_id. This grouping associates multiple representations of the same real-world object, even when they originate from separate documents or data streams.

Value Comparison Workflow

For each target property (such as name or type), the detector collects distinct values supplied by each source. When more than one unique value exists for the same property on the same entity, the system creates a Conflict dataclass instance.

The detection workflow leverages the generic progress-tracking utilities in utils.progress_tracker to report status. This scales from per-item updates on small datasets to periodic updates on large-scale ingestion jobs.

Source Provenance and Severity Scoring

When track_provenance=True (the default), every value links to a SourceReference object via the SourceTracker. This captures document origin, page numbers, extraction confidence, and other metadata for compliance and auditing.

Two private helper methods assign risk metrics:

• _calculate_severity() → assigns low, medium, high, or critical levels based on field importance and source divergence
• _calculate_conflict_confidence() → produces a numeric score considering source count and value variance

Detected conflicts trigger self.logger.warning calls and accumulate in self.detected_conflicts. The class also exposes get_conflict_report() for downstream consumption.

Extended Conflict Dimensions

Beyond simple value mismatches, the detector examines:

• Relationship conflicts via detect_relationship_conflicts()
• Type mismatches between sources
• Temporal inconsistencies in date-stamped data
• Logical rule violations against domain constraints

Deduplication Pipeline

The deduplication subsystem in semantica/deduplication/* handles identifying and resolving duplicate entity records. The DuplicateDetector class orchestrates a three-stage pipeline.

Stage 1: Similarity Calculation

The calculate_similarity function in semantica/deduplication/methods.py applies configurable metrics to candidate entity pairs. Available metrics include:

• Lexical similarity — string-based comparisons
• Semantic similarity — embedding-based comparisons
• Fuzzy matching — tolerance for typographical variation

Stage 2: Duplicate Detection

The detect_duplicates function in the same file generates duplicate pairs or clusters based on thresholding. The function supports "magic" strategies such as semantic_v2, which incorporates synonym maps and domain-specific heuristics for improved recall.

Configuration example:

from semantica.deduplication import DuplicateDetector

dup_detector = DuplicateDetector(
    strategy="semantic_v2",
    min_similarity=0.75
)
duplicate_pairs = dup_detector.detect_duplicates(entities)

Stage 3: Entity Merging

When consolidation is required, the EntityMerger class in semantica/deduplication/entity_merger.py performs the merge. It preserves:

• Source provenance from all original entities
• Metadata tags and extraction confidence scores
• The most confident values for each property

The merger also updates the underlying graph store to maintain referential integrity across relationships.

CLI Integration and Dry-Run Mode

The deduplication subsystem exposes a CLI command (semantica deduplicate) defined in semantica/cli.py. This supports:

# Preview duplicates without modifying data

semantica deduplicate --dry-run --output json

# Execute merge with specific strategy

semantica deduplicate --action merge --strategy semantic_v2

Interaction Between Conflict Detection and Deduplication

Both subsystems share foundational infrastructure that enables seamless coordination:

• Common entity identifier model — conflicts detected pre-deduplication remain valid after merging
• Shared SourceTracker — provenance entries merge automatically when entities consolidate
• Conflict retention — the merger preserves previously detected conflicts and reattributes them to the consolidated entity

This design allows flexible pipeline arrangements. Users can run conflict detection before deduplication to surface high-severity contradictions for manual resolution, or run deduplication first to reduce noise in subsequent conflict analysis.

Practical Implementation Examples

Conflict Detection

from semantica.conflicts import ConflictDetector

detector = ConflictDetector(
    track_provenance=True,
    auto_resolve_low=False
)

conflicts = detector.detect_value_conflicts(
    entities,
    property_name="name"
)

for c in conflicts:
    print(f"{c.entity_id}: {c.conflicting_values}")
    print(f"  Severity: {c.severity}, Confidence: {c.confidence}")

Deduplication with Merge

from semantica.deduplication import DuplicateDetector

dup_detector = DuplicateDetector(
    strategy="semantic_v2",
    min_similarity=0.75
)

# Detection

duplicate_pairs = dup_detector.detect_duplicates(entities)

# Dry-run to preview

preview = dup_detector.merge_duplicates(
    duplicate_pairs,
    dry_run=True
)
print(f"Would merge {len(preview)} entities")

# Execute merge

merged_entities = dup_detector.merge_duplicates(
    duplicate_pairs,
    dry_run=False
)

Key Source Files and Responsibilities

Component	Source Path	Primary Role
Conflict detection core	semantica/conflicts/conflict_detector.py	Detects value, type, relationship, temporal, and logical conflicts; manages severity scoring and provenance
Similarity methods	semantica/deduplication/methods.py	Implements calculate_similarity and detect_duplicates with strategy handling
Entity merging	semantica/deduplication/entity_merger.py	Consolidates duplicates via EntityMerger with provenance preservation
Public deduplication API	semantica/deduplication/__init__.py	Exports DuplicateDetector for programmatic and CLI use
CLI command routing	semantica/cli.py	Wires deduplication to command-line with dry-run, output format, and strategy options

Summary

• Semantica's conflict detection uses the ConflictDetector class in semantica/conflicts/conflict_detector.py to identify contradictory values, types, and relationships across sources while tracking provenance and severity
• Deduplication employs the DuplicateDetector class exposed from semantica/deduplication/__init__.py, coordinating similarity calculation, duplicate pair detection, and entity merging
• The three-stage deduplication pipeline—similarity scoring, duplicate detection, and entity merging—supports configurable strategies including the semantic_v2 heuristic with synonym awareness
• Both subsystems share the entity identifier model and SourceTracker provenance infrastructure, enabling coherent operation before or after one another
• Dry-run modes and structured reporting support safe, auditable data quality workflows at scale

Frequently Asked Questions

What triggers a conflict in Semantica's conflict detection system?

A conflict triggers when the ConflictDetector observes multiple distinct values for the same property on entities sharing a common identifier (id or entity_id). The detector groups entities by this stable identifier, collects values per source, and creates a Conflict instance whenever uniqueness violations occur. Severity and confidence scores then characterize the divergence for prioritization.

How does Semantica choose which values to keep when merging duplicates?

The EntityMerger class in semantica/deduplication/entity_merger.py applies a confidence-preservation strategy: it retains the most confident value for each property while preserving all source provenance in the merged record. Metadata and extraction confidence from all constituent entities survive the merge, ensuring no source attribution is lost during consolidation.

Can I run conflict detection and deduplication independently?

Yes. The subsystems are decoupled by design. You can execute ConflictDetector.detect_value_conflicts() without any deduplication step, or run DuplicateDetector.detect_duplicates() and merge_duplicates() without conflict analysis. Many deployments run conflict detection first to surface critical contradictions for manual review, then deduplicate to clean remaining redundancy.

What similarity strategies does Semantica's deduplication support?

The DuplicateDetector accepts a strategy parameter that routes to implementations in semantica/deduplication/methods.py. The base implementation supports lexical, semantic, and fuzzy metrics. The semantic_v2 strategy extends this with synonym map resolution and domain-specific heuristics for improved matching on semantically equivalent but superficially different entities.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}