How Semantica Performs Conflict Detection and Deduplication on Ingested Data

Semantica detects conflicts and deduplicates ingested data through a three-stage vector-based pipeline—embedding entities, clustering by cosine similarity, and merging duplicates—while the reasoning layer hashes and deduplicates rules to prevent contradictory inference paths.

The semantica-agi/semantica repository implements a robust conflict detection and deduplication system designed for knowledge graph ingestion. When raw RDF-style triplets enter the system, Semantica coordinates between its deduplication package and reasoning engine to eliminate redundant entities and prevent conflicting inference rules. This article examines the exact implementation files and methods that power this process.

The Three-Stage Deduplication Pipeline

Semantica’s deduplication logic resides in the semantica/deduplication/ package and executes in three distinct phases before data reaches the graph store.

Vector Embedding Generation

Every entity—whether a subject, object, or predicate—is transformed into a dense vector representation using the configured embedding provider. This occurs during the initialization of the DuplicateDetector class in semantica/deduplication/duplicate_detector.py.

The __init__ method loads the embedding provider from semantica/deduplication/embedding_provider.py, which interfaces with vector stores such as Weaviate or Qdrant. Converting raw textual entities into embeddings enables mathematical similarity comparisons across the entire dataset.

Similarity Calculation and Clustering

Once embedded, Semantica calculates pairwise cosine similarity between all entities. The core logic lives in semantica/deduplication/methods.py, specifically within the calculate_similarity and detect_duplicate_groups functions.

Entities exceeding a configurable similarity threshold (defaulting to 0.85 in typical configurations) are grouped into duplicate clusters. The system supports multiple clustering strategies controlled by the strategy parameter:

• Single-link clustering (default): Groups entities if any pair meets the threshold, creating broader clusters.
• Complete-link clustering: Only groups entities where all pairs exceed the threshold, producing tighter, more conservative clusters.

The build_clusters function in semantica/deduplication/methods.py implements these algorithms, returning duplicate groups ready for merging.

Entity Merging and Canonicalization

The final stage resolves duplicate clusters into single canonical entities. The EntityMerger class in semantica/deduplication/entity_merger.py handles this through its merge_duplicates method.

For each cluster, the merger:

1. Constructs a canonical representation by aggregating metadata from all duplicate entities.
2. Updates the underlying graph store to point all references to the canonical entity.
3. Records provenance data tracking which triples were collapsed, enabling downstream audit trails.

This process ensures that ingested knowledge graphs remain compact and semantically consistent without losing historical context.

Rule-Level Conflict Detection

Beyond entity deduplication, Semantica prevents logical conflicts during inference. The reasoning engine, implemented in semantica/reasoning/reasoner.py, deduplicates identical rules before adding them to the Rete network.

The Reasoner.add_rule method hashes the rule’s conditions and conclusions to generate a unique fingerprint. When a new rule arrives, the engine checks this hash against existing rules. If a match exists, the new rule is silently ignored, preventing redundant or contradictory inference paths. This idempotent behavior is verified in tests/reasoning/test_reasoner.py, which confirms that adding the same rule twice results in a single stored rule while distinct rules with different conditions are preserved separately.

Using the Conflict Detection API

Python Implementation

Integrate deduplication directly into your ingestion pipeline using the DuplicateDetector class:

from semantica.deduplication import DuplicateDetector

# Initialize with vector store and similarity threshold

detector = DuplicateDetector(
    vector_store=my_vector_store,
    similarity_threshold=0.85,
    strategy="single_link"  # or "complete_link"

)

# Detect duplicate groups from ingested triplets

duplicate_groups = detector.detect_duplicate_groups(triplets)

# Execute merge and update graph store

detector.merge_duplicates(duplicate_groups)

To prevent rule conflicts in the reasoning layer:

from semantica.reasoning import Reasoner

reasoner = Reasoner()

# Duplicate rules are automatically ignored based on content hash

reasoner.add_rule(
    conditions=[("person", "hasAge", "?age"), ("?age", ">", "18")],
    conclusion=("person", "isAdult", "true")
)

# Second call with identical parameters is deduplicated automatically

reasoner.add_rule(
    conditions=[("person", "hasAge", "?age"), ("?age", ">", "18")],
    conclusion=("person", "isAdult", "true")
)

Command-Line Interface

Semantica exposes deduplication operations through its CLI for operational workflows:

# Preview potential merges without modifying data

semantica deduplicate --dry-run --strategy single_link

# Execute merge and generate JSON report

semantica deduplicate --action merge --output ./deduplication-report.json

# Adjust similarity threshold for stricter matching

semantica deduplicate --similarity-threshold 0.92 --action merge

Summary

• Three-stage pipeline: Semantica deduplicates via embedding (embedding_provider.py), cosine similarity clustering (methods.py), and canonical merging (entity_merger.py).
• Configurable clustering: Choose between single-link and complete-link strategies in detect_duplicate_groups based on precision requirements.
• Hash-based rule deduplication: The Reasoner.add_rule method in semantica/reasoning/reasoner.py prevents conflicting inference by hashing rule conditions and conclusions.
• Provenance tracking: The EntityMerger records merge history, ensuring traceability from raw ingestion to canonical graph state.
• Dual interfaces: Both Python API and CLI support dry-run capabilities and threshold tuning for production data pipelines.

Frequently Asked Questions

How does Semantica calculate similarity between entities?

Semantica converts entities to dense vectors using the embedding provider, then computes cosine similarity between vector pairs. The calculate_similarity function in semantica/deduplication/methods.py performs this calculation, comparing embeddings against a configurable threshold (typically 0.85) to determine potential duplicates.

What clustering strategies does Semantica support for deduplication?

The system supports single-link and complete-link clustering algorithms, controlled via the strategy parameter in semantica/deduplication/methods.py. Single-link clustering (default) groups entities if any pairwise similarity exceeds the threshold, while complete-link requires all pairs within a cluster to meet the threshold, yielding more conservative groupings.

How does the reasoner prevent duplicate inference rules?

Before adding a rule to the Rete network, Reasoner.add_rule in semantica/reasoning/reasoner.py generates a hash from the rule’s conditions and conclusions. If this hash matches an existing rule, the new addition is discarded. This ensures idempotent rule insertion and prevents contradictory or redundant inference paths from entering the knowledge base.

Can the similarity threshold be adjusted for different data types?

Yes. The DuplicateDetector class accepts a similarity_threshold parameter during initialization, allowing pipelines to tune sensitivity per dataset. Higher thresholds (e.g., 0.95) reduce false positives for messy data, while lower thresholds (e.g., 0.80) catch more potential duplicates in clean, standardized vocabularies.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}