How ProvenanceManager Tracks Lineage Across Multi-Source Entity Ingestion

ProvenanceManager tracks lineage across multi-source entity ingestion by normalizing input sources at ingestion time, persisting a directed graph of parent-child relationships through the ProvenanceStorage interface, and executing breadth-first search traversals to reconstruct complete provenance chains spanning disparate documents, APIs, and files.

The ProvenanceManager in the semantica-agi/semantica repository serves as the central authority for unified provenance tracking across entities, relationships, chunks, and property sources. When identical or derived entities are ingested from multiple upstream origins—ranging from academic DOIs to local file paths—the manager constructs a complete audit trail that satisfies W3C PROV-O compliance requirements. Understanding how ProvenanceManager tracks lineage across multi-source entity ingestion is essential for building trustworthy AI systems that demand full data traceability and source accountability.

The Three-Layer Lineage Architecture

The lineage-tracking mechanism operates through three distinct architectural layers that work together to maintain provenance integrity across heterogeneous data sources.

Input Normalization and Parent Resolution

When an entity is first tracked via track_entity, track_chunk, or track_property_source, the manager immediately records the source document and attempts to resolve a parent entity. In semantica/provenance/manager.py (lines 63-75), the method examines three potential sources of parentage:

• Explicit parent_entity_id: Supplied directly by the caller as a string identifier
• Metadata derivation: The metadata["derived_from"] key contains a reference to an upstream entity
• Source heuristic: If the source string itself exists as an entity in storage, it becomes the implicit parent

This resolution logic appears in lines 108-122 of manager.py, where the system checks for explicit IDs first, falls back to metadata keys, and finally treats the source document as a parent node if a corresponding entry exists in the storage backend. This creates the explicit edges necessary for cross-source graph traversal.

Persistent Lineage Graph Storage

Every provenance entry persists through a backend implementing the ProvenanceStorage interface. Both InMemoryStorage and SQLiteStorage expose the trace_lineage(entity_id, max_depth=None) method, which performs a breadth-first search (BFS) starting from the queried entity.

The BFS follows three distinct relationship types defined in semantica/provenance/storage.py:

• parent_entity_id: Represents direct derivation relationships
• used_entities: Captures prov:used semantic links
• derived_from_id: Tracks cross-source derivation chains

The InMemoryStorage implementation (lines 70-81 and 84-99) maintains the frontier as a queue, expanding entries until exhausting the graph or reaching max_depth. The SQLiteStorage implementation (lines 93-106 and 115-136) executes batched queries to minimize database round-trips while preserving identical traversal semantics. These edges concatenate into a single lineage chain capable of spanning dozens of distinct source documents.

High-Level Lineage Retrieval

The ProvenanceManager.get_lineage(entity_id) method (lines 47-61 in manager.py) orchestrates the retrieval process by calling the storage layer's trace_lineage and aggregating metadata from each visited entry. The returned dictionary contains:

• lineage_chain: An ordered list of ProvenanceEntry objects representing the traversal path
• source_documents: A deduplicated set of all source documents contributing to the entity
• metadata: Merged metadata from all entries, with precedence given to the most recent ingestion
• integrity_verified: Boolean indicating verification status of the chain

This aggregation provides callers with a unified, cross-source view of how the entity originated and transformed through upstream facts.

How Multi-Source Ingestion Is Handled

The system handles entities ingested from multiple origins through a strict recording and linking protocol:

1. Source Recording: Each call to track_entity stores the source argument as the source_document field within the ProvenanceEntry dataclass (defined in semantica/provenance/schemas.py, lines 88-92). This field persists the origin URI, DOI, or file path regardless of how many times the same entity identifier is ingested.

2. Parent Resolution: The three-tier fallback mechanism (explicit ID, metadata key, source heuristic) ensures that even when ingesting the same entity from a new source, the manager links it to previous occurrences or explicit derivation statements. This creates a directed acyclic graph where nodes can have multiple parents representing different source contributions.

3. BFS Traversal Across Sources: When trace_lineage executes, the BFS frontier expands across all three edge types without regard to source boundaries. Because parent_entity_id and derived_from_id edges can point to entries with different source_document values, the resulting lineage chain naturally spans heterogeneous ingestion sources while preserving the temporal and logical derivation order.

Practical Implementation Examples

The following examples demonstrate tracking an entity across multiple academic sources and retrieving its complete lineage:

from semantica.provenance.manager import ProvenanceManager

# Initialise a manager (in‑memory storage for quick demo)

prov_mgr = ProvenanceManager()

# Ingest an entity from source A

prov_mgr.track_entity(
    entity_id="gene_ABC1",
    source="doi:10.1234/exampleA",
    metadata={"label": "Gene ABC1"},
)

# Ingest the same entity from source B, indicating cross‑source derivation

prov_mgr.track_entity(
    entity_id="gene_ABC1",
    source="doi:10.5678/exampleB",
    metadata={"derived_from": "doi:10.1234/exampleA"},
    confidence=0.95,
)

# Retrieve the full lineage (covers both sources)

lineage = prov_mgr.get_lineage("gene_ABC1")
print("Sources involved:", lineage["source_documents"])
print("Aggregated metadata:", lineage["metadata"])

For batch ingestion scenarios, use the track_entities_batch method to efficiently record multiple entities from a common source:

# Batch ingest many entities from a common source

entities = [
    {"id": "protein_X", "metadata": {"function": "kinase"}},
    {"id": "protein_Y", "metadata": {"function": "phosphatase"}},
]
prov_mgr.track_entities_batch(entities, source="file:/data/proteins.tsv")

To analyze downstream impact rather than upstream lineage, retrieve descendant entities:

# Trace downstream descendants (who used this entity)

descendants = prov_mgr.get_descendants("gene_ABC1")
print("Number of downstream facts:", descendants["entity_count"])

Summary

• Input normalization resolves parent relationships through a three-tier cascade: explicit parent_entity_id, metadata["derived_from"] keys, or source-document heuristics.
• ProvenanceStorage backends (both in-memory and SQLite) implement trace_lineage using breadth-first search across parent_entity_id, used_entities, and derived_from_id edges.
• Cross-source traversal occurs naturally during BFS expansion, allowing lineage chains to span disparate documents, APIs, and files without manual source correlation.
• Unified aggregation via get_lineage produces a coherent view containing the ordered chain, deduplicated source documents, and merged metadata for any entity ingested from multiple origins.

Frequently Asked Questions

How does ProvenanceManager resolve parent entities when the same ID is ingested from multiple sources?

The manager applies a deterministic resolution hierarchy defined in semantica/provenance/manager.py (lines 108-122). It first checks for an explicit parent_entity_id argument, then examines metadata["derived_from"], and finally falls back to treating the source string itself as a parent if that source exists as an entity in storage. This ensures that multi-source ingestions create explicit graph edges linking new entries to their upstream origins, regardless of source heterogeneity.

What storage backends support lineage tracing in the Semantica framework?

The framework provides two concrete implementations of the ProvenanceStorage interface: InMemoryStorage for high-performance ephemeral tracking and SQLiteStorage for persistent provenance databases. Both implement trace_lineage with identical BFS semantics, though the SQLite version uses batched SQL queries (lines 115-136 in storage.py) for efficient retrieval of deep lineage chains.

What relationship types does the lineage tracer traverse during graph exploration?

The trace_lineage method follows three specific edge types to reconstruct provenance paths: parent_entity_id for direct derivation links, used_entities for PROV-O style usage relationships, and derived_from_id for cross-source derivation mappings. These edges are traversed breadth-first starting from the queried entity until exhausting the graph or reaching the optional max_depth limit.

How does the batch ingestion method handle provenance tracking?

The track_entities_batch method accepts a list of entity dictionaries and a common source argument, creating individual ProvenanceEntry records for each entity while applying the same parent resolution logic used in single-entity tracking. This ensures that bulk imports from files or API responses maintain full lineage fidelity without requiring iterative calls to track_entity, significantly improving throughput for large-scale ingestion pipelines.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}