# How ProvenanceManager Tracks Lineage Across Multi-Source Entity Ingestion

> Discover how ProvenanceManager tracks lineage across multi-source entity ingestion by normalizing sources, persisting relationships, and performing graph traversals for complete provenance chains.

- Repository: [Semantica /semantica](https://github.com/semantica-agi/semantica)
- Tags: how-to-guide
- Published: 2026-08-17

---

**ProvenanceManager tracks lineage across multi-source entity ingestion by normalizing input sources at ingestion time, persisting a directed graph of parent-child relationships through the `ProvenanceStorage` interface, and executing breadth-first search traversals to reconstruct complete provenance chains spanning disparate documents, APIs, and files.**

The `ProvenanceManager` in the semantica-agi/semantica repository serves as the central authority for unified provenance tracking across entities, relationships, chunks, and property sources. When identical or derived entities are ingested from multiple upstream origins—ranging from academic DOIs to local file paths—the manager constructs a complete audit trail that satisfies W3C PROV-O compliance requirements. Understanding how ProvenanceManager tracks lineage across multi-source entity ingestion is essential for building trustworthy AI systems that demand full data traceability and source accountability.

## The Three-Layer Lineage Architecture

The lineage-tracking mechanism operates through three distinct architectural layers that work together to maintain provenance integrity across heterogeneous data sources.

### Input Normalization and Parent Resolution

When an entity is first tracked via `track_entity`, `track_chunk`, or `track_property_source`, the manager immediately records the source document and attempts to resolve a parent entity. In [`semantica/provenance/manager.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance/manager.py) (lines 63-75), the method examines three potential sources of parentage:

- **Explicit `parent_entity_id`**: Supplied directly by the caller as a string identifier
- **Metadata derivation**: The `metadata["derived_from"]` key contains a reference to an upstream entity
- **Source heuristic**: If the source string itself exists as an entity in storage, it becomes the implicit parent

This resolution logic appears in lines 108-122 of [`manager.py`](https://github.com/semantica-agi/semantica/blob/main/manager.py), where the system checks for explicit IDs first, falls back to metadata keys, and finally treats the source document as a parent node if a corresponding entry exists in the storage backend. This creates the explicit edges necessary for cross-source graph traversal.

### Persistent Lineage Graph Storage

Every provenance entry persists through a backend implementing the `ProvenanceStorage` interface. Both `InMemoryStorage` and `SQLiteStorage` expose the `trace_lineage(entity_id, max_depth=None)` method, which performs a **breadth-first search (BFS)** starting from the queried entity.

The BFS follows three distinct relationship types defined in [`semantica/provenance/storage.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance/storage.py):

- **`parent_entity_id`**: Represents direct derivation relationships
- **`used_entities`**: Captures `prov:used` semantic links
- **`derived_from_id`**: Tracks cross-source derivation chains

The `InMemoryStorage` implementation (lines 70-81 and 84-99) maintains the frontier as a queue, expanding entries until exhausting the graph or reaching `max_depth`. The `SQLiteStorage` implementation (lines 93-106 and 115-136) executes batched queries to minimize database round-trips while preserving identical traversal semantics. These edges concatenate into a single lineage chain capable of spanning dozens of distinct source documents.

### High-Level Lineage Retrieval

The `ProvenanceManager.get_lineage(entity_id)` method (lines 47-61 in [`manager.py`](https://github.com/semantica-agi/semantica/blob/main/manager.py)) orchestrates the retrieval process by calling the storage layer's `trace_lineage` and aggregating metadata from each visited entry. The returned dictionary contains:

- **`lineage_chain`**: An ordered list of `ProvenanceEntry` objects representing the traversal path
- **`source_documents`**: A deduplicated set of all source documents contributing to the entity
- **`metadata`**: Merged metadata from all entries, with precedence given to the most recent ingestion
- **`integrity_verified`**: Boolean indicating verification status of the chain

This aggregation provides callers with a unified, cross-source view of how the entity originated and transformed through upstream facts.

## How Multi-Source Ingestion Is Handled

The system handles entities ingested from multiple origins through a strict recording and linking protocol:

1. **Source Recording**: Each call to `track_entity` stores the `source` argument as the `source_document` field within the `ProvenanceEntry` dataclass (defined in [`semantica/provenance/schemas.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance/schemas.py), lines 88-92). This field persists the origin URI, DOI, or file path regardless of how many times the same entity identifier is ingested.

2. **Parent Resolution**: The three-tier fallback mechanism (explicit ID, metadata key, source heuristic) ensures that even when ingesting the same entity from a new source, the manager links it to previous occurrences or explicit derivation statements. This creates a directed acyclic graph where nodes can have multiple parents representing different source contributions.

3. **BFS Traversal Across Sources**: When `trace_lineage` executes, the BFS frontier expands across all three edge types without regard to source boundaries. Because `parent_entity_id` and `derived_from_id` edges can point to entries with different `source_document` values, the resulting lineage chain naturally spans heterogeneous ingestion sources while preserving the temporal and logical derivation order.

## Practical Implementation Examples

The following examples demonstrate tracking an entity across multiple academic sources and retrieving its complete lineage:

```python
from semantica.provenance.manager import ProvenanceManager

# Initialise a manager (in‑memory storage for quick demo)

prov_mgr = ProvenanceManager()

# Ingest an entity from source A

prov_mgr.track_entity(
    entity_id="gene_ABC1",
    source="doi:10.1234/exampleA",
    metadata={"label": "Gene ABC1"},
)

# Ingest the same entity from source B, indicating cross‑source derivation

prov_mgr.track_entity(
    entity_id="gene_ABC1",
    source="doi:10.5678/exampleB",
    metadata={"derived_from": "doi:10.1234/exampleA"},
    confidence=0.95,
)

# Retrieve the full lineage (covers both sources)

lineage = prov_mgr.get_lineage("gene_ABC1")
print("Sources involved:", lineage["source_documents"])
print("Aggregated metadata:", lineage["metadata"])

```

For batch ingestion scenarios, use the `track_entities_batch` method to efficiently record multiple entities from a common source:

```python

# Batch ingest many entities from a common source

entities = [
    {"id": "protein_X", "metadata": {"function": "kinase"}},
    {"id": "protein_Y", "metadata": {"function": "phosphatase"}},
]
prov_mgr.track_entities_batch(entities, source="file:/data/proteins.tsv")

```

To analyze downstream impact rather than upstream lineage, retrieve descendant entities:

```python

# Trace downstream descendants (who used this entity)

descendants = prov_mgr.get_descendants("gene_ABC1")
print("Number of downstream facts:", descendants["entity_count"])

```

## Summary

- **Input normalization** resolves parent relationships through a three-tier cascade: explicit `parent_entity_id`, `metadata["derived_from"]` keys, or source-document heuristics.
- **ProvenanceStorage backends** (both in-memory and SQLite) implement `trace_lineage` using breadth-first search across `parent_entity_id`, `used_entities`, and `derived_from_id` edges.
- **Cross-source traversal** occurs naturally during BFS expansion, allowing lineage chains to span disparate documents, APIs, and files without manual source correlation.
- **Unified aggregation** via `get_lineage` produces a coherent view containing the ordered chain, deduplicated source documents, and merged metadata for any entity ingested from multiple origins.

## Frequently Asked Questions

### How does ProvenanceManager resolve parent entities when the same ID is ingested from multiple sources?

The manager applies a deterministic resolution hierarchy defined in [`semantica/provenance/manager.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance/manager.py) (lines 108-122). It first checks for an explicit `parent_entity_id` argument, then examines `metadata["derived_from"]`, and finally falls back to treating the source string itself as a parent if that source exists as an entity in storage. This ensures that multi-source ingestions create explicit graph edges linking new entries to their upstream origins, regardless of source heterogeneity.

### What storage backends support lineage tracing in the Semantica framework?

The framework provides two concrete implementations of the `ProvenanceStorage` interface: **`InMemoryStorage`** for high-performance ephemeral tracking and **`SQLiteStorage`** for persistent provenance databases. Both implement `trace_lineage` with identical BFS semantics, though the SQLite version uses batched SQL queries (lines 115-136 in [`storage.py`](https://github.com/semantica-agi/semantica/blob/main/storage.py)) for efficient retrieval of deep lineage chains.

### What relationship types does the lineage tracer traverse during graph exploration?

The `trace_lineage` method follows three specific edge types to reconstruct provenance paths: **`parent_entity_id`** for direct derivation links, **`used_entities`** for PROV-O style usage relationships, and **`derived_from_id`** for cross-source derivation mappings. These edges are traversed breadth-first starting from the queried entity until exhausting the graph or reaching the optional `max_depth` limit.

### How does the batch ingestion method handle provenance tracking?

The `track_entities_batch` method accepts a list of entity dictionaries and a common `source` argument, creating individual `ProvenanceEntry` records for each entity while applying the same parent resolution logic used in single-entity tracking. This ensures that bulk imports from files or API responses maintain full lineage fidelity without requiring iterative calls to `track_entity`, significantly improving throughput for large-scale ingestion pipelines.