# How Semantica's ProvenanceManager Tracks Lineage Across Entities and Relationships

> Discover how Semantica's ProvenanceManager tracks lineage across entities and relationships using a hash-chained system for full-graph bidirectional tracing. Understand your data's journey.

- Repository: [Semantica /semantica](https://github.com/semantica-agi/semantica)
- Tags: how-to-guide
- Published: 2026-08-20

---

**Semantica's `ProvenanceManager` provides unified lineage tracking for entities, relationships, chunks, and property sources through a hash-chained `ProvenanceEntry` system that enables full-graph bidirectional tracing.**

The `ProvenanceManager` in semantica-agi/semantica serves as the central authority for recording and querying data provenance. It implements a W3C PROV-O compliant model where every tracked item becomes a node in a traversable lineage graph, with cryptographic checksums ensuring integrity across the entire chain.

## Core Tracking Mechanisms

### Entity Provenance with Parent Linking

The `track_entity()` method in [`semantica/provenance/manager.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance/manager.py) (lines 66-78) creates provenance records that capture derivation relationships through multiple inference strategies:

- Explicit `parent_entity_id` parameter
- `derived_from` field in the metadata dictionary
- Heuristic fallback treating the source document as parent

```python
from semantica.provenance.manager import ProvenanceManager

prov = ProvenanceManager()

# Track an entity with explicit parent linkage

prov.track_entity(
    entity_id="paper_002",
    source="doi:10.1000/abc456",
    metadata={"derived_from": "paper_001", "title": "Extension of X"},
)

```

### Relationship Provenance

Relationships participate equally in the lineage graph. The `track_relationship()` method (lines 22-28) uses the same `ProvenanceEntry` schema but sets `entity_type="relationship"`, enabling edges between entities to have their own audit trail:

```python
prov.track_relationship(
    relationship_id="cites_001",
    source="doi:10.1000/abc456",
    metadata={"type": "cites"},
)

```

## Storage and Integrity Guarantees

### Hash-Chained Entry Persistence

The `_save_entry()` method (lines 60-71) implements a tamper-evident storage layer:

1. Assigns monotonically increasing `sequence_id`
2. Computes cryptographic checksum of entry contents
3. Links to `previous_checksum` forming a **Merkle-style chain**
4. Writes to configured backend (in-memory or SQLite)

This checksum chain enables independent verification of lineage integrity without trusting the storage layer.

### Batch Atomic Operations

For high-throughput scenarios, `track_entities_batch()` and `track_chunks_batch()` (lines 48-57) wrap operations in SQLite savepoints, preserving lineage links across thousands of records with atomic commit semantics:

```python

# Batch insert with lineage preservation

entities = [
    {"entity_id": f"chunk_{i}", "source": "doc.txt", "metadata": {"index": i}}
    for i in range(1000)
]
prov.track_entities_batch(entities)  # Single atomic transaction

```

## Bidirectional Lineage Queries

### Upstream Ancestor Tracing

`get_lineage()` (lines 55-66) performs breadth-first search upstream via `parent_entity_id` and `used_entities` references, returning:

- Complete ancestor chain
- Aggregated metadata across all predecessors
- Deduplicated source documents
- Integrity verification status

```python
lineage = prov.get_lineage("paper_002")
print(lineage["entity_count"])       # → 2

print(lineage["source_documents"])   # → ['doi:10.1000/abc456', 'doi:10.1000/xyz123']

```

### Downstream Descendant Tracing

`trace_descendants()` (lines 44-48) mirrors the upstream logic but follows reverse edges—entities that reference the target `entity_id`. The `get_descendants()` wrapper formats results identically to `get_lineage()`:

```python
descendants = prov.get_descendants("paper_001")
print(descendants["entity_count"])   # → 1 (who cites this paper)

```

## Schema and Storage Architecture

| Component | File Path | Responsibility |
|-----------|-----------|--------------|
| `ProvenanceManager` | [`semantica/provenance/manager.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance/manager.py) | Entry creation, lineage queries, integrity checks, RDF export |
| `ProvenanceEntry` | [`semantica/provenance/schemas.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance/schemas.py) | Dataclass defining `entity_id`, `entity_type`, `parent_entity_id`, `checksum`, `metadata`, timestamps |
| Storage backends | [`semantica/provenance/storage.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance/storage.py) | `InMemoryStorage` for testing, `SQLiteStorage` for production with `trace_lineage()` and `trace_descendants()` graph traversal |

## Complete Working Example

```python
from semantica.provenance.manager import ProvenanceManager

prov = ProvenanceManager()

# Establish derivation chain

prov.track_entity(
    entity_id="paper_001",
    source="doi:10.1000/xyz123",
    metadata={"title": "Deep Learning for X"},
)

prov.track_entity(
    entity_id="paper_002",
    source="doi:10.1000/abc456",
    metadata={"derived_from": "paper_001", "title": "Extension of X"},
)

# Record citation relationship with its own provenance

prov.track_relationship(
    relationship_id="cites_001",
    source="doi:10.1000/abc456",
    metadata={"type": "cites", "from": "paper_002", "to": "paper_001"},
)

# Query both directions

upstream = prov.get_lineage("paper_002")
downstream = prov.get_descendants("paper_001")

# Verify chain integrity

assert upstream["integrity_verified"] is True

```

## Summary

- **`track_entity()`** and **`track_relationship()`** create unified `ProvenanceEntry` records with parent inference
- **Hash-chained checksums** in `_save_entry()` guarantee tamper-evident lineage storage
- **Bidirectional graph traversal** via `get_lineage()` (ancestors) and `get_descendants()` (successors)
- **Batch operations** preserve atomicity and linkage integrity at scale
- **W3C PROV-O compliance** enables RDF export for external provenance systems

## Frequently Asked Questions

### How does Semantica infer parent relationships when not explicitly provided?

The `track_entity()` method applies a three-tier fallback: first checking the `parent_entity_id` parameter, then examining `metadata["derived_from"]`, and finally defaulting to the source document identifier as the logical parent. This heuristic ensures every entity has placement in the lineage graph even with incomplete metadata.

### What storage backends does ProvenanceManager support?

According to [`semantica/provenance/storage.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance/storage.py), the manager supports `InMemoryStorage` for development/testing and `SQLiteStorage` for production persistence. Both implement identical `trace_lineage()` and `trace_descendants()` graph traversal methods, with SQLite providing durability and concurrent access.

### Can lineage be verified independently of the storage system?

Yes. Each `ProvenanceEntry` contains a cryptographic checksum and references the previous entry's checksum, forming a chain that can be recomputed and verified against stored values. The `get_lineage()` response includes an `integrity_verified` boolean indicating whether the recomputed chain matches stored checksums.

### How does relationship tracking differ from entity tracking?

Relationships use the identical `ProvenanceEntry` schema but set `entity_type="relationship"`, allowing them to participate in lineage queries as first-class nodes. The `track_relationship()` method accepts `relationship_id` rather than `entity_id` but otherwise follows the same parent-linking and storage logic as `track_entity()`.