How Semantica's ProvenanceManager Tracks Lineage Across Entities and Relationships

Semantica's ProvenanceManager provides unified lineage tracking for entities, relationships, chunks, and property sources through a hash-chained ProvenanceEntry system that enables full-graph bidirectional tracing.

The ProvenanceManager in semantica-agi/semantica serves as the central authority for recording and querying data provenance. It implements a W3C PROV-O compliant model where every tracked item becomes a node in a traversable lineage graph, with cryptographic checksums ensuring integrity across the entire chain.

Core Tracking Mechanisms

Entity Provenance with Parent Linking

The track_entity() method in semantica/provenance/manager.py (lines 66-78) creates provenance records that capture derivation relationships through multiple inference strategies:

• Explicit parent_entity_id parameter
• derived_from field in the metadata dictionary
• Heuristic fallback treating the source document as parent

from semantica.provenance.manager import ProvenanceManager

prov = ProvenanceManager()

# Track an entity with explicit parent linkage

prov.track_entity(
    entity_id="paper_002",
    source="doi:10.1000/abc456",
    metadata={"derived_from": "paper_001", "title": "Extension of X"},
)

Relationship Provenance

Relationships participate equally in the lineage graph. The track_relationship() method (lines 22-28) uses the same ProvenanceEntry schema but sets entity_type="relationship", enabling edges between entities to have their own audit trail:

prov.track_relationship(
    relationship_id="cites_001",
    source="doi:10.1000/abc456",
    metadata={"type": "cites"},
)

Storage and Integrity Guarantees

Hash-Chained Entry Persistence

The _save_entry() method (lines 60-71) implements a tamper-evident storage layer:

1. Assigns monotonically increasing sequence_id
2. Computes cryptographic checksum of entry contents
3. Links to previous_checksum forming a Merkle-style chain
4. Writes to configured backend (in-memory or SQLite)

This checksum chain enables independent verification of lineage integrity without trusting the storage layer.

Batch Atomic Operations

For high-throughput scenarios, track_entities_batch() and track_chunks_batch() (lines 48-57) wrap operations in SQLite savepoints, preserving lineage links across thousands of records with atomic commit semantics:

# Batch insert with lineage preservation

entities = [
    {"entity_id": f"chunk_{i}", "source": "doc.txt", "metadata": {"index": i}}
    for i in range(1000)
]
prov.track_entities_batch(entities)  # Single atomic transaction

Bidirectional Lineage Queries

Upstream Ancestor Tracing

get_lineage() (lines 55-66) performs breadth-first search upstream via parent_entity_id and used_entities references, returning:

• Complete ancestor chain
• Aggregated metadata across all predecessors
• Deduplicated source documents
• Integrity verification status

lineage = prov.get_lineage("paper_002")
print(lineage["entity_count"])       # → 2

print(lineage["source_documents"])   # → ['doi:10.1000/abc456', 'doi:10.1000/xyz123']

Downstream Descendant Tracing

trace_descendants() (lines 44-48) mirrors the upstream logic but follows reverse edges—entities that reference the target entity_id. The get_descendants() wrapper formats results identically to get_lineage():

descendants = prov.get_descendants("paper_001")
print(descendants["entity_count"])   # → 1 (who cites this paper)

Schema and Storage Architecture

Component	File Path	Responsibility
ProvenanceManager	semantica/provenance/manager.py	Entry creation, lineage queries, integrity checks, RDF export
ProvenanceEntry	semantica/provenance/schemas.py	Dataclass defining entity_id, entity_type, parent_entity_id, checksum, metadata, timestamps
Storage backends	semantica/provenance/storage.py	InMemoryStorage for testing, SQLiteStorage for production with trace_lineage() and trace_descendants() graph traversal

Complete Working Example

from semantica.provenance.manager import ProvenanceManager

prov = ProvenanceManager()

# Establish derivation chain

prov.track_entity(
    entity_id="paper_001",
    source="doi:10.1000/xyz123",
    metadata={"title": "Deep Learning for X"},
)

prov.track_entity(
    entity_id="paper_002",
    source="doi:10.1000/abc456",
    metadata={"derived_from": "paper_001", "title": "Extension of X"},
)

# Record citation relationship with its own provenance

prov.track_relationship(
    relationship_id="cites_001",
    source="doi:10.1000/abc456",
    metadata={"type": "cites", "from": "paper_002", "to": "paper_001"},
)

# Query both directions

upstream = prov.get_lineage("paper_002")
downstream = prov.get_descendants("paper_001")

# Verify chain integrity

assert upstream["integrity_verified"] is True

Summary

• track_entity() and track_relationship() create unified ProvenanceEntry records with parent inference
• Hash-chained checksums in _save_entry() guarantee tamper-evident lineage storage
• Bidirectional graph traversal via get_lineage() (ancestors) and get_descendants() (successors)
• Batch operations preserve atomicity and linkage integrity at scale
• W3C PROV-O compliance enables RDF export for external provenance systems

Frequently Asked Questions

How does Semantica infer parent relationships when not explicitly provided?

The track_entity() method applies a three-tier fallback: first checking the parent_entity_id parameter, then examining metadata["derived_from"], and finally defaulting to the source document identifier as the logical parent. This heuristic ensures every entity has placement in the lineage graph even with incomplete metadata.

What storage backends does ProvenanceManager support?

According to semantica/provenance/storage.py, the manager supports InMemoryStorage for development/testing and SQLiteStorage for production persistence. Both implement identical trace_lineage() and trace_descendants() graph traversal methods, with SQLite providing durability and concurrent access.

Can lineage be verified independently of the storage system?

Yes. Each ProvenanceEntry contains a cryptographic checksum and references the previous entry's checksum, forming a chain that can be recomputed and verified against stored values. The get_lineage() response includes an integrity_verified boolean indicating whether the recomputed chain matches stored checksums.

How does relationship tracking differ from entity tracking?

Relationships use the identical ProvenanceEntry schema but set entity_type="relationship", allowing them to participate in lineage queries as first-class nodes. The track_relationship() method accepts relationship_id rather than entity_id but otherwise follows the same parent-linking and storage logic as track_entity().

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}