How to Track Data Provenance End-to-End with Semantica's ProvenanceManager

Semantica's ProvenanceManager provides a unified, W3C PROV-O-compatible API that records, stores, and queries tamper-evident provenance across all platform modules using SHA-256 checksum chaining and pluggable storage backends.

The ProvenanceManager class in semantica/provenance/manager.py serves as the central orchestrator for data provenance in the Semantica framework. It unifies provenance tracking from the knowledge graph, document splitting, and conflict resolution subsystems into a single, auditable ledger—ensuring complete traceability from raw sources to derived entities.

Core Architecture of the Provenance System

The provenance stack is organized into five distinct layers, each with a dedicated implementation file:

Layer	Purpose	Key File
API façade	High-level interface (track_entity, track_chunk, get_lineage, export_prov)	semantica/provenance/manager.py
Schema layer	Data models: ProvenanceEntry, AgentRecord, ActivityRecord	semantica/provenance/schemas.py
Storage abstraction	Pluggable backends (SQLite for persistence, in-memory for testing)	semantica/provenance/storage.py
Integrity utilities	SHA-256 checksum computation and chain verification	semantica/provenance/integrity.py
Integration points	Per-subsystem trackers that delegate to the manager	semantica/kg/kg_provenance.py, semantica/split/provenance_tracker.py, semantica/conflicts/conflicts_provenance.py

Every major subsystem creates a private manager instance via self._prov_manager = ProvenanceManager() and forwards provenance calls through it. This design ensures all records converge into a single, queryable graph regardless of origin.

How Provenance Tracking Works

The ProvenanceManager implements a hash-linked ledger with the following lifecycle:

1. Initialization — Supply a storage_path to enable persistent SQLite storage; omit it for InMemoryStorage.

2. Entry creation — Methods like track_entity() construct a ProvenanceEntry from schemas.ProvenanceEntry.

3. Checksum chaining — Before persistence, _save_entry() fetches the chain head, sets previous_checksum, and computes a new SHA-256 checksum (lines 60-68 of manager.py). This creates tamper-evidence through cryptographic linking.

4. Persistence — The storage layer executes within a transaction for atomicity.

5. Querying — get_lineage() walks ancestors; get_descendants() walks downstream dependencies.

6. Export — export_prov(format="json") serializes using DEFAULT_BASE_URI = "https://semantica.dev/ns#", ensuring URI consistency with knowledge graph exports.

Initializing the ProvenanceManager

Create a manager with persistent storage for production use:

from semantica.provenance import ProvenanceManager

# Persistent SQLite backend

prov = ProvenanceManager(storage_path="provenance.db")

# In-memory backend for testing (omit storage_path or pass None)

prov_test = ProvenanceManager()

The manager uses SQLiteStorage when storage_path is provided, falling back to InMemoryStorage otherwise. Both implementations share the same interface defined in semantica/provenance/storage.py.

Tracking Entities and Their Origins

Record the provenance of knowledge graph entities using track_entity():

# Track an entity derived from a source document

prov.track_entity(
    entity_id="person:alice",
    source="doc:contract_2024",
    activity="ingest",
    agent="pipeline:ingest_v1"
)

The source parameter captures the originating document, while activity and agent record what happened and who performed it—core concepts from the W3C PROV-O standard.

Tracking Document Chunks

For split or chunked documents, use track_chunk() to maintain parent-child relationships:

prov.track_chunk(
    chunk_id="chunk:alice_1",
    parent_entity="person:alice",
    activity="split",
    agent="pipeline:split_v2"
)

This creates a linkage between the chunk and its parent entity, enabling downstream reconstruction of document structure from provenance records alone.

Retrieving Data Lineage

Query upstream provenance for any tracked identifier:

# Get full lineage from source to current state

lineage = prov.get_lineage("person:alice")
for entry in lineage:
    print(f"{entry.timestamp}: {entry.activity} by {entry.agent}")

get_lineage() returns a list of ProvenanceEntry objects ordered from oldest to newest. For downstream impact analysis, use get_descendants() to find all entities derived from a given source.

Exporting Provenance Data

Serialize the provenance graph for external audit or compliance:

# Export as JSON-LD compatible with PROV-O

json_data = prov.export_prov(format="json")

# The export uses the default base URI for consistent namespace resolution

The export includes all entries with their checksum chain intact, enabling verification by external tools.

Batch Operations for Performance

For high-volume ingestion, use batch mode to reduce I/O overhead:

entries = [
    {"entity_id": "doc:1", "source": "web", "activity": "ingest"},
    {"entity_id": "doc:2", "source": "pdf", "activity": "ingest"},
    {"entity_id": "doc:3", "source": "api", "activity": "ingest"},
]

with prov._get_or_create_transaction() as txn:
    for ent in entries:
        prov.track_entity(**ent, _conn=txn)

The _conn parameter (lines 89-97 of manager.py) injects a shared transaction object, ensuring atomic persistence of all entries. This pattern reduces SQLite round-trips by 10-100× compared to individual calls.

Verifying Chain Integrity

Detect tampering or storage corruption using the integrity subsystem:

is_valid = prov.verify_chain()
if is_valid:
    print("Provenance chain integrity: VERIFIED")
else:
    raise SecurityError("Provenance chain corruption detected")

verify_chain() delegates to compute_checksum() and verify_checksum() in semantica/provenance/integrity.py, recomputing hashes and verifying the previous_checksum linkage across all entries.

Using the CLI for Provenance Operations

The semantica provenance command group provides operational access without code:

# List available provenance commands

semantica provenance --help

# Export complete provenance graph

semantica provenance export --format json > provenance_dump.json

# Audit a specific entity's history

semantica provenance audit person:alice --output alice_audit.json

# Trace lineage via command line

semantica provenance lineage person:alice --depth 5

CLI commands instantiate ProvenanceManager using configuration from semantica/cli.py (around lines 2500-2600), ensuring consistent behavior with programmatic usage.

Integration Across Subsystems

The following integration modules demonstrate how each subsystem wires into the provenance manager:

• semantica/kg/kg_provenance.py — Tracks entity and relationship creation in the knowledge graph
• semantica/split/provenance_tracker.py — Records document chunking operations
• semantica/conflicts/conflicts_provenance.py — Captures source conflict detection and resolution

Each module follows the same pattern: instantiate ProvenanceManager, then delegate tracking calls through it. This ensures a unified provenance graph despite heterogeneous data processing stages.

Summary

• The ProvenanceManager in semantica/provenance/manager.py provides a single API for end-to-end provenance tracking across all Semantica modules
• SHA-256 checksum chaining creates a tamper-evident ledger via previous_checksum linkages in ProvenanceEntry
• Pluggable storage supports both production SQLite persistence (storage_path) and in-memory testing
• Batch transactions via _get_or_create_transaction() optimize high-volume ingestion
• W3C PROV-O compatibility ensures interoperability through export_prov() and standardized schema definitions
• CLI integration enables operational workflows without requiring Python code

Frequently Asked Questions

What provenance standards does Semantica follow?

Semantica implements W3C PROV-O compatibility through its schema definitions in semantica/provenance/schemas.py and export formats. The DEFAULT_BASE_URI = "https://semantica.dev/ns#" ensures that exported URIs resolve consistently with knowledge graph identifiers.

How does the checksum chain prevent tampering?

Each ProvenanceEntry stores a previous_checksum referencing the SHA-256 hash of the prior entry. Before persistence, manager.py computes a new checksum combining entry content with the previous hash (lines 60-68). Any modification breaks the chain, detectable via verify_chain().

Can I use ProvenanceManager without SQLite?

Yes—omit storage_path to use InMemoryStorage for testing or transient pipelines. The storage abstraction in semantica/provenance/storage.py allows implementing custom backends (e.g., PostgreSQL, S3) by subclassing the base storage interface.

How do I track provenance across distributed pipeline stages?

Each stage instantiates ProvenanceManager with the same storage_path (shared filesystem or network-accessible SQLite). For truly distributed deployments, implement a custom storage backend that writes to a central provenance service while maintaining the same ProvenanceEntry schema and checksum semantics.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}