# How to Track Data Provenance End-to-End with Semantica's ProvenanceManager

> Master end-to-end data provenance tracking with Semantica's ProvenanceManager. Learn how this W3C PROV-O compatible API ensures tamper-evident recording and querying across your platform.

- Repository: [Semantica /semantica](https://github.com/semantica-agi/semantica)
- Tags: how-to-guide
- Published: 2026-08-15

---

**Semantica's `ProvenanceManager` provides a unified, W3C PROV-O-compatible API that records, stores, and queries tamper-evident provenance across all platform modules using SHA-256 checksum chaining and pluggable storage backends.**

The `ProvenanceManager` class in [`semantica/provenance/manager.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance/manager.py) serves as the central orchestrator for data provenance in the Semantica framework. It unifies provenance tracking from the knowledge graph, document splitting, and conflict resolution subsystems into a single, auditable ledger—ensuring complete traceability from raw sources to derived entities.

## Core Architecture of the Provenance System

The provenance stack is organized into five distinct layers, each with a dedicated implementation file:

| Layer | Purpose | Key File |
|-------|---------|----------|
| **API façade** | High-level interface (`track_entity`, `track_chunk`, `get_lineage`, `export_prov`) | [`semantica/provenance/manager.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance/manager.py) |
| **Schema layer** | Data models: `ProvenanceEntry`, `AgentRecord`, `ActivityRecord` | [`semantica/provenance/schemas.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance/schemas.py) |
| **Storage abstraction** | Pluggable backends (SQLite for persistence, in-memory for testing) | [`semantica/provenance/storage.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance/storage.py) |
| **Integrity utilities** | SHA-256 checksum computation and chain verification | [`semantica/provenance/integrity.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance/integrity.py) |
| **Integration points** | Per-subsystem trackers that delegate to the manager | [`semantica/kg/kg_provenance.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/kg/kg_provenance.py), [`semantica/split/provenance_tracker.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/split/provenance_tracker.py), [`semantica/conflicts/conflicts_provenance.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/conflicts/conflicts_provenance.py) |

Every major subsystem creates a private manager instance via `self._prov_manager = ProvenanceManager()` and forwards provenance calls through it. This design ensures all records converge into a single, queryable graph regardless of origin.

## How Provenance Tracking Works

The `ProvenanceManager` implements a hash-linked ledger with the following lifecycle:

1. **Initialization** — Supply a `storage_path` to enable persistent SQLite storage; omit it for `InMemoryStorage`.

2. **Entry creation** — Methods like `track_entity()` construct a `ProvenanceEntry` from `schemas.ProvenanceEntry`.

3. **Checksum chaining** — Before persistence, `_save_entry()` fetches the chain head, sets `previous_checksum`, and computes a new SHA-256 checksum (lines 60-68 of [`manager.py`](https://github.com/semantica-agi/semantica/blob/main/manager.py)). This creates tamper-evidence through cryptographic linking.

4. **Persistence** — The storage layer executes within a transaction for atomicity.

5. **Querying** — `get_lineage()` walks ancestors; `get_descendants()` walks downstream dependencies.

6. **Export** — `export_prov(format="json")` serializes using `DEFAULT_BASE_URI = "https://semantica.dev/ns#"`, ensuring URI consistency with knowledge graph exports.

## Initializing the ProvenanceManager

Create a manager with persistent storage for production use:

```python
from semantica.provenance import ProvenanceManager

# Persistent SQLite backend

prov = ProvenanceManager(storage_path="provenance.db")

# In-memory backend for testing (omit storage_path or pass None)

prov_test = ProvenanceManager()

```

The manager uses `SQLiteStorage` when `storage_path` is provided, falling back to `InMemoryStorage` otherwise. Both implementations share the same interface defined in [`semantica/provenance/storage.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance/storage.py).

## Tracking Entities and Their Origins

Record the provenance of knowledge graph entities using `track_entity()`:

```python

# Track an entity derived from a source document

prov.track_entity(
    entity_id="person:alice",
    source="doc:contract_2024",
    activity="ingest",
    agent="pipeline:ingest_v1"
)

```

The `source` parameter captures the originating document, while `activity` and `agent` record *what* happened and *who* performed it—core concepts from the W3C PROV-O standard.

## Tracking Document Chunks

For split or chunked documents, use `track_chunk()` to maintain parent-child relationships:

```python
prov.track_chunk(
    chunk_id="chunk:alice_1",
    parent_entity="person:alice",
    activity="split",
    agent="pipeline:split_v2"
)

```

This creates a linkage between the chunk and its parent entity, enabling downstream reconstruction of document structure from provenance records alone.

## Retrieving Data Lineage

Query upstream provenance for any tracked identifier:

```python

# Get full lineage from source to current state

lineage = prov.get_lineage("person:alice")
for entry in lineage:
    print(f"{entry.timestamp}: {entry.activity} by {entry.agent}")

```

`get_lineage()` returns a list of `ProvenanceEntry` objects ordered from oldest to newest. For downstream impact analysis, use `get_descendants()` to find all entities derived from a given source.

## Exporting Provenance Data

Serialize the provenance graph for external audit or compliance:

```python

# Export as JSON-LD compatible with PROV-O

json_data = prov.export_prov(format="json")

# The export uses the default base URI for consistent namespace resolution

```

The export includes all entries with their checksum chain intact, enabling verification by external tools.

## Batch Operations for Performance

For high-volume ingestion, use batch mode to reduce I/O overhead:

```python
entries = [
    {"entity_id": "doc:1", "source": "web", "activity": "ingest"},
    {"entity_id": "doc:2", "source": "pdf", "activity": "ingest"},
    {"entity_id": "doc:3", "source": "api", "activity": "ingest"},
]

with prov._get_or_create_transaction() as txn:
    for ent in entries:
        prov.track_entity(**ent, _conn=txn)

```

The `_conn` parameter (lines 89-97 of [`manager.py`](https://github.com/semantica-agi/semantica/blob/main/manager.py)) injects a shared transaction object, ensuring atomic persistence of all entries. This pattern reduces SQLite round-trips by 10-100× compared to individual calls.

## Verifying Chain Integrity

Detect tampering or storage corruption using the integrity subsystem:

```python
is_valid = prov.verify_chain()
if is_valid:
    print("Provenance chain integrity: VERIFIED")
else:
    raise SecurityError("Provenance chain corruption detected")

```

`verify_chain()` delegates to `compute_checksum()` and `verify_checksum()` in [`semantica/provenance/integrity.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance/integrity.py), recomputing hashes and verifying the `previous_checksum` linkage across all entries.

## Using the CLI for Provenance Operations

The `semantica provenance` command group provides operational access without code:

```bash

# List available provenance commands

semantica provenance --help

# Export complete provenance graph

semantica provenance export --format json > provenance_dump.json

# Audit a specific entity's history

semantica provenance audit person:alice --output alice_audit.json

# Trace lineage via command line

semantica provenance lineage person:alice --depth 5

```

CLI commands instantiate `ProvenanceManager` using configuration from [`semantica/cli.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/cli.py) (around lines 2500-2600), ensuring consistent behavior with programmatic usage.

## Integration Across Subsystems

The following integration modules demonstrate how each subsystem wires into the provenance manager:

- **[`semantica/kg/kg_provenance.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/kg/kg_provenance.py)** — Tracks entity and relationship creation in the knowledge graph
- **[`semantica/split/provenance_tracker.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/split/provenance_tracker.py)** — Records document chunking operations
- **[`semantica/conflicts/conflicts_provenance.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/conflicts/conflicts_provenance.py)** — Captures source conflict detection and resolution

Each module follows the same pattern: instantiate `ProvenanceManager`, then delegate tracking calls through it. This ensures a unified provenance graph despite heterogeneous data processing stages.

## Summary

- The `ProvenanceManager` in [`semantica/provenance/manager.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance/manager.py) provides a single API for end-to-end provenance tracking across all Semantica modules
- **SHA-256 checksum chaining** creates a tamper-evident ledger via `previous_checksum` linkages in `ProvenanceEntry`
- **Pluggable storage** supports both production SQLite persistence (`storage_path`) and in-memory testing
- **Batch transactions** via `_get_or_create_transaction()` optimize high-volume ingestion
- **W3C PROV-O compatibility** ensures interoperability through `export_prov()` and standardized schema definitions
- **CLI integration** enables operational workflows without requiring Python code

## Frequently Asked Questions

### What provenance standards does Semantica follow?

Semantica implements **W3C PROV-O** compatibility through its schema definitions in [`semantica/provenance/schemas.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance/schemas.py) and export formats. The `DEFAULT_BASE_URI = "https://semantica.dev/ns#"` ensures that exported URIs resolve consistently with knowledge graph identifiers.

### How does the checksum chain prevent tampering?

Each `ProvenanceEntry` stores a `previous_checksum` referencing the SHA-256 hash of the prior entry. Before persistence, [`manager.py`](https://github.com/semantica-agi/semantica/blob/main/manager.py) computes a new checksum combining entry content with the previous hash (lines 60-68). Any modification breaks the chain, detectable via `verify_chain()`.

### Can I use ProvenanceManager without SQLite?

Yes—omit `storage_path` to use `InMemoryStorage` for testing or transient pipelines. The storage abstraction in [`semantica/provenance/storage.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance/storage.py) allows implementing custom backends (e.g., PostgreSQL, S3) by subclassing the base storage interface.

### How do I track provenance across distributed pipeline stages?

Each stage instantiates `ProvenanceManager` with the same `storage_path` (shared filesystem or network-accessible SQLite). For truly distributed deployments, implement a custom storage backend that writes to a central provenance service while maintaining the same `ProvenanceEntry` schema and checksum semantics.