# How to Implement W3C PROV-O Provenance Lineage Tracking in Python with Semantica

> Implement W3C PROV-O provenance lineage tracking in Python with Semantica. Track entities, query lineage, and export RDF-PROV effortlessly. Get started now.

- Repository: [Semantica /semantica](https://github.com/semantica-agi/semantica)
- Tags: how-to-guide
- Published: 2026-08-16

---

**To implement W3C PROV-O provenance lineage tracking in Semantica, instantiate `ProvenanceManager` from [`semantica/provenance/manager.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance/manager.py), use `track_entity()` to record agent-attributed activities, then query lineage via `get_lineage()` and export RDF-PROV with `export_prov()`.**

The `semantica-agi/semantica` repository provides a production-grade, **W3C PROV-O-compliant** provenance subsystem that captures complete entity derivation history, cryptographic integrity, and standardized RDF export. This guide walks through the complete implementation using the actual source architecture.

---

## Why W3C PROV-O Compliance Matters

**W3C PROV-O** is the W3C recommendation for representing provenance as linked data. Semantica's implementation maps every provenance record to standard PROV classes: `prov:Entity`, `prov:Agent`, and `prov:Activity`, with relationships like `prov:wasAttributedTo`, `prov:qualifiedAssociation`, and optional `prov:actedOnBehalfOf` delegation.

This ensures your provenance graphs are **interoperable** with external systems, **queryable** via SPARQL, and **verifiable** through cryptographic hash-chaining.

---

## Core Architecture Components

| Component | Role | Source Location |
|-----------|------|-----------------|
| **`ProvenanceManager`** | Central API for tracking, lineage queries, and RDF export | [`semantica/provenance/manager.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance/manager.py) |
| **`ProvenanceEntry`** | Immutable dataclass for provenance records | [`semantica/provenance/schemas.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance/schemas.py) |
| **`ProvenanceStorage`** | Abstract storage with `InMemoryStorage` and `SQLiteStorage` | [`semantica/provenance/storage.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance/storage.py) |
| **Integrity helpers** | `compute_checksum` / `verify_checksum` for tamper-evidence | [`semantica/provenance/integrity.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance/integrity.py) |
| **PROV-O Exporter** | `export_prov()` method generating Turtle, N-Triples, or JSON-LD | [`semantica/provenance/manager.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance/manager.py) |

The lineage engine traverses a **directed acyclic graph** built from `parent_entity_id`, `used_entities`, and `derived_from_id` relationships in each entry.

---

## Step 1: Initialize the ProvenanceManager

Create a manager with either in-memory or persistent SQLite storage:

```python
from semantica.provenance.manager import ProvenanceManager

# In-memory storage (default)

prov_mgr = ProvenanceManager()

# Persistent SQLite storage with custom path

prov_mgr = ProvenanceManager(storage_path="my_provenance.db")

```

For custom backends (e.g., PostgreSQL or cloud storage), subclass `ProvenanceStorage` from [`semantica/provenance/storage.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance/storage.py) and pass an instance to the constructor.

---

## Step 2: Track Entity Provenance

Call `track_entity()` to record how an entity was generated, by whom, and from what sources:

```python
prov_mgr.track_entity(
    entity_id="entity_42",
    source="doi:10.1000/xyz123",
    metadata={"label": "Apple Inc.", "type": "ORG"},
    confidence=0.96,
    activity_started_at_time="2024-01-15T09:30:00Z",
    activity_ended_at_time="2024-01-15T09:30:05Z",
    agent_id="nlp_extractor",
    agent_type="software_agent",
    role="generator"
)

```

**Key parameters:**
- `entity_id` – unique identifier for the tracked entity
- `source` – URI or identifier for the originating document
- `agent_id` / `agent_type` – who performed the activity (`software_agent`, `person`, or `organization`)
- `role` – PROV role in the qualified association (`generator`, `validator`, `deriver`, etc.)

The manager resolves these into proper `AgentRecord` and `ActivityRecord` objects internally via `_resolve_agent_kwargs()` and `_resolve_activity_kwargs()`.

---

## Step 3: Track Document Chunks

For **chunk-level granularity**, use `track_chunk()` to preserve text segmentation provenance:

```python
prov_mgr.track_chunk(
    chunk_id="chunk_001",
    source_document="doc_7",
    source_path="/data/doc_7.pdf",
    start_index=0,
    end_index=512,
    parent_chunk_id=None,  # Set to parent ID for hierarchical chunks

    agent_id="chunker",
    agent_type="software_agent",
    role="splitter"
)

```

Chunk tracking automatically:
- Links to `source_document` via `derived_from_id`
- Establishes parent-child relationships when `parent_chunk_id` is provided
- Enables **full document reconstruction** from lineage queries

---

## Step 4: Query W3C PROV-O Lineage

Retrieve complete ancestry with `get_lineage()`:

```python
lineage = prov_mgr.get_lineage("entity_42")

print("Source documents:", lineage["source_documents"])
print("Aggregated metadata:", lineage["metadata"])
print("Chain verified:", lineage["integrity_verified"])

```

The `trace_lineage()` method returns the raw chain of `ProvenanceEntry` objects for custom analysis, while `trace_descendants()` walks forward to find all derived entities.

**Integrity verification:** Every entry contains a `checksum` computed by `compute_checksum()` in [`semantica/provenance/integrity.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance/integrity.py), which incorporates the previous entry's hash for **tamper-evident audit trails**.

---

## Step 5: Export Standard RDF-PROV

Generate **W3C-standard PROV-O** output for external consumption:

```python

# Export as Turtle (default)

turtle = prov_mgr.export_prov(format="turtle")

# Alternative formats

jsonld = prov_mgr.export_prov(format="json-ld")
ntriples = prov_mgr.export_prov(format="nt")

# Persist to file

with open("provenance.ttl", "w") as f:
    f.write(turtle)

```

The exporter uses `rdflib` with the configurable `DEFAULT_BASE_URI` (`https://semantica.dev/ns#`) to generate consistent, dereferenceable URIs for all entities, agents, and activities.

---

## Step 6: Handle Entity Invalidation

Use `invalidate()` for **tombstone records** when sources are retracted or erroneous:

```python
prov_mgr.invalidate(
    entity_id="entity_42",
    agent_id="reviewer_jane",
    reason="Source document retracted",
    metadata={"retraction_note": "PDF corrupted"}
)

```

This:
- Archives the previous version
- Creates a new `ProvenanceEntry` with `invalidated=True`
- **Preserves the hash-chain** for complete audit history

---

## Batch Operations for Performance

For high-throughput scenarios, use `track_entities_batch()` and `track_chunks_batch()`:

```python
entities = [...]  # List of entity dictionaries

prov_mgr.track_entities_batch(entities, batch_size=1000)

```

These methods leverage SQLite savepoints from [`semantica/provenance/storage.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance/storage.py) with a **default batch size of 1,000**, maintaining atomicity while minimizing transaction overhead.

---

## HTTP API Integration

The FastAPI route in [`semantica/explorer/routes/provenance.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/explorer/routes/provenance.py) exposes these capabilities via REST:

```

GET  /provenance/{entity_id}/lineage     → JSON lineage response
GET  /provenance/{entity_id}/descendants → Forward tracing
POST /provenance/export                  → RDF download

```

Deploy this for **distributed provenance services** with the same underlying storage backends.

---

## Summary

- **`ProvenanceManager`** from [`semantica/provenance/manager.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance/manager.py) provides the unified API for W3C PROV-O provenance lineage tracking
- **`track_entity()`** and **`track_chunk()`** capture agent-attributed activities with full metadata
- **Lineage queries** via `get_lineage()` aggregate sources, verify checksums, and traverse derivation graphs
- **`export_prov()`** generates standard RDF-PROV in Turtle, JSON-LD, or N-Triples
- **Hash-chaining** in [`semantica/provenance/integrity.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance/integrity.py) ensures tamper-evident audit trails
- **Batch operations** and **invalidation** support production-scale deployment

---

## Frequently Asked Questions

### How does Semantica ensure PROV-O compliance?

Semantica implements the complete W3C PROV-O ontology by mapping all records to `prov:Entity`, `prov:Agent`, and `prov:Activity` classes with standard properties (`wasAttributedTo`, `qualifiedAssociation`, `actedOnBehalfOf`). The `export_prov()` method in [`manager.py`](https://github.com/semantica-agi/semantica/blob/main/manager.py) uses `rdflib` with the official PROV namespace to guarantee valid, interoperable RDF output.

### What storage backends are supported for provenance data?

The abstract `ProvenanceStorage` class in [`semantica/provenance/storage.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance/storage.py) enables pluggable backends. Semantica ships with `InMemoryStorage` for testing and `SQLiteStorage` for production. Custom implementations (PostgreSQL, S3, graph databases) subclass this interface and inject via the `ProvenanceManager` constructor.

### How is provenance integrity verified against tampering?

Every `ProvenanceEntry` contains a `checksum` computed by `compute_checksum()` in [`semantica/provenance/integrity.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance/integrity.py). This function hashes the entry content plus the previous entry's checksum, creating a cryptographic chain. `verify_checksum()` traverses this chain to detect any modifications; `get_lineage()` reports verification status via the `integrity_verified` flag.

### Can provenance be exported for use in external knowledge graphs?

Yes. The `export_prov()` method outputs standard PROV-O that imports directly into any RDF-compatible system. The configurable `DEFAULT_BASE_URI` ensures URI consistency with your knowledge graph namespace, enabling seamless integration with tools like Apache Jena, GraphDB, or custom SPARQL endpoints.