# Setting Up W3C PROV-O Provenance Tracking in Semantica: A Complete Implementation Guide

> Learn to set up W3C PROV-O provenance tracking in Semantica. This guide covers implementation for a robust provenance engine with pluggable storage and RDF export.

- Repository: [Semantica /semantica](https://github.com/semantica-agi/semantica)
- Tags: how-to-guide
- Published: 2026-08-18

---

**Semantica provides a production-ready, W3C PROV-O-compliant provenance engine with pluggable storage backends, cryptographic integrity verification, and native RDF export capabilities.**

Setting up W3C PROV-O provenance tracking in Semantica enables full auditability for AI-generated knowledge graphs by capturing entities, activities, and agents according to standardized semantic web specifications. The implementation resides in the `semantica-agi/semantica` repository and requires no external provenance servers—everything runs embedded within your Python application.

## Understanding the Three-Layer Architecture

Semantica's provenance system is organized into distinct layers that separate concerns between data modeling, persistence, and high-level orchestration.

### Data Model Layer

The foundation resides in [`semantica/provenance/schemas.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance/schemas.py), where the `ProvenanceEntry` dataclass implements the W3C PROV-O specification. This schema maps directly to core provenance concepts:

- **Entities**: `entity_id`, `entity_type`
- **Activities**: `activity_id`, `activity_started_at_time`, `activity_ended_at_time`
- **Agents**: `agent_id`, `agent_type`, `is_automated`
- **Relationships**: `parent_entity_id`, `derived_from_id`, `used_entities`, `previous_version_id`

The module also provides lightweight wrappers including `AgentRecord`, `ActivityRecord`, `SourceReference`, and `PropertySource`, all featuring `to_dict`/`from_dict` serialization helpers for interoperability.

### Storage Layer

The [`semantica/provenance/storage.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance/storage.py) module defines an abstract `ProvenanceStorage` interface with two concrete implementations:

**InMemoryStorage** (lines 91-150) maintains entries in a Python dictionary with hash-chain integrity checking and BFS lineage tracing. This backend suits unit testing and development workflows where persistence is unnecessary.

**SQLiteStorage** (lines 91-1040) provides production-grade persistence with a fully W3C PROV-O-compliant database schema, indexed lookups for fast lineage queries, and transactional safety through SQLite's ACID guarantees.

Both implementations expose identical APIs (`store`, `retrieve`, `retrieve_all`, `trace_lineage`, `clear`), enabling seamless backend swaps without code changes.

### Manager Layer

[`semantica/provenance/manager.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance/manager.py) contains the `ProvenanceManager` façade that coordinates tracking, querying, and export operations. Key responsibilities include:

- **Entity tracking**: `track_entity`, `track_relationship`, `track_chunk`
- **Batch operations**: `track_entities_batch`, `track_chunks_batch` (default batch size 1,000)
- **Lineage reconstruction**: `get_lineage` with hash-chain validation
- **Invalidation**: `invalidate` method creating tombstone entries (lines 110-149)
- **RDF export**: `export_prov` supporting Turtle, N-Triples, and JSON-LD formats (lines 302-327)

## Configuring the Storage Backend

Select your storage implementation based on environmental requirements. The constructor interfaces differ slightly between volatile and persistent modes.

For development and testing, instantiate `InMemoryStorage`:

```python
from semantica.provenance.storage import InMemoryStorage

storage = InMemoryStorage()

```

For production deployments requiring durability, use `SQLiteStorage` with a file path:

```python
from semantica.provenance.storage import SQLiteStorage

storage = SQLiteStorage("provenance.db")

```

Both backends support the transaction context manager (`with storage.transaction():`), ensuring atomic operations even during batch inserts.

## Initializing the Provenance Manager

Create the `ProvenanceManager` instance by passing your configured storage backend. The manager optionally accepts a custom base URI for RDF generation (default: `https://semantica.dev/ns#`).

```python
from semantica.provenance.manager import ProvenanceManager

prov_mgr = ProvenanceManager(storage=storage)

```

The manager internally normalizes agent and activity parameters through `_resolve_agent_kwargs` and `_resolve_activity_kwargs` (lines 24-38), allowing flexible input types ranging from dataclass instances to raw dictionaries.

## Tracking Entities with PROV-O Metadata

The `track_entity` method captures provenance records with full W3C PROV-O semantics. You must provide source identification and may optionally specify agents, activities, and derivation relationships.

```python
from semantica.provenance.schemas import AgentRecord, ActivityRecord

# Define the responsible agent

agent = AgentRecord(
    id="reviewer_jane",
    agent_type="person",
    is_automated=False
)

# Define the processing activity

activity = ActivityRecord(
    id="ner_extraction",
    activity_type="nlp_process",
    started_at_time="2026-08-18T12:00:00",
    ended_at_time="2026-08-18T12:00:02",
)

# Record the entity provenance

entry = prov_mgr.track_entity(
    entity_id="entity_123",
    source="DOI:10.1371/journal.pone.0023601",
    metadata={"confidence": 0.94},
    agent=agent,
    activity=activity,
    role="generator",
)

```

For high-throughput scenarios, use `track_entities_batch` which implements chunked processing with configurable batch sizes (default 1,000) and per-item savepoints for isolated failure rollback:

```python
entities = [
    {"id": f"entity_{i}", "metadata": {"confidence": 0.9 - i * 0.01}}
    for i in range(2500)
]

tracked_count = prov_mgr.track_entities_batch(
    entities, 
    source="dataset_v1",
    batch_size=1000
)

```

## Querying Lineage and Descendants

Retrieve complete ancestry chains using `get_lineage`, which aggregates metadata from all upstream entities and validates hash-chain integrity:

```python
lineage = prov_mgr.get_lineage("entity_123")

print(lineage["source_documents"])  # List of source IDs

print(lineage["metadata"])          # Merged metadata from ancestors

print(lineage["integrity_verified"])  # Boolean indicating checksum validity

```

For downstream impact analysis, `get_descendants` traverses the reverse graph to identify all entities derived from a specific source.

## Exporting W3C PROV-O RDF

Semantica generates standards-compliant RDF serializations through the `export_prov` method. The export binds the Provenance namespace (`prov`) and a custom namespace (`ex`), producing interoperable graphs for external provenance tools.

```python

# Generate Turtle format

turtle_data = prov_mgr.export_prov(format="turtle")

# Generate JSON-LD

jsonld_data = prov_mgr.export_prov(format="jsonld")

with open("provenance.ttl", "w") as f:
    f.write(turtle_data)

```

Supported formats include `turtle`, `ntriples`, and `jsonld`. The base URI parameter ensures exported identifiers align with your knowledge graph namespace.

## Advanced Configuration Topics

### Hash-Chain Integrity Verification

Every `ProvenanceEntry` receives a SHA-256 checksum computed via `compute_checksum`, linking entries through the `previous_checksum` field. The manager automatically verifies this chain during lineage queries, setting `integrity_verified` to `False` if tampering is detected.

### Versioning vs. Derivation Semantics

Semantica distinguishes between editorial corrections and semantic derivations:

- **`previous_version_id`**: Records a correction of the same factual content (versioning)
- **`derived_from_id`**: Records transformation from a distinct source (derivation)

Access complete revision histories through the `revision_history` API.

### Entity Invalidation (Tombstones)

The `invalidate` method (lines 110-149) archives the pre-invalidation entry, creates a tombstone record marked with `invalidated=True`, and preserves the hash chain integrity for audit trails.

### Custom RDF Base URIs

When integrating with Semantica's knowledge graph exporter, ensure the `ProvenanceManager` base URI matches your KG namespace to maintain URI consistency across exported assets.

## Summary

- **Semantica implements native W3C PROV-O compliance** through three layered modules: [`schemas.py`](https://github.com/semantica-agi/semantica/blob/main/schemas.py) (data model), [`storage.py`](https://github.com/semantica-agi/semantica/blob/main/storage.py) (persistence), and [`manager.py`](https://github.com/semantica-agi/semantica/blob/main/manager.py) (API).
- **Two storage backends** provide flexibility: `InMemoryStorage` for testing and `SQLiteStorage` for production workloads with ACID guarantees.
- **The `ProvenanceManager` façade** handles entity tracking, batch processing, lineage queries, and RDF export without requiring direct storage manipulation.
- **Cryptographic integrity** is maintained through SHA-256 hash chains that are validated during lineage reconstruction.
- **RDF export capabilities** support Turtle, N-Triples, and JSON-LD formats with configurable base URIs for standards-compliant interoperability.

## Frequently Asked Questions

### What is the difference between `previous_version_id` and `derived_from_id` in Semantica's provenance model?

`previous_version_id` indicates an editorial correction or update to the same entity (versioning), while `derived_from_id` indicates that the entity was algorithmically or manually derived from a distinct source entity. Both fields are tracked in [`semantica/provenance/schemas.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance/schemas.py) and exposed through the `revision_history` API for complete audit trails.

### How does Semantica ensure provenance data integrity during concurrent writes?

The `SQLiteStorage` implementation in [`semantica/provenance/storage.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance/storage.py) uses SQLite's native transaction support through the `transaction` context manager. All manager operations including `track_entities_batch` wrap database interactions in savepoints, ensuring atomicity and preventing partial writes during concurrent access or batch processing failures.

### Can I export provenance data for use in external PROV-O tools like ProvStore or ProvToolbox?

Yes. The `ProvenanceManager.export_prov()` method generates fully W3C PROV-O-compliant RDF in Turtle, N-Triples, or JSON-LD formats. These serializations utilize standard `prov:` namespace bindings and can be directly imported into ProvStore, ProvToolbox, or any other standards-compliant provenance analysis platform.

### What is the recommended batch size for high-volume provenance tracking?

The default batch size of 1,000 entries in `track_entities_batch` derives from empirical benchmarks balancing SQLite write performance against memory constraints. For datasets exceeding 10,000 entries, maintain this default; for constrained environments, reduce to 500. The method implements per-item savepoints, ensuring that failures within a batch do not invalidate successfully processed chunks.