Setting Up W3C PROV-O Provenance Tracking in Semantica: A Complete Implementation Guide

Semantica provides a production-ready, W3C PROV-O-compliant provenance engine with pluggable storage backends, cryptographic integrity verification, and native RDF export capabilities.

Setting up W3C PROV-O provenance tracking in Semantica enables full auditability for AI-generated knowledge graphs by capturing entities, activities, and agents according to standardized semantic web specifications. The implementation resides in the semantica-agi/semantica repository and requires no external provenance servers—everything runs embedded within your Python application.

Understanding the Three-Layer Architecture

Semantica's provenance system is organized into distinct layers that separate concerns between data modeling, persistence, and high-level orchestration.

Data Model Layer

The foundation resides in semantica/provenance/schemas.py, where the ProvenanceEntry dataclass implements the W3C PROV-O specification. This schema maps directly to core provenance concepts:

• Entities: entity_id, entity_type
• Activities: activity_id, activity_started_at_time, activity_ended_at_time
• Agents: agent_id, agent_type, is_automated
• Relationships: parent_entity_id, derived_from_id, used_entities, previous_version_id

The module also provides lightweight wrappers including AgentRecord, ActivityRecord, SourceReference, and PropertySource, all featuring to_dict/from_dict serialization helpers for interoperability.

Storage Layer

The semantica/provenance/storage.py module defines an abstract ProvenanceStorage interface with two concrete implementations:

InMemoryStorage (lines 91-150) maintains entries in a Python dictionary with hash-chain integrity checking and BFS lineage tracing. This backend suits unit testing and development workflows where persistence is unnecessary.

SQLiteStorage (lines 91-1040) provides production-grade persistence with a fully W3C PROV-O-compliant database schema, indexed lookups for fast lineage queries, and transactional safety through SQLite's ACID guarantees.

Both implementations expose identical APIs (store, retrieve, retrieve_all, trace_lineage, clear), enabling seamless backend swaps without code changes.

Manager Layer

semantica/provenance/manager.py contains the ProvenanceManager façade that coordinates tracking, querying, and export operations. Key responsibilities include:

• Entity tracking: track_entity, track_relationship, track_chunk
• Batch operations: track_entities_batch, track_chunks_batch (default batch size 1,000)
• Lineage reconstruction: get_lineage with hash-chain validation
• Invalidation: invalidate method creating tombstone entries (lines 110-149)
• RDF export: export_prov supporting Turtle, N-Triples, and JSON-LD formats (lines 302-327)

Configuring the Storage Backend

Select your storage implementation based on environmental requirements. The constructor interfaces differ slightly between volatile and persistent modes.

For development and testing, instantiate InMemoryStorage:

from semantica.provenance.storage import InMemoryStorage

storage = InMemoryStorage()

For production deployments requiring durability, use SQLiteStorage with a file path:

from semantica.provenance.storage import SQLiteStorage

storage = SQLiteStorage("provenance.db")

Both backends support the transaction context manager (with storage.transaction():), ensuring atomic operations even during batch inserts.

Initializing the Provenance Manager

Create the ProvenanceManager instance by passing your configured storage backend. The manager optionally accepts a custom base URI for RDF generation (default: https://semantica.dev/ns#).

from semantica.provenance.manager import ProvenanceManager

prov_mgr = ProvenanceManager(storage=storage)

The manager internally normalizes agent and activity parameters through _resolve_agent_kwargs and _resolve_activity_kwargs (lines 24-38), allowing flexible input types ranging from dataclass instances to raw dictionaries.

Tracking Entities with PROV-O Metadata

The track_entity method captures provenance records with full W3C PROV-O semantics. You must provide source identification and may optionally specify agents, activities, and derivation relationships.

from semantica.provenance.schemas import AgentRecord, ActivityRecord

# Define the responsible agent

agent = AgentRecord(
    id="reviewer_jane",
    agent_type="person",
    is_automated=False
)

# Define the processing activity

activity = ActivityRecord(
    id="ner_extraction",
    activity_type="nlp_process",
    started_at_time="2026-08-18T12:00:00",
    ended_at_time="2026-08-18T12:00:02",
)

# Record the entity provenance

entry = prov_mgr.track_entity(
    entity_id="entity_123",
    source="DOI:10.1371/journal.pone.0023601",
    metadata={"confidence": 0.94},
    agent=agent,
    activity=activity,
    role="generator",
)

For high-throughput scenarios, use track_entities_batch which implements chunked processing with configurable batch sizes (default 1,000) and per-item savepoints for isolated failure rollback:

entities = [
    {"id": f"entity_{i}", "metadata": {"confidence": 0.9 - i * 0.01}}
    for i in range(2500)
]

tracked_count = prov_mgr.track_entities_batch(
    entities, 
    source="dataset_v1",
    batch_size=1000
)

Querying Lineage and Descendants

Retrieve complete ancestry chains using get_lineage, which aggregates metadata from all upstream entities and validates hash-chain integrity:

lineage = prov_mgr.get_lineage("entity_123")

print(lineage["source_documents"])  # List of source IDs

print(lineage["metadata"])          # Merged metadata from ancestors

print(lineage["integrity_verified"])  # Boolean indicating checksum validity

For downstream impact analysis, get_descendants traverses the reverse graph to identify all entities derived from a specific source.

Exporting W3C PROV-O RDF

Semantica generates standards-compliant RDF serializations through the export_prov method. The export binds the Provenance namespace (prov) and a custom namespace (ex), producing interoperable graphs for external provenance tools.

# Generate Turtle format

turtle_data = prov_mgr.export_prov(format="turtle")

# Generate JSON-LD

jsonld_data = prov_mgr.export_prov(format="jsonld")

with open("provenance.ttl", "w") as f:
    f.write(turtle_data)

Supported formats include turtle, ntriples, and jsonld. The base URI parameter ensures exported identifiers align with your knowledge graph namespace.

Advanced Configuration Topics

Hash-Chain Integrity Verification

Every ProvenanceEntry receives a SHA-256 checksum computed via compute_checksum, linking entries through the previous_checksum field. The manager automatically verifies this chain during lineage queries, setting integrity_verified to False if tampering is detected.

Versioning vs. Derivation Semantics

Semantica distinguishes between editorial corrections and semantic derivations:

• previous_version_id: Records a correction of the same factual content (versioning)
• derived_from_id: Records transformation from a distinct source (derivation)

Access complete revision histories through the revision_history API.

Entity Invalidation (Tombstones)

The invalidate method (lines 110-149) archives the pre-invalidation entry, creates a tombstone record marked with invalidated=True, and preserves the hash chain integrity for audit trails.

Custom RDF Base URIs

When integrating with Semantica's knowledge graph exporter, ensure the ProvenanceManager base URI matches your KG namespace to maintain URI consistency across exported assets.

Summary

• Semantica implements native W3C PROV-O compliance through three layered modules: schemas.py (data model), storage.py (persistence), and manager.py (API).
• Two storage backends provide flexibility: InMemoryStorage for testing and SQLiteStorage for production workloads with ACID guarantees.
• The ProvenanceManager façade handles entity tracking, batch processing, lineage queries, and RDF export without requiring direct storage manipulation.
• Cryptographic integrity is maintained through SHA-256 hash chains that are validated during lineage reconstruction.
• RDF export capabilities support Turtle, N-Triples, and JSON-LD formats with configurable base URIs for standards-compliant interoperability.

Frequently Asked Questions

What is the difference between previous_version_id and derived_from_id in Semantica's provenance model?

previous_version_id indicates an editorial correction or update to the same entity (versioning), while derived_from_id indicates that the entity was algorithmically or manually derived from a distinct source entity. Both fields are tracked in semantica/provenance/schemas.py and exposed through the revision_history API for complete audit trails.

How does Semantica ensure provenance data integrity during concurrent writes?

The SQLiteStorage implementation in semantica/provenance/storage.py uses SQLite's native transaction support through the transaction context manager. All manager operations including track_entities_batch wrap database interactions in savepoints, ensuring atomicity and preventing partial writes during concurrent access or batch processing failures.

Can I export provenance data for use in external PROV-O tools like ProvStore or ProvToolbox?

Yes. The ProvenanceManager.export_prov() method generates fully W3C PROV-O-compliant RDF in Turtle, N-Triples, or JSON-LD formats. These serializations utilize standard prov: namespace bindings and can be directly imported into ProvStore, ProvToolbox, or any other standards-compliant provenance analysis platform.

What is the recommended batch size for high-volume provenance tracking?

The default batch size of 1,000 entries in track_entities_batch derives from empirical benchmarks balancing SQLite write performance against memory constraints. For datasets exceeding 10,000 entries, maintain this default; for constrained environments, reduce to 500. The method implements per-item savepoints, ensuring that failures within a batch do not invalidate successfully processed chunks.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}