How to Export Audit Trails in W3C PROV-O Format Using Semantica

Semantica exports complete audit trails as W3C PROV-O RDF using the ProvenanceManager.export_prov() method or the semantica provenance export CLI command, supporting Turtle, N-Triples, and JSON-LD serialization.

Semantica automatically records provenance for every entity, activity, and agent involved in pipeline execution, storing this data in a centralized ProvenanceManager. When you need to share or archive these records, you can export audit trails in W3C PROV-O format to generate standards-compliant RDF graphs that interoperate with any provenance-aware tool.

Architecture of the PROV-O Export System

The export pipeline relies on a unified storage and serialization layer defined in semantica/provenance/manager.py. Understanding this flow helps you optimize large exports and troubleshoot validation errors.

Provenance Storage Layer

Audit entries are persisted via ProvenanceStorage implementations—either SQLiteStorage for durability or InMemoryStorage for transient workflows. Each entry contains entity_id, activity_id, agent_id, timestamps, and optional metadata such as role and bundle identifiers. The storage interface is defined in semantica/provenance/storage.py, and the manager retrieves all entries via self.storage.retrieve_all() before serialization.

RDF Graph Construction

The export_prov() method (lines 1260–1398 in semantica/provenance/manager.py) constructs an rdflib.Graph and binds it to the W3C PROV namespace (http://www.w3.org/ns/prov#). For every provenance record, the code emits:

• Entity triples (prov:Entity) with optional prov:generatedAtTime timestamps
• Agent triples (prov:Agent) refined as Person, SoftwareAgent, or Organization subclasses
• Activity triples (prov:Activity) with start and end times
• Relationship triples including qualified association, generation, usage, derivation, invalidation, and bundle membership

You can namespace all minted URIs under a custom base URI (default: https://semantica.dev/ns#) by passing the base_uri parameter to export_prov() (lines 1238–1249).

Serialization and Formats

Once the graph is populated, the manager selects the requested RDF format—turtle, ntriples, or jsonld—and returns the serialized string (lines 1399–1400). The CLI command semantica provenance export (defined in semantica/cli.py, lines 2902–2936) forwards its --format, --base-uri, and --output options directly to this method.

Exporting Via the Command Line

The semantica provenance export command provides the fastest path to generate PROV-O files without writing code.

Basic Export to Turtle

semantica provenance export --output prov.ttl

This reads the current provenance database (respecting the ProvenanceManager configuration in your environment) and writes a Turtle file containing the complete audit trail.

Alternative Formats and Custom Namespaces

# Export as JSON-LD for web integration

semantica provenance export --format jsonld --output prov.jsonld

# Use a custom base URI for your organization

semantica provenance export \
  --base-uri https://example.org/mykg# \

  --format turtle \
  --output my_provenance.ttl

If you omit --output, the RDF streams to stdout for piping into other tools.

Pre-Export Validation

Before exporting, verify the integrity of your audit trail to ensure no tampering has occurred:

semantica provenance check
semantica provenance verify-chain

These commands (lines 2842–2865 in semantica/cli.py) validate the hash chain stored in semantica/provenance/storage.py and exit with an error if the provenance record has been altered.

Exporting Programmatically

For pipeline automation or web service integration, call ProvenanceManager.export_prov() directly from Python.

Basic Python Export

from semantica.provenance import ProvenanceManager

# Initialize with persistent SQLite storage

pm = ProvenanceManager(config={"provenance": {"storage_path": "prov.db"}})

# Export as Turtle (default)

turtle_rdf = pm.export_prov(format="turtle")

with open("audit_prov.ttl", "w", encoding="utf-8") as f:
    f.write(turtle_rdf)

Custom Base URI and Formats

# Export as JSON-LD with custom namespace

jsonld_rdf = pm.export_prov(
    format="jsonld", 
    base_uri="https://example.org/ns#"
)

The method returns a Python string containing the complete RDF serialization, allowing you to stream the result to object storage, databases, or HTTP responses without intermediate files.

Advanced Graph Inspection

You can parse the exported string back into an rdflib.Graph for post-processing or validation:

from rdflib import Graph

pm = ProvenanceManager()
rdf_str = pm.export_prov(format="jsonld")

g = Graph()
g.parse(data=rdf_str, format="json-ld")

# Query for all SoftwareAgents

for s, p, o in g.triples((None, None, None)):
    if "SoftwareAgent" in str(o):
        print(f"Agent: {s}")

Data Schema and Key Files

The PROV-O export relies on strict data classes defined in semantica/provenance/schemas.py. These include ProvenanceEntry, AgentRecord, and ActivityRecord, which map directly to W3C PROV-O classes. When the manager builds the RDF graph, it uses these schemas to ensure type safety and compliance with the provenance ontology.

Key source files:

• semantica/provenance/manager.py – Central manager; implements export_prov() and RDF graph construction (lines 1238–1400)
• semantica/cli.py – Click command provenance export that exposes functionality to end users (lines 2902–2936)
• semantica/provenance/storage.py – Storage backends that hold audit entries used by the exporter
• semantica/provenance/schemas.py – W3C PROV-O-compatible data classes

Summary

• Semantica captures exhaustive provenance for every pipeline run and stores it in ProvenanceManager, backed by SQLite or in-memory storage.
• Use semantica provenance export (CLI) or ProvenanceManager.export_prov() (Python) to generate W3C PROV-O RDF.
• Choose from Turtle, N-Triples, or JSON-LD output formats via the --format or format parameter.
• Customize the namespace by providing a base_uri to ensure URIs match your organizational standards.
• Validate audit integrity before export using semantica provenance check to verify hash chains.
• The export implementation resides in semantica/provenance/manager.py and uses rdflib to construct standards-compliant graphs.

Frequently Asked Questions

What is W3C PROV-O and why does Semantica use it?

W3C PROV-O is the OWL2 ontology representation of the PROV data model, providing a standardized vocabulary for describing provenance. Semantica uses it to ensure your audit trails are interoperable with external tools like GraphDB, Apache Jena, and ProvToolbox. According to the semantica-agi/semantica source code, the export_prov() method explicitly binds the prov prefix to http://www.w3.org/ns/prov# and maps internal records to PROV-O classes such as Entity, Activity, and Agent.

Can I export provenance from a running pipeline without stopping execution?

Yes. The ProvenanceManager supports concurrent access to its storage backends. You can instantiate a separate manager instance in a background thread or external process, call export_prov(), and stream the result while the main pipeline continues. The SQLite storage implementation in semantica/provenance/storage.py uses proper transaction isolation to ensure read consistency during export.

How do I change the default namespace for all exported URIs?

Pass the base_uri parameter to export_prov() in Python or use the --base-uri flag in the CLI. For example, --base-uri https://myorg.com/data# replaces the default https://semantica.dev/ns# prefix. This affects every Entity, Agent, and Activity URI minted during export, making the provenance graph resolvable against your own vocabulary infrastructure.

Which RDF formats are supported for PROV-O export?

Semantica supports Turtle (default), N-Triples, and JSON-LD. You specify the format via the format parameter in Python ("turtle", "ntriples", or "jsonld") or the --format option in the CLI. The serialization logic at lines 1399–1400 of semantica/provenance/manager.py delegates to rdflib's native serializers, ensuring full compatibility with RDF 1.1 specifications.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}