How to Implement W3C PROV-O Provenance Lineage Tracking in Python with Semantica

To implement W3C PROV-O provenance lineage tracking in Semantica, instantiate ProvenanceManager from semantica/provenance/manager.py, use track_entity() to record agent-attributed activities, then query lineage via get_lineage() and export RDF-PROV with export_prov().

The semantica-agi/semantica repository provides a production-grade, W3C PROV-O-compliant provenance subsystem that captures complete entity derivation history, cryptographic integrity, and standardized RDF export. This guide walks through the complete implementation using the actual source architecture.

---

Why W3C PROV-O Compliance Matters

W3C PROV-O is the W3C recommendation for representing provenance as linked data. Semantica's implementation maps every provenance record to standard PROV classes: prov:Entity, prov:Agent, and prov:Activity, with relationships like prov:wasAttributedTo, prov:qualifiedAssociation, and optional prov:actedOnBehalfOf delegation.

This ensures your provenance graphs are interoperable with external systems, queryable via SPARQL, and verifiable through cryptographic hash-chaining.

---

Core Architecture Components

Component	Role	Source Location
ProvenanceManager	Central API for tracking, lineage queries, and RDF export	semantica/provenance/manager.py
ProvenanceEntry	Immutable dataclass for provenance records	semantica/provenance/schemas.py
ProvenanceStorage	Abstract storage with InMemoryStorage and SQLiteStorage	semantica/provenance/storage.py
Integrity helpers	compute_checksum / verify_checksum for tamper-evidence	semantica/provenance/integrity.py
PROV-O Exporter	export_prov() method generating Turtle, N-Triples, or JSON-LD	semantica/provenance/manager.py

The lineage engine traverses a directed acyclic graph built from parent_entity_id, used_entities, and derived_from_id relationships in each entry.

---

Step 1: Initialize the ProvenanceManager

Create a manager with either in-memory or persistent SQLite storage:

from semantica.provenance.manager import ProvenanceManager

# In-memory storage (default)

prov_mgr = ProvenanceManager()

# Persistent SQLite storage with custom path

prov_mgr = ProvenanceManager(storage_path="my_provenance.db")

For custom backends (e.g., PostgreSQL or cloud storage), subclass ProvenanceStorage from semantica/provenance/storage.py and pass an instance to the constructor.

---

Step 2: Track Entity Provenance

Call track_entity() to record how an entity was generated, by whom, and from what sources:

prov_mgr.track_entity(
    entity_id="entity_42",
    source="doi:10.1000/xyz123",
    metadata={"label": "Apple Inc.", "type": "ORG"},
    confidence=0.96,
    activity_started_at_time="2024-01-15T09:30:00Z",
    activity_ended_at_time="2024-01-15T09:30:05Z",
    agent_id="nlp_extractor",
    agent_type="software_agent",
    role="generator"
)

Key parameters:

• entity_id – unique identifier for the tracked entity
• source – URI or identifier for the originating document
• agent_id / agent_type – who performed the activity (software_agent, person, or organization)
• role – PROV role in the qualified association (generator, validator, deriver, etc.)

The manager resolves these into proper AgentRecord and ActivityRecord objects internally via _resolve_agent_kwargs() and _resolve_activity_kwargs().

---

Step 3: Track Document Chunks

For chunk-level granularity, use track_chunk() to preserve text segmentation provenance:

prov_mgr.track_chunk(
    chunk_id="chunk_001",
    source_document="doc_7",
    source_path="/data/doc_7.pdf",
    start_index=0,
    end_index=512,
    parent_chunk_id=None,  # Set to parent ID for hierarchical chunks

    agent_id="chunker",
    agent_type="software_agent",
    role="splitter"
)

Chunk tracking automatically:

• Links to source_document via derived_from_id
• Establishes parent-child relationships when parent_chunk_id is provided
• Enables full document reconstruction from lineage queries

---

Step 4: Query W3C PROV-O Lineage

Retrieve complete ancestry with get_lineage():

lineage = prov_mgr.get_lineage("entity_42")

print("Source documents:", lineage["source_documents"])
print("Aggregated metadata:", lineage["metadata"])
print("Chain verified:", lineage["integrity_verified"])

The trace_lineage() method returns the raw chain of ProvenanceEntry objects for custom analysis, while trace_descendants() walks forward to find all derived entities.

Integrity verification: Every entry contains a checksum computed by compute_checksum() in semantica/provenance/integrity.py, which incorporates the previous entry's hash for tamper-evident audit trails.

---

Step 5: Export Standard RDF-PROV

Generate W3C-standard PROV-O output for external consumption:

# Export as Turtle (default)

turtle = prov_mgr.export_prov(format="turtle")

# Alternative formats

jsonld = prov_mgr.export_prov(format="json-ld")
ntriples = prov_mgr.export_prov(format="nt")

# Persist to file

with open("provenance.ttl", "w") as f:
    f.write(turtle)

The exporter uses rdflib with the configurable DEFAULT_BASE_URI (https://semantica.dev/ns#) to generate consistent, dereferenceable URIs for all entities, agents, and activities.

---

Step 6: Handle Entity Invalidation

Use invalidate() for tombstone records when sources are retracted or erroneous:

prov_mgr.invalidate(
    entity_id="entity_42",
    agent_id="reviewer_jane",
    reason="Source document retracted",
    metadata={"retraction_note": "PDF corrupted"}
)

This:

• Archives the previous version
• Creates a new ProvenanceEntry with invalidated=True
• Preserves the hash-chain for complete audit history

---

Batch Operations for Performance

For high-throughput scenarios, use track_entities_batch() and track_chunks_batch():

entities = [...]  # List of entity dictionaries

prov_mgr.track_entities_batch(entities, batch_size=1000)

These methods leverage SQLite savepoints from semantica/provenance/storage.py with a default batch size of 1,000, maintaining atomicity while minimizing transaction overhead.

---

HTTP API Integration

The FastAPI route in semantica/explorer/routes/provenance.py exposes these capabilities via REST:

GET  /provenance/{entity_id}/lineage     → JSON lineage response
GET  /provenance/{entity_id}/descendants → Forward tracing
POST /provenance/export                  → RDF download

Deploy this for distributed provenance services with the same underlying storage backends.

---

Summary

• ProvenanceManager from semantica/provenance/manager.py provides the unified API for W3C PROV-O provenance lineage tracking
• track_entity() and track_chunk() capture agent-attributed activities with full metadata
• Lineage queries via get_lineage() aggregate sources, verify checksums, and traverse derivation graphs
• export_prov() generates standard RDF-PROV in Turtle, JSON-LD, or N-Triples
• Hash-chaining in semantica/provenance/integrity.py ensures tamper-evident audit trails
• Batch operations and invalidation support production-scale deployment

---

Frequently Asked Questions

How does Semantica ensure PROV-O compliance?

Semantica implements the complete W3C PROV-O ontology by mapping all records to prov:Entity, prov:Agent, and prov:Activity classes with standard properties (wasAttributedTo, qualifiedAssociation, actedOnBehalfOf). The export_prov() method in manager.py uses rdflib with the official PROV namespace to guarantee valid, interoperable RDF output.

What storage backends are supported for provenance data?

The abstract ProvenanceStorage class in semantica/provenance/storage.py enables pluggable backends. Semantica ships with InMemoryStorage for testing and SQLiteStorage for production. Custom implementations (PostgreSQL, S3, graph databases) subclass this interface and inject via the ProvenanceManager constructor.

How is provenance integrity verified against tampering?

Every ProvenanceEntry contains a checksum computed by compute_checksum() in semantica/provenance/integrity.py. This function hashes the entry content plus the previous entry's checksum, creating a cryptographic chain. verify_checksum() traverses this chain to detect any modifications; get_lineage() reports verification status via the integrity_verified flag.

Can provenance be exported for use in external knowledge graphs?

Yes. The export_prov() method outputs standard PROV-O that imports directly into any RDF-compatible system. The configurable DEFAULT_BASE_URI ensures URI consistency with your knowledge graph namespace, enabling seamless integration with tools like Apache Jena, GraphDB, or custom SPARQL endpoints.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}