How Semantica's ProvenanceManager Tracks W3C PROV-O Lineage for Audit Compliance

Semantica's ProvenanceManager implements a standards-compliant tracking system that records W3C PROV-O lineage by mapping every knowledge-graph operation to Entity, Activity, and Agent concepts, persisting them in SQLite or memory, and exposing audit-ready retrieval methods.

The semantica-agi/semantica repository provides a robust provenance layer that ensures every piece of data in your knowledge graph maintains a complete audit trail. By implementing the W3C PROV-O ontology, the ProvenanceManager enables regulatory compliance and full traceability from raw sources to derived insights.

The Three-Layer Architecture

The provenance system operates through three tightly coupled layers that translate user actions into standardized PROV-O triples.

Data Model Layer

The foundational layer defines PROV-O concepts using a JSON-serializable schema. In semantica/provenance/schemas.py, the data classes SourceReference and ProvenanceEntry mirror the PROV-O RDF model. These classes capture Entity, Activity, and Agent relationships while maintaining compatibility with external semantic tools.

Storage Abstraction Layer

The persistence layer in semantica/provenance/storage.py implements two backends: InMemoryStorage for fast testing and SQLiteStorage for production environments. The manager transparently swaps backends based on the storage_path argument, storing provenance records in a compact prov_entries table that supports efficient audit queries.

Manager API Layer

The ProvenanceManager class in semantica/provenance/manager.py provides high-level methods including add_entity(), add_activity(), add_agent(), link(), and get_lineage(). These methods validate entries against the schema, translate user-level actions into PROV-O triples, and reconstruct full ancestry graphs on demand.

How Lineage Is Built

The ProvenanceManager constructs audit-ready lineage through a five-step process that maps every graph operation to standardized provenance relationships.

1. Creating Provenance Entries

When ingesting a new graph node—whether a document chunk, extracted entity, or derived relationship—the manager creates a ProvenanceEntry with a unique identifier, timestamp, and SourceReference objects. These references capture original URLs, PubMed IDs, or confidence scores, establishing the initial Entity record.

2. Linking Activities

Transformations such as semantic extraction steps are recorded as Activity objects. The manager automatically inserts wasGeneratedBy and wasDerivedFrom relationships between affected Entity objects and the executing Activity, creating a complete transformation chain.

3. Assigning Agents

Every executing component—whether an LLM, extraction pipeline, or human user—is captured as an Agent. The manager adds wasAttributedTo edges, ensuring every piece of data can be traced back to a responsible party for accountability.

4. Persisting the Graph

All triples are stored in the selected backend. In production, SQLiteStorage persists them in the prov_entries table with indexed relationships. The storage layer handles serialization while maintaining the semantic integrity required for PROV-O compliance.

5. Retrieving Lineage

The get_lineage(entity_id) method in semantica/provenance/manager.py walks stored triples and reconstructs the full ancestry DAG using PROV-O relationships. Results can be exported as RDF, JSON-LD, or visualized directly in the UI through the provenanceCount field consumed by explorer/src/workspaces/GraphWorkspace/useLoadGraph.ts.

Implementing Audit-Ready Provenance in Code

The following patterns demonstrate how to record and query provenance within a Semantica pipeline.

Initializing the Manager

Initialize the manager with default in-memory storage for testing or specify a SQLite path for production audit trails.

from semantica.provenance import ProvenanceManager

# In-memory for fast testing

prov_mgr = ProvenanceManager()

# SQLite for persistent audit logs

prov_mgr = ProvenanceManager(storage_path="/path/to/audit.db")

Recording Entities and Activities

Track the ingestion of raw documents and link extracted entities to their transformation activities.

# Track ingestion of a raw document

doc_id = prov_mgr.add_entity(
    entity_type="Document",
    identifier="doc-123",
    attributes={"title": "Annual Report 2024"},
    source_refs=[{"source": "https://example.com/report.pdf"}],
)

# Record an extraction activity

activity_id = prov_mgr.add_activity(
    activity_type="Extraction",
    label="NER on doc-123",
    used=[doc_id],
    generated=[],
)

# Create extracted entity and link to activity

entity_id = prov_mgr.add_entity(
    entity_type="Entity",
    identifier="entity-456",
    attributes={"text": "Acme Corp"},
    source_refs=[],
)

# Establish provenance relationships

prov_mgr.link(
    generated=entity_id,
    activity=activity_id,
    used=[doc_id],
    agents=["ner-service"]
)

Retrieving Lineage

Retrieve complete ancestry for any entity to satisfy audit requirements.

# Get full PROV-O lineage as JSON-LD

lineage = prov_mgr.get_lineage(entity_id)
print(lineage)

# Output includes @type, @id, wasGeneratedBy, wasDerivedFrom, wasAttributedTo

Exporting for External Audits

Export provenance data to standard RDF formats for compatibility with external audit tools like ProvToolbox.

# Export to Turtle format for external tools

rdf_graph = prov_mgr.export_rdf(format="ttl")
with open("audit_provenance.ttl", "w") as f:
    f.write(rdf_graph)

UI Integration and Visualization

The provenance layer integrates with the Semantica Explorer UI through the provenanceCount field. The GraphWorkspace component in explorer/src/workspaces/GraphWorkspace/graphTheme.ts renders visual badges based on this count, allowing users to instantly identify nodes with complete audit trails.

When loading graph data, useLoadGraph.ts extracts provenanceCount from node attributes and drives badge rendering, providing immediate visual feedback on data lineage quality without requiring full lineage retrieval.

Summary

• Standards Compliance: The ProvenanceManager in semantica/provenance/manager.py implements the complete W3C PROV-O ontology through Entity, Activity, and Agent abstractions.
• Flexible Storage: SQLiteStorage and InMemoryStorage in semantica/provenance/storage.py provide production-grade persistence and testing capabilities.
• Complete Traceability: Methods like add_entity(), link(), and get_lineage() ensure every transformation is recorded with wasGeneratedBy, wasDerivedFrom, and wasAttributedTo relationships.
• Audit Export: The export_rdf() method supports Turtle and other RDF formats for external compliance verification.
• Visual Integration: The UI layer consumes provenanceCount to provide immediate visual indicators of audit readiness.

Frequently Asked Questions

What is W3C PROV-O and why does Semantica use it?

W3C PROV-O is the World Wide Web Consortium's standard ontology for provenance, defining core concepts like Entity, Activity, and Agent along with their relationships. Semantica uses this standard to ensure interoperability with external audit tools and regulatory frameworks, allowing provenance data to be exchanged with systems like ProvToolbox without semantic loss.

How does the ProvenanceManager handle storage backend switching?

The manager accepts a storage_path parameter during initialization. When a path is provided, it instantiates SQLiteStorage from semantica/provenance/storage.py; otherwise, it defaults to InMemoryStorage. This abstraction allows identical API usage across testing and production environments while ensuring audit logs persist to disk when compliance requires durable storage.

Can provenance data be exported for external audit tools?

Yes. The export_rdf() method in semantica/provenance/manager.py serializes the provenance graph to standard RDF formats including Turtle (ttl). This exported data maintains full PROV-O semantics, enabling direct import into external audit platforms, semantic validators, or regulatory submission systems.

How does the UI visualize provenance lineage?

The GraphWorkspace component reads the provenanceCount field from node attributes via useLoadGraph.ts. Nodes with provenance records receive distinct visual badges defined in graphTheme.ts, providing immediate visual indication of audit trail availability. For detailed inspection, the full lineage can be retrieved through get_lineage() and displayed as structured PROV-O relationships.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}