How Semantica Preserves Data Lineage from Databricks: A Technical Deep Dive

Semantica captures and preserves data lineage from Databricks through a three-layer architecture that extracts metadata via Unity Catalog APIs, stores it as versioned ProvenanceEntry records, and exposes it through deterministic traversal APIs.

Data lineage forms the backbone of trustworthy AI pipelines, enabling organizations to trace data from source to model. The open-source semantica-agi/semantica repository implements a platform-agnostic provenance system that specifically integrates with enterprise platforms like Databricks Unity Catalog to maintain complete audit trails. This article examines the exact mechanisms Semantica uses to ingest, persist, and query data lineage from Databricks environments.

The Three-Layer Architecture for Data Lineage Preservation

Semantica preserves data lineage through a tightly coupled stack that abstracts platform-specific metadata into a unified provenance model.

Ingestion Layer: Extracting Lineage from Unity Catalog

The DatabricksIngestor class in semantica/ingest/databricks_ingestor.py establishes authenticated connections to Databricks workspaces and interfaces directly with Unity Catalog's lineage APIs. The DatabricksIngestor.get_table_lineage method (lines 825-846) constructs fully-qualified table names and issues REST calls to GET /api/2.0/lineage-tracking/table-lineage, returning upstream tables, downstream consumers, and optional per-column lineage.

Provenance Layer: Storing and Traversing Lineage Records

Lineage metadata persists as ProvenanceEntry records via the ProvenanceManager in semantica/provenance/manager.py. The trace_lineage method performs breadth-first search (BFS) traversal across stored entries, while get_lineage (lines 752-815) aggregates these records into rich lineage chains complete with metadata merging, checksum validation, and temporal annotations.

Consumption Layer: Visualizing and Querying Lineage

Downstream components consume normalized lineage through semantica/visualization/visualization_provenance.py, which renders provenance data for vector-store indexing, graph visualizations, and audit logging. This layer ensures lineage data remains accessible regardless of the original enterprise platform.

Extracting Lineage from Databricks Unity Catalog

Semantica connects to Databricks using PAT or OAuth M2M authentication, lazily instantiating a Unity Catalog WorkspaceClient upon first metadata request. The ingestion process captures both table-level and column-level lineage through structured API calls.

from semantica.ingest import DatabricksIngestor

# Initialize the ingestor with Databricks workspace credentials

ingestor = DatabricksIngestor(
    host="https://adb-123.azuredatabricks.net",
    token="dapi-xxxxxx",
    http_path="/sql/1.0/warehouses/xxxxxx",
    catalog="main",
    schema="default",
)

# Retrieve comprehensive lineage including column-level details

lineage = ingestor.get_table_lineage(
    table_name="customers",
    include_column_lineage=True,
)

print(lineage["upstream"])      # → ['main.default.raw_customers']

print(lineage["downstream"])    # → ['main.default.customer_summary']

print(lineage["columns"]["id"]["upstream"])

# → ['main.default.raw_customers.customer_id']

Source: DatabricksIngestor.get_table_lineage

The method returns a dictionary containing three critical fields: upstream (source tables), downstream (consuming tables), and columns (per-column lineage mappings). This raw lineage feeds directly into Semantica's provenance storage layer.

Persisting and Traversing Lineage with ProvenanceManager

Once ingested, lineage data persists through ProvenanceStorage implementations (SQLite, PostgreSQL, etc.) as ProvenanceEntry records. These entries include entity_id, source_document, metadata, timestamps, and checksums for integrity verification.

from semantica.provenance.manager import ProvenanceManager

prov_mgr = ProvenanceManager()

# Query lineage for a previously ingested entity

lineage_info = prov_mgr.get_lineage("tbl_customers")

print(lineage_info["source_documents"])

# → ['databricks://main.default.customers']

print(lineage_info["metadata"])

# → {'owner': 'data_engineering', 'last_refresh': '2024-03-01'}

# Access full ancestry chain for visualization

for step in lineage_info["lineage_chain"]:
    print(step["entity_id"], step["activity_id"])

Source: ProvenanceManager.get_lineage

The get_lineage method aggregates ProvenanceEntry records into a comprehensive dictionary containing:

• lineage_chain: Ordered list of ancestor entities
• source_documents: Deduplicated source references
• first_seen / last_updated: Temporal boundaries
• metadata: Combined key-value pairs with precedence given to recent entries
• integrity_verified: Boolean checksum validation status

For scenarios requiring raw traversal control, trace_lineage(entity_id) returns the BFS-ordered list of ProvenanceEntry objects without aggregation.

End-to-End Data Lineage Flow

Semantica implements a deterministic pipeline for maintaining data lineage from ingestion to consumption:

1. Ingest: Connect to Databricks and extract table metadata via DatabricksIngestor
2. Request: Call get_table_lineage() to fetch upstream/downstream relationships from Unity Catalog
3. Persist: Store returned lineage as ProvenanceEntry records through the ingestion pipeline
4. Query: Retrieve aggregated lineage via ProvenanceManager.get_lineage() or traverse raw chains with trace_lineage()
5. Consume: Feed lineage data into vector stores, graph visualizers, or audit systems via the uniform provenance schema

Because the provenance model remains platform-agnostic, this same pattern extends to Snowflake, BigQuery, and other enterprise sources by implementing connector classes that map source metadata APIs to the ProvenanceEntry schema.

Summary

• Unity Catalog Integration: Semantica extracts data lineage via DatabricksIngestor.get_table_lineage calling Databricks REST APIs at semantica/ingest/databricks_ingestor.py.
• Persistent Storage: Lineage records store as ProvenanceEntry objects with full metadata, timestamps, and checksums through ProvenanceManager.
• Deterministic Traversal: trace_lineage provides BFS traversal while get_lineage (lines 752-815) aggregates rich lineage dictionaries with merged metadata.
• Cross-Platform Support: The provenance abstraction in semantica/provenance/manager.py enables data lineage preservation from any enterprise platform implementing the storage interface.
• Downstream Consumption: Visualization utilities in semantica/visualization/visualization_provenance.py render lineage for knowledge graphs and vector stores.

Frequently Asked Questions

How does Semantica connect to Databricks to extract lineage metadata?

Semantica establishes connections through the DatabricksConnector class using either Personal Access Tokens (PAT) or OAuth M2M authentication. The connector lazily instantiates a Unity Catalog WorkspaceClient and exposes SQL connectivity for metadata extraction. According to the source code in semantica/ingest/databricks_ingestor.py, the ingestor builds fully-qualified table names before calling the Unity Catalog lineage tracking endpoints.

What is the difference between trace_lineage and get_lineage in Semantica?

trace_lineage performs a raw breadth-first search across stored ProvenanceEntry records, returning an ordered list of entry objects representing the full upstream ancestry. In contrast, get_lineage (implemented at lines 752-815 in provenance/manager.py) aggregates these entries into a structured dictionary that merges metadata, deduplicates source documents, and includes integrity verification status. Use trace_lineage for granular traversal control and get_lineage for comprehensive lineage reports.

Can Semantica track column-level lineage from Databricks?

Yes. When calling DatabricksIngestor.get_table_lineage(), setting include_column_lineage=True triggers additional REST calls to Unity Catalog's column-lineage endpoints. The method returns a nested columns dictionary mapping each column to its upstream sources, enabling fine-grained impact analysis across table schemas.

Is Semantica's data lineage model limited to Databricks only?

No. While DatabricksIngestor handles Unity Catalog specifically, the underlying ProvenanceEntry schema and ProvenanceManager APIs remain platform-agnostic. The architecture supports Snowflake, BigQuery, and other enterprise platforms by implementing ingestion connectors that map source-specific metadata APIs to the uniform provenance storage interface defined in semantica/provenance/manager.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}