# How Semantica Preserves Data Lineage from Databricks: A Technical Deep Dive

> Discover how Semantica preserves data lineage from Databricks using a three-layer architecture. Learn about metadata extraction, versioned records, and API exposure.

- Repository: [Semantica /semantica](https://github.com/semantica-agi/semantica)
- Tags: deep-dive
- Published: 2026-09-08

---

**Semantica captures and preserves data lineage from Databricks through a three-layer architecture that extracts metadata via Unity Catalog APIs, stores it as versioned ProvenanceEntry records, and exposes it through deterministic traversal APIs.**

Data lineage forms the backbone of trustworthy AI pipelines, enabling organizations to trace data from source to model. The open-source **semantica-agi/semantica** repository implements a platform-agnostic provenance system that specifically integrates with enterprise platforms like Databricks Unity Catalog to maintain complete audit trails. This article examines the exact mechanisms Semantica uses to ingest, persist, and query data lineage from Databricks environments.

## The Three-Layer Architecture for Data Lineage Preservation

Semantica preserves data lineage through a tightly coupled stack that abstracts platform-specific metadata into a unified provenance model.

### Ingestion Layer: Extracting Lineage from Unity Catalog

The **DatabricksIngestor** class in [`semantica/ingest/databricks_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/databricks_ingestor.py) establishes authenticated connections to Databricks workspaces and interfaces directly with Unity Catalog's lineage APIs. The `DatabricksIngestor.get_table_lineage` method (lines 825-846) constructs fully-qualified table names and issues REST calls to `GET /api/2.0/lineage-tracking/table-lineage`, returning upstream tables, downstream consumers, and optional per-column lineage.

### Provenance Layer: Storing and Traversing Lineage Records

Lineage metadata persists as **ProvenanceEntry** records via the **ProvenanceManager** in [`semantica/provenance/manager.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance/manager.py). The `trace_lineage` method performs breadth-first search (BFS) traversal across stored entries, while `get_lineage` (lines 752-815) aggregates these records into rich lineage chains complete with metadata merging, checksum validation, and temporal annotations.

### Consumption Layer: Visualizing and Querying Lineage

Downstream components consume normalized lineage through [`semantica/visualization/visualization_provenance.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/visualization/visualization_provenance.py), which renders provenance data for vector-store indexing, graph visualizations, and audit logging. This layer ensures lineage data remains accessible regardless of the original enterprise platform.

## Extracting Lineage from Databricks Unity Catalog

Semantica connects to Databricks using PAT or OAuth M2M authentication, lazily instantiating a Unity Catalog `WorkspaceClient` upon first metadata request. The ingestion process captures both table-level and column-level lineage through structured API calls.

```python
from semantica.ingest import DatabricksIngestor

# Initialize the ingestor with Databricks workspace credentials

ingestor = DatabricksIngestor(
    host="https://adb-123.azuredatabricks.net",
    token="dapi-xxxxxx",
    http_path="/sql/1.0/warehouses/xxxxxx",
    catalog="main",
    schema="default",
)

# Retrieve comprehensive lineage including column-level details

lineage = ingestor.get_table_lineage(
    table_name="customers",
    include_column_lineage=True,
)

print(lineage["upstream"])      # → ['main.default.raw_customers']

print(lineage["downstream"])    # → ['main.default.customer_summary']

print(lineage["columns"]["id"]["upstream"])

# → ['main.default.raw_customers.customer_id']

```

*Source: [`DatabricksIngestor.get_table_lineage`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/databricks_ingestor.py#L825-L846)*

The method returns a dictionary containing three critical fields: `upstream` (source tables), `downstream` (consuming tables), and `columns` (per-column lineage mappings). This raw lineage feeds directly into Semantica's provenance storage layer.

## Persisting and Traversing Lineage with ProvenanceManager

Once ingested, lineage data persists through **ProvenanceStorage** implementations (SQLite, PostgreSQL, etc.) as `ProvenanceEntry` records. These entries include `entity_id`, `source_document`, `metadata`, timestamps, and checksums for integrity verification.

```python
from semantica.provenance.manager import ProvenanceManager

prov_mgr = ProvenanceManager()

# Query lineage for a previously ingested entity

lineage_info = prov_mgr.get_lineage("tbl_customers")

print(lineage_info["source_documents"])

# → ['databricks://main.default.customers']

print(lineage_info["metadata"])

# → {'owner': 'data_engineering', 'last_refresh': '2024-03-01'}

# Access full ancestry chain for visualization

for step in lineage_info["lineage_chain"]:
    print(step["entity_id"], step["activity_id"])

```

*Source: [`ProvenanceManager.get_lineage`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance/manager.py#L752-L815)*

The `get_lineage` method aggregates `ProvenanceEntry` records into a comprehensive dictionary containing:
- **lineage_chain**: Ordered list of ancestor entities
- **source_documents**: Deduplicated source references
- **first_seen** / **last_updated**: Temporal boundaries
- **metadata**: Combined key-value pairs with precedence given to recent entries
- **integrity_verified**: Boolean checksum validation status

For scenarios requiring raw traversal control, `trace_lineage(entity_id)` returns the BFS-ordered list of `ProvenanceEntry` objects without aggregation.

## End-to-End Data Lineage Flow

Semantica implements a deterministic pipeline for maintaining data lineage from ingestion to consumption:

1. **Ingest**: Connect to Databricks and extract table metadata via `DatabricksIngestor`
2. **Request**: Call `get_table_lineage()` to fetch upstream/downstream relationships from Unity Catalog
3. **Persist**: Store returned lineage as `ProvenanceEntry` records through the ingestion pipeline
4. **Query**: Retrieve aggregated lineage via `ProvenanceManager.get_lineage()` or traverse raw chains with `trace_lineage()`
5. **Consume**: Feed lineage data into vector stores, graph visualizers, or audit systems via the uniform provenance schema

Because the provenance model remains platform-agnostic, this same pattern extends to Snowflake, BigQuery, and other enterprise sources by implementing connector classes that map source metadata APIs to the `ProvenanceEntry` schema.

## Summary

- **Unity Catalog Integration**: Semantica extracts data lineage via `DatabricksIngestor.get_table_lineage` calling Databricks REST APIs at [`semantica/ingest/databricks_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/databricks_ingestor.py).
- **Persistent Storage**: Lineage records store as `ProvenanceEntry` objects with full metadata, timestamps, and checksums through `ProvenanceManager`.
- **Deterministic Traversal**: `trace_lineage` provides BFS traversal while `get_lineage` (lines 752-815) aggregates rich lineage dictionaries with merged metadata.
- **Cross-Platform Support**: The provenance abstraction in [`semantica/provenance/manager.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance/manager.py) enables data lineage preservation from any enterprise platform implementing the storage interface.
- **Downstream Consumption**: Visualization utilities in [`semantica/visualization/visualization_provenance.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/visualization/visualization_provenance.py) render lineage for knowledge graphs and vector stores.

## Frequently Asked Questions

### How does Semantica connect to Databricks to extract lineage metadata?

Semantica establishes connections through the `DatabricksConnector` class using either Personal Access Tokens (PAT) or OAuth M2M authentication. The connector lazily instantiates a Unity Catalog `WorkspaceClient` and exposes SQL connectivity for metadata extraction. According to the source code in [`semantica/ingest/databricks_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/databricks_ingestor.py), the ingestor builds fully-qualified table names before calling the Unity Catalog lineage tracking endpoints.

### What is the difference between trace_lineage and get_lineage in Semantica?

`trace_lineage` performs a raw breadth-first search across stored `ProvenanceEntry` records, returning an ordered list of entry objects representing the full upstream ancestry. In contrast, `get_lineage` (implemented at lines 752-815 in [`provenance/manager.py`](https://github.com/semantica-agi/semantica/blob/main/provenance/manager.py)) aggregates these entries into a structured dictionary that merges metadata, deduplicates source documents, and includes integrity verification status. Use `trace_lineage` for granular traversal control and `get_lineage` for comprehensive lineage reports.

### Can Semantica track column-level lineage from Databricks?

Yes. When calling `DatabricksIngestor.get_table_lineage()`, setting `include_column_lineage=True` triggers additional REST calls to Unity Catalog's column-lineage endpoints. The method returns a nested `columns` dictionary mapping each column to its upstream sources, enabling fine-grained impact analysis across table schemas.

### Is Semantica's data lineage model limited to Databricks only?

No. While `DatabricksIngestor` handles Unity Catalog specifically, the underlying `ProvenanceEntry` schema and `ProvenanceManager` APIs remain platform-agnostic. The architecture supports Snowflake, BigQuery, and other enterprise platforms by implementing ingestion connectors that map source-specific metadata APIs to the uniform provenance storage interface defined in [`semantica/provenance/manager.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance/manager.py).