# How to Integrate Databricks Unity Catalog as a Native Data Source with Lineage in Semantica

> Integrate Databricks Unity Catalog as a native data source with Semantica. Pull table or column lineage directly from the Unity Catalog API, no ETL needed.

- Repository: [Semantica /semantica](https://github.com/semantica-agi/semantica)
- Tags: how-to-guide
- Published: 2026-08-20

---

**Semantica's first-class Databricks connector reads Unity Catalog tables and pulls table-level or column-level lineage directly from the Unity Catalog API, eliminating the need for external ETL or manual provenance tracking.**

The `semantica-agi/semantica` repository makes it straightforward to integrate Databricks Unity Catalog as a native data source with lineage inside its knowledge-graph framework. By combining SQL-based ingestion with direct Unity Catalog API calls, the connector lets you discover schemas, ingest rows, and retrieve upstream and downstream lineage in a single workflow. This guide walks through the implementation details found in [`semantica/ingest/databricks_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/databricks_ingestor.py).

## How Semantica Integrates Databricks Unity Catalog as a Native Data Source

### DatabricksConnector

The **`DatabricksConnector`** class in [`semantica/ingest/databricks_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/databricks_ingestor.py) manages low-level connectivity, supporting both **Personal Access Token (PAT)** and **OAuth-M2M** authentication (lines [79–106]). It initializes a SQL connection via `databricks-sql-connector` and lazily instantiates a `WorkspaceClient` for Unity Catalog API access (lines [443–465]). When OAuth-M2M credentials are supplied, `connect()` wires a `credentials_provider` callable to handle token refresh automatically (lines [206–224]).

### DatabricksIngestor

The **`DatabricksIngestor`** class serves as the high-level façade, exposing schema discovery, data ingestion, and lineage retrieval methods (class definition starting at line [303]). Through this interface, you can list catalogs, read table schemas, run ingestion jobs, and export results as Semantica documents without interacting with the low-level connector directly.

### Unity Catalog Lineage API

Instead of parsing query logs, the connector calls the Unity Catalog lineage-tracking endpoints directly. The `get_table_lineage` method uses the SDK’s `api_client.do` to hit `/api/2.0/lineage-tracking/table-lineage`, then optionally iterates columns to call `/lineage-tracking/column-lineage` for fine-grained lineage (lines [826–903]).

## Authenticating to Databricks

### Personal Access Token (PAT)

PAT authentication is the quickest way to get started. You provide `host`, `token`, `http_path`, and optional default `catalog` and `schema` values. If omitted, the connector falls back to environment variables such as `DATABRICKS_HOST` and `DATABRICKS_TOKEN`.

```python
from semantica.ingest import DatabricksIngestor

ingestor = DatabricksIngestor(
    host="https://adb-123.azuredatabricks.net",
    token="dapi-xxxxxxxxxxxxxxxxxxxxxxxx",
    http_path="/sql/1.0/warehouses/0123456789abcdef",
    catalog="main",
    schema="default",
)

assert ingestor.connector.test_connection()

```

*Relevant implementation:* `DatabricksIngestor.__init__` (lines [333–373]).

### OAuth M2M (Service Principal)

For production workloads, supply a `client_id` and `client_secret`. The `DatabricksConnector.connect()` method detects these credentials and builds an OAuth `credentials_provider` automatically (lines [206–224]).

```python
ingestor = DatabricksIngestor(
    host="https://adb-123.azuredatabricks.net",
    client_id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    client_secret="yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy",
    http_path="/sql/1.0/warehouses/0123456789abcdef",
)

assert ingestor.connector.test_connection()

```

## Discovering Unity Catalog Metadata

Before ingesting data, use the ingestor’s discovery methods to navigate the Unity Catalog hierarchy. These methods map directly to Unity Catalog API calls and return catalogs, schemas, tables, and column definitions.

```python
catalogs = ingestor.list_catalogs()
schemas = ingestor.list_schemas(catalog="main")
tables = ingestor.list_tables(catalog="main", schema="default")
schema_info = ingestor.get_table_schema("customers")

```

*Key methods:* `list_catalogs` (lines [724–740]), `list_schemas` (lines [746–770]), `list_tables` (lines [795–819]), and `get_table_schema` (lines [558–580]).

## Retrieving Table and Column Lineage

Because lineage is pulled natively from Unity Catalog, no separate ETL pipeline is required. Call `get_table_lineage` with `include_column_lineage=False` for table-level provenance, or set it to `True` to also fetch column-level lineage.

```python
lineage = ingestor.get_table_lineage(
    table_name="customers",
    catalog="main",
    schema="default",
    include_column_lineage=False,
)

print("Upstream:", lineage["upstream"])
print("Downstream:", lineage["downstream"])

```

*Core implementation:* `get_table_lineage` (lines [826–860]).

## Ingesting Data and Exporting Documents

After lineage is verified, ingest table rows or arbitrary SQL queries and convert the results into Semantica documents for downstream knowledge-graph processing.

```python
data = ingestor.ingest_table("customers", limit=10000)

documents = ingestor.export_as_documents(
    data,
    text_fields=["name", "notes"]
)

print(f"Generated {len(documents)} documents")

```

*Key methods:* `ingest_table` (lines [416–461]) and `export_as_documents` (lines [976–1025]).

## Summary

- The **`DatabricksIngestor`** in [`semantica/ingest/databricks_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/databricks_ingestor.py) provides a unified interface to discover, ingest, and trace lineage from Databricks Unity Catalog.
- Authentication supports both **PAT** and **OAuth-M2M**, with automatic credential-provider wiring for service principals.
- Lineage is retrieved natively via the Unity Catalog `/api/2.0/lineage-tracking/*` endpoints, supporting both table-level and column-level granularity.
- Ingested results can be exported directly into Semantica documents using `export_as_documents`, streamlining knowledge-graph construction.

## Frequently Asked Questions

### Does the Semantica Databricks connector support OAuth authentication?

Yes. The `DatabricksConnector.connect()` method supports OAuth-M2M by accepting `client_id` and `client_secret` and automatically constructing a `credentials_provider` callable for token refresh, as implemented in lines [206–224] of [`semantica/ingest/databricks_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/databricks_ingestor.py).

### What lineage granularity does the Unity Catalog integration provide?

The connector exposes both table-level and column-level lineage. `get_table_lineage` calls `/lineage-tracking/table-lineage` by default, and when `include_column_lineage=True`, it iterates over columns to additionally call `/lineage-tracking/column-lineage` (lines [826–903]).

### Can I ingest data from Unity Catalog without manually extracting lineage first?

Yes. Data ingestion and lineage retrieval are independent operations. You can call `ingest_table` or `ingest_query` without ever calling `get_table_lineage`, or you can run both in the same session because the connector pulls lineage directly from Unity Catalog with no extra ETL required.

### Which source file contains the core Databricks and lineage logic?

All core logic resides in [`semantica/ingest/databricks_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/databricks_ingestor.py). This single file defines `DatabricksConnector`, `DatabricksIngestor`, and the `get_table_lineage` method that interfaces with the Unity Catalog API.