How to Integrate Databricks Unity Catalog as a Native Data Source with Lineage in Semantica

Semantica's first-class Databricks connector reads Unity Catalog tables and pulls table-level or column-level lineage directly from the Unity Catalog API, eliminating the need for external ETL or manual provenance tracking.

The semantica-agi/semantica repository makes it straightforward to integrate Databricks Unity Catalog as a native data source with lineage inside its knowledge-graph framework. By combining SQL-based ingestion with direct Unity Catalog API calls, the connector lets you discover schemas, ingest rows, and retrieve upstream and downstream lineage in a single workflow. This guide walks through the implementation details found in semantica/ingest/databricks_ingestor.py.

How Semantica Integrates Databricks Unity Catalog as a Native Data Source

DatabricksConnector

The DatabricksConnector class in semantica/ingest/databricks_ingestor.py manages low-level connectivity, supporting both Personal Access Token (PAT) and OAuth-M2M authentication (lines [79–106]). It initializes a SQL connection via databricks-sql-connector and lazily instantiates a WorkspaceClient for Unity Catalog API access (lines [443–465]). When OAuth-M2M credentials are supplied, connect() wires a credentials_provider callable to handle token refresh automatically (lines [206–224]).

DatabricksIngestor

The DatabricksIngestor class serves as the high-level façade, exposing schema discovery, data ingestion, and lineage retrieval methods (class definition starting at line [303]). Through this interface, you can list catalogs, read table schemas, run ingestion jobs, and export results as Semantica documents without interacting with the low-level connector directly.

Unity Catalog Lineage API

Instead of parsing query logs, the connector calls the Unity Catalog lineage-tracking endpoints directly. The get_table_lineage method uses the SDK’s api_client.do to hit /api/2.0/lineage-tracking/table-lineage, then optionally iterates columns to call /lineage-tracking/column-lineage for fine-grained lineage (lines [826–903]).

Authenticating to Databricks

Personal Access Token (PAT)

PAT authentication is the quickest way to get started. You provide host, token, http_path, and optional default catalog and schema values. If omitted, the connector falls back to environment variables such as DATABRICKS_HOST and DATABRICKS_TOKEN.

from semantica.ingest import DatabricksIngestor

ingestor = DatabricksIngestor(
    host="https://adb-123.azuredatabricks.net",
    token="dapi-xxxxxxxxxxxxxxxxxxxxxxxx",
    http_path="/sql/1.0/warehouses/0123456789abcdef",
    catalog="main",
    schema="default",
)

assert ingestor.connector.test_connection()

Relevant implementation: DatabricksIngestor.__init__ (lines [333–373]).

OAuth M2M (Service Principal)

For production workloads, supply a client_id and client_secret. The DatabricksConnector.connect() method detects these credentials and builds an OAuth credentials_provider automatically (lines [206–224]).

ingestor = DatabricksIngestor(
    host="https://adb-123.azuredatabricks.net",
    client_id="xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    client_secret="yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy",
    http_path="/sql/1.0/warehouses/0123456789abcdef",
)

assert ingestor.connector.test_connection()

Discovering Unity Catalog Metadata

Before ingesting data, use the ingestor’s discovery methods to navigate the Unity Catalog hierarchy. These methods map directly to Unity Catalog API calls and return catalogs, schemas, tables, and column definitions.

catalogs = ingestor.list_catalogs()
schemas = ingestor.list_schemas(catalog="main")
tables = ingestor.list_tables(catalog="main", schema="default")
schema_info = ingestor.get_table_schema("customers")

Key methods: list_catalogs (lines [724–740]), list_schemas (lines [746–770]), list_tables (lines [795–819]), and get_table_schema (lines [558–580]).

Retrieving Table and Column Lineage

Because lineage is pulled natively from Unity Catalog, no separate ETL pipeline is required. Call get_table_lineage with include_column_lineage=False for table-level provenance, or set it to True to also fetch column-level lineage.

lineage = ingestor.get_table_lineage(
    table_name="customers",
    catalog="main",
    schema="default",
    include_column_lineage=False,
)

print("Upstream:", lineage["upstream"])
print("Downstream:", lineage["downstream"])

Core implementation: get_table_lineage (lines [826–860]).

Ingesting Data and Exporting Documents

After lineage is verified, ingest table rows or arbitrary SQL queries and convert the results into Semantica documents for downstream knowledge-graph processing.

data = ingestor.ingest_table("customers", limit=10000)

documents = ingestor.export_as_documents(
    data,
    text_fields=["name", "notes"]
)

print(f"Generated {len(documents)} documents")

Key methods: ingest_table (lines [416–461]) and export_as_documents (lines [976–1025]).

Summary

• The DatabricksIngestor in semantica/ingest/databricks_ingestor.py provides a unified interface to discover, ingest, and trace lineage from Databricks Unity Catalog.
• Authentication supports both PAT and OAuth-M2M, with automatic credential-provider wiring for service principals.
• Lineage is retrieved natively via the Unity Catalog /api/2.0/lineage-tracking/* endpoints, supporting both table-level and column-level granularity.
• Ingested results can be exported directly into Semantica documents using export_as_documents, streamlining knowledge-graph construction.

Frequently Asked Questions

Does the Semantica Databricks connector support OAuth authentication?

Yes. The DatabricksConnector.connect() method supports OAuth-M2M by accepting client_id and client_secret and automatically constructing a credentials_provider callable for token refresh, as implemented in lines [206–224] of semantica/ingest/databricks_ingestor.py.

What lineage granularity does the Unity Catalog integration provide?

The connector exposes both table-level and column-level lineage. get_table_lineage calls /lineage-tracking/table-lineage by default, and when include_column_lineage=True, it iterates over columns to additionally call /lineage-tracking/column-lineage (lines [826–903]).

Can I ingest data from Unity Catalog without manually extracting lineage first?

Yes. Data ingestion and lineage retrieval are independent operations. You can call ingest_table or ingest_query without ever calling get_table_lineage, or you can run both in the same session because the connector pulls lineage directly from Unity Catalog with no extra ETL required.

Which source file contains the core Databricks and lineage logic?

All core logic resides in semantica/ingest/databricks_ingestor.py. This single file defines DatabricksConnector, DatabricksIngestor, and the get_table_lineage method that interfaces with the Unity Catalog API.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}