How to Configure DatabricksIngestor for Unity Catalog Lineage Tracking

The DatabricksIngestor class in the semantica-agi/semantica repository enables Unity Catalog lineage tracking through the get_table_lineage method, which retrieves upstream and downstream dependencies via the Databricks WorkspaceClient API.

The DatabricksIngestor serves as the primary entry point for extracting data and interrogating Unity Catalog metadata from Databricks lakehouses. Implemented in semantica/ingest/databricks_ingestor.py, this component orchestrates SQL connections, manages authentication, and exposes dedicated methods for lineage retrieval at both table and column granularity. Proper configuration requires setting up the underlying connector and understanding how the ingestor delegates API calls to Unity Catalog endpoints.

DatabricksIngestor Architecture Overview

The ingestion framework consists of three tightly-coupled components that handle distinct responsibilities when querying Unity Catalog lineage.

DatabricksConnector

The DatabricksConnector manages low-level SQL connectivity and lazily instantiates a WorkspaceClient for Unity Catalog API calls. Located in semantica/ingest/databricks_ingestor.py (lines 34-42), this component supports both Personal Access Token (PAT) and OAuth M2M authentication modes. The connector validates that host and either token or (client_id + client_secret) are present; missing values raise a ValidationError (lines 144-158). When using OAuth M2M, the connector builds a credentials_provider callable that supplies bearer tokens to the SQL connector (lines 206-224).

DatabricksIngestor

The DatabricksIngestor orchestrates data ingestion and provides Unity Catalog helper methods: list_catalogs, list_schemas, list_tables, get_table_schema, and get_table_lineage. These methods delegate to the connector's WorkspaceClient (lines 124-138). The core lineage implementation resides in get_table_lineage (lines 825-842), which issues GET /api/2.0/lineage-tracking/table-lineage requests.

DatabricksData

This dataclass stores retrieved rows with provenance metadata including table name, catalog, schema, query text, and ingestion timestamp (lines 64-76). It ensures lineage context persists alongside extracted data.

Configuring Authentication and Environment Variables

Before retrieving Unity Catalog lineage, you must configure authentication through either explicit constructor arguments or environment variables.

Supported Authentication Methods

• PAT Authentication: Provide token (Databricks Personal Access Token)
• OAuth M2M: Provide client_id and client_secret for service principal authentication

Environment Variable Mapping

The connector reads configuration from the following environment variables if arguments are omitted:

• DATABRICKS_HOST
• DATABRICKS_TOKEN (for PAT)
• DATABRICKS_HTTP_PATH
• DATABRICKS_CATALOG
• DATABRICKS_SCHEMA
• DATABRICKS_CLIENT_ID (for OAuth)
• DATABRICKS_CLIENT_SECRET (for OAuth)

Unity Catalog defaults (catalog="main", schema="default") can be overridden at the ingestor level or via these environment variables.

Retrieving Table-Level Lineage

To obtain lineage information for a specific table, instantiate the ingestor with appropriate catalog and schema scope, then call get_table_lineage.

The method returns upstream and downstream table relationships:

from semantica.ingest import DatabricksIngestor

ingestor = DatabricksIngestor(
    host="https://adb-123.azuredatabricks.net",
    token="dapi-xxxxxxxxxxxxxxxx",
    http_path="/sql/1.0/warehouses/1111111111111111",
    catalog="sales",
    schema="default"
)

# Retrieve table-level lineage

lineage = ingestor.get_table_lineage("orders")
print("Upstream tables:", lineage["upstream"])
print("Downstream tables:", lineage["downstream"])

This delegates to the Unity Catalog API endpoint at GET /api/2.0/lineage-tracking/table-lineage via the WorkspaceClient.

Enabling Column-Level Lineage

For granular dependency tracking, enable column-level lineage by setting include_column_lineage=True. This feature iterates over each column in the table schema and queries the /api/2.0/lineage-tracking/column-lineage endpoint.

The helper method _get_column_lineage (lines 910-939) builds a dictionary mapping each column to its upstream and downstream references:

# Request column-level lineage (computationally expensive; use judiciously)

col_lineage = ingestor.get_table_lineage(
    "orders",
    catalog="sales", 
    schema="default",
    include_column_lineage=True
)

for col, details in col_lineage["columns"].items():
    print(f"Column {col} upstream: {details['upstream']}")
    print(f"Column {col} downstream: {details['downstream']}")

Note that column-level lineage requires additional API calls per column and should be used selectively for large tables.

Complete Configuration Examples

Basic PAT Authentication with Lineage

from semantica.ingest import DatabricksIngestor

ingestor = DatabricksIngestor(
    host="https://adb-123.azuredatabricks.net",
    token="dapi-xxxxxxxxxxxxxxxx",
    http_path="/sql/1.0/warehouses/1111111111111111",
    catalog="marketing",
    schema="sales"
)

# Ingest data and track lineage

table_data = ingestor.ingest_table("customers", limit=5000)
lineage = ingestor.get_table_lineage("customers")

OAuth M2M with Service Principal

ingestor = DatabricksIngestor(
    host="https://adb-123.azuredatabricks.net",
    client_id="service-principal-id",
    client_secret="service-principal-secret",
    http_path="/sql/1.0/warehouses/1111111111111111"
)

# Query execution with lineage context

query_data = ingestor.ingest_query(
    "SELECT order_id, total_amount FROM sales.orders WHERE order_date > '2024-01-01'"
)
lineage = ingestor.get_table_lineage("orders", catalog="sales")

Summary

• DatabricksIngestor provides the primary interface for Unity Catalog lineage tracking in the semantica-agi/semantica repository.
• Authentication supports both PAT and OAuth M2M methods, configured via constructor arguments or environment variables (DATABRICKS_HOST, DATABRICKS_TOKEN, etc.).
• Table-level lineage retrieval uses get_table_lineage() in semantica/ingest/databricks_ingestor.py (lines 825-842), calling the WorkspaceClient API.
• Column-level lineage requires include_column_lineage=True and invokes _get_column_lineage (lines 910-939) for granular dependency mapping.
• The DatabricksConnector validates credentials and manages the underlying WorkspaceClient used for all Unity Catalog API interactions.
• The ingestor automatically enables global progress tracking for monitoring long-running lineage queries.

Frequently Asked Questions

What authentication methods does DatabricksIngestor support for Unity Catalog lineage?

DatabricksIngestor supports Personal Access Token (PAT) and OAuth M2M (Machine-to-Machine) authentication. For PAT, provide the token parameter; for OAuth M2M, provide both client_id and client_secret. The underlying DatabricksConnector validates that either token or both OAuth credentials are present alongside host, raising a ValidationError if configuration is incomplete according to lines 144-158 of semantica/ingest/databricks_ingestor.py.

How does column-level lineage differ from table-level lineage in DatabricksIngestor?

Table-level lineage returns upstream and downstream table relationships via a single API call to /api/2.0/lineage-tracking/table-lineage, while column-level lineage requires iterative queries to /api/2.0/lineage-tracking/column-lineage for each column. When include_column_lineage=True is passed to get_table_lineage(), the ingestor calls the private _get_column_lineage method (lines 910-939), which builds a comprehensive mapping of column dependencies. This operation is significantly more expensive and should be used sparingly on large schemas.

Can I use environment variables instead of constructor arguments for configuration?

Yes, the DatabricksConnector automatically falls back to environment variables when constructor arguments are omitted. The recognized variables are DATABRICKS_HOST, DATABRICKS_TOKEN, DATABRICKS_HTTP_PATH, DATABRICKS_CATALOG, DATABRICKS_SCHEMA, DATABRICKS_CLIENT_ID, and DATABRICKS_CLIENT_SECRET. Default values for catalog and schema can be set via DATABRICKS_CATALOG and DATABRICKS_SCHEMA, overriding the standard Unity Catalog defaults of "main" and "default".

Where is the lineage API logic implemented in the source code?

The lineage retrieval logic resides in semantica/ingest/databricks_ingestor.py, specifically in the get_table_lineage method (lines 825-842) for table-level dependencies and _get_column_lineage (lines 910-939) for column-level tracking. These methods delegate to the WorkspaceClient managed by DatabricksConnector, which handles the actual HTTP requests to Databricks Unity Catalog REST endpoints.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}