# How to Configure DatabricksIngestor for Unity Catalog Lineage Tracking

> Configure DatabricksIngestor for Unity Catalog lineage tracking using the get_table_lineage method. Learn how to retrieve upstream and downstream dependencies with the Databricks WorkspaceClient API.

- Repository: [Semantica /semantica](https://github.com/semantica-agi/semantica)
- Tags: how-to-guide
- Published: 2026-08-21

---

**The DatabricksIngestor class in the semantica-agi/semantica repository enables Unity Catalog lineage tracking through the `get_table_lineage` method, which retrieves upstream and downstream dependencies via the Databricks WorkspaceClient API.**

The `DatabricksIngestor` serves as the primary entry point for extracting data and interrogating Unity Catalog metadata from Databricks lakehouses. Implemented in [`semantica/ingest/databricks_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/databricks_ingestor.py), this component orchestrates SQL connections, manages authentication, and exposes dedicated methods for lineage retrieval at both table and column granularity. Proper configuration requires setting up the underlying connector and understanding how the ingestor delegates API calls to Unity Catalog endpoints.

## DatabricksIngestor Architecture Overview

The ingestion framework consists of three tightly-coupled components that handle distinct responsibilities when querying Unity Catalog lineage.

### DatabricksConnector

The **DatabricksConnector** manages low-level SQL connectivity and lazily instantiates a **WorkspaceClient** for Unity Catalog API calls. Located in [`semantica/ingest/databricks_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/databricks_ingestor.py) (lines 34-42), this component supports both Personal Access Token (PAT) and OAuth M2M authentication modes. The connector validates that `host` and either `token` or (`client_id` + `client_secret`) are present; missing values raise a `ValidationError` (lines 144-158). When using OAuth M2M, the connector builds a `credentials_provider` callable that supplies bearer tokens to the SQL connector (lines 206-224).

### DatabricksIngestor

The **DatabricksIngestor** orchestrates data ingestion and provides Unity Catalog helper methods: `list_catalogs`, `list_schemas`, `list_tables`, `get_table_schema`, and `get_table_lineage`. These methods delegate to the connector's `WorkspaceClient` (lines 124-138). The core lineage implementation resides in `get_table_lineage` (lines 825-842), which issues `GET /api/2.0/lineage-tracking/table-lineage` requests.

### DatabricksData

This dataclass stores retrieved rows with provenance metadata including table name, catalog, schema, query text, and ingestion timestamp (lines 64-76). It ensures lineage context persists alongside extracted data.

## Configuring Authentication and Environment Variables

Before retrieving Unity Catalog lineage, you must configure authentication through either explicit constructor arguments or environment variables.

### Supported Authentication Methods

- **PAT Authentication**: Provide `token` (Databricks Personal Access Token)
- **OAuth M2M**: Provide `client_id` and `client_secret` for service principal authentication

### Environment Variable Mapping

The connector reads configuration from the following environment variables if arguments are omitted:

- `DATABRICKS_HOST`
- `DATABRICKS_TOKEN` (for PAT)
- `DATABRICKS_HTTP_PATH`
- `DATABRICKS_CATALOG`
- `DATABRICKS_SCHEMA`
- `DATABRICKS_CLIENT_ID` (for OAuth)
- `DATABRICKS_CLIENT_SECRET` (for OAuth)

Unity Catalog defaults (`catalog="main"`, `schema="default"`) can be overridden at the ingestor level or via these environment variables.

## Retrieving Table-Level Lineage

To obtain lineage information for a specific table, instantiate the ingestor with appropriate catalog and schema scope, then call `get_table_lineage`.

The method returns upstream and downstream table relationships:

```python
from semantica.ingest import DatabricksIngestor

ingestor = DatabricksIngestor(
    host="https://adb-123.azuredatabricks.net",
    token="dapi-xxxxxxxxxxxxxxxx",
    http_path="/sql/1.0/warehouses/1111111111111111",
    catalog="sales",
    schema="default"
)

# Retrieve table-level lineage

lineage = ingestor.get_table_lineage("orders")
print("Upstream tables:", lineage["upstream"])
print("Downstream tables:", lineage["downstream"])

```

This delegates to the Unity Catalog API endpoint at `GET /api/2.0/lineage-tracking/table-lineage` via the `WorkspaceClient`.

## Enabling Column-Level Lineage

For granular dependency tracking, enable column-level lineage by setting `include_column_lineage=True`. This feature iterates over each column in the table schema and queries the `/api/2.0/lineage-tracking/column-lineage` endpoint.

The helper method `_get_column_lineage` (lines 910-939) builds a dictionary mapping each column to its upstream and downstream references:

```python

# Request column-level lineage (computationally expensive; use judiciously)

col_lineage = ingestor.get_table_lineage(
    "orders",
    catalog="sales", 
    schema="default",
    include_column_lineage=True
)

for col, details in col_lineage["columns"].items():
    print(f"Column {col} upstream: {details['upstream']}")
    print(f"Column {col} downstream: {details['downstream']}")

```

Note that column-level lineage requires additional API calls per column and should be used selectively for large tables.

## Complete Configuration Examples

### Basic PAT Authentication with Lineage

```python
from semantica.ingest import DatabricksIngestor

ingestor = DatabricksIngestor(
    host="https://adb-123.azuredatabricks.net",
    token="dapi-xxxxxxxxxxxxxxxx",
    http_path="/sql/1.0/warehouses/1111111111111111",
    catalog="marketing",
    schema="sales"
)

# Ingest data and track lineage

table_data = ingestor.ingest_table("customers", limit=5000)
lineage = ingestor.get_table_lineage("customers")

```

### OAuth M2M with Service Principal

```python
ingestor = DatabricksIngestor(
    host="https://adb-123.azuredatabricks.net",
    client_id="service-principal-id",
    client_secret="service-principal-secret",
    http_path="/sql/1.0/warehouses/1111111111111111"
)

# Query execution with lineage context

query_data = ingestor.ingest_query(
    "SELECT order_id, total_amount FROM sales.orders WHERE order_date > '2024-01-01'"
)
lineage = ingestor.get_table_lineage("orders", catalog="sales")

```

## Summary

- **DatabricksIngestor** provides the primary interface for Unity Catalog lineage tracking in the semantica-agi/semantica repository.
- Authentication supports both PAT and OAuth M2M methods, configured via constructor arguments or environment variables (`DATABRICKS_HOST`, `DATABRICKS_TOKEN`, etc.).
- **Table-level lineage** retrieval uses `get_table_lineage()` in [`semantica/ingest/databricks_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/databricks_ingestor.py) (lines 825-842), calling the WorkspaceClient API.
- **Column-level lineage** requires `include_column_lineage=True` and invokes `_get_column_lineage` (lines 910-939) for granular dependency mapping.
- The **DatabricksConnector** validates credentials and manages the underlying `WorkspaceClient` used for all Unity Catalog API interactions.
- The ingestor automatically enables global progress tracking for monitoring long-running lineage queries.

## Frequently Asked Questions

### What authentication methods does DatabricksIngestor support for Unity Catalog lineage?

**DatabricksIngestor supports Personal Access Token (PAT) and OAuth M2M (Machine-to-Machine) authentication.** For PAT, provide the `token` parameter; for OAuth M2M, provide both `client_id` and `client_secret`. The underlying `DatabricksConnector` validates that either `token` or both OAuth credentials are present alongside `host`, raising a `ValidationError` if configuration is incomplete according to lines 144-158 of [`semantica/ingest/databricks_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/databricks_ingestor.py).

### How does column-level lineage differ from table-level lineage in DatabricksIngestor?

**Table-level lineage returns upstream and downstream table relationships via a single API call to `/api/2.0/lineage-tracking/table-lineage`, while column-level lineage requires iterative queries to `/api/2.0/lineage-tracking/column-lineage` for each column.** When `include_column_lineage=True` is passed to `get_table_lineage()`, the ingestor calls the private `_get_column_lineage` method (lines 910-939), which builds a comprehensive mapping of column dependencies. This operation is significantly more expensive and should be used sparingly on large schemas.

### Can I use environment variables instead of constructor arguments for configuration?

**Yes, the DatabricksConnector automatically falls back to environment variables when constructor arguments are omitted.** The recognized variables are `DATABRICKS_HOST`, `DATABRICKS_TOKEN`, `DATABRICKS_HTTP_PATH`, `DATABRICKS_CATALOG`, `DATABRICKS_SCHEMA`, `DATABRICKS_CLIENT_ID`, and `DATABRICKS_CLIENT_SECRET`. Default values for catalog and schema can be set via `DATABRICKS_CATALOG` and `DATABRICKS_SCHEMA`, overriding the standard Unity Catalog defaults of "main" and "default".

### Where is the lineage API logic implemented in the source code?

**The lineage retrieval logic resides in [`semantica/ingest/databricks_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/databricks_ingestor.py)**, specifically in the `get_table_lineage` method (lines 825-842) for table-level dependencies and `_get_column_lineage` (lines 910-939) for column-level tracking. These methods delegate to the `WorkspaceClient` managed by `DatabricksConnector`, which handles the actual HTTP requests to Databricks Unity Catalog REST endpoints.