How to Authenticate DatabricksIngestor with Unity Catalog: PAT and OAuth M2M Guide

Authenticate DatabricksIngestor with Unity Catalog by supplying either a Personal Access Token (PAT) or OAuth M2M service principal credentials to the DatabricksConnector, which centrally manages authentication for both SQL queries and Unity Catalog metadata operations.

The semantica-agi/semantica library provides the DatabricksIngestor class to streamline data ingestion from Databricks Unity Catalog. Authentication is handled transparently through the internal DatabricksConnector component, ensuring that both SQL warehouse connections and Unity Catalog API calls use the same validated credentials.

Authentication Architecture

The ingestor relies on two distinct clients that share authentication material from DatabricksConnector:

• SQL Connection: Built using databricks-sql-connector for warehouse queries
• Unity Catalog Client: Implemented via databricks-sdk WorkspaceClient for metadata operations

Both clients receive credentials from the same source, ensuring consistent access control across SQL and Unity Catalog operations as implemented in semantica/ingest/databricks_ingestor.py.

Supported Authentication Methods

DatabricksConnector enforces that exactly one authentication method is provided during instantiation. Supplying both or neither triggers a ValidationError (defined in semantica/utils/exceptions.py and raised in semantica/ingest/databricks_ingestor.py lines 44-58).

Personal Access Token (PAT)

Pass your Databricks workspace URL and personal access token directly to the constructor.

When connect() is invoked, the token is added to the SQL connection parameters as access_token (lines 100-126 in semantica/ingest/databricks_ingestor.py). The identical token initializes the WorkspaceClient for Unity Catalog operations (lines 56-66).

from semantica.ingest import DatabricksIngestor

ingestor = DatabricksIngestor(
    host="https://adb-123.azuredatabricks.net",
    token="dapi-EXAMPLEPAT",
    http_path="/sql/1.0/warehouses/1234abcd",
    catalog="my_catalog",
    schema="my_schema",
)
data = ingestor.ingest_table("customers", limit=100)
print(data.row_count)

OAuth M2M Service Principal

For production workloads, use OAuth machine-to-machine authentication with a service principal's client ID and secret.

In semantica/ingest/databricks_ingestor.py, the connect() method constructs a credentials_provider callable using databricks.sdk.core.oauth_service_principal (lines 106-124). These same credentials instantiate the WorkspaceClient for Unity Catalog access (lines 56-66).

from semantica.ingest import DatabricksIngestor

ingestor = DatabricksIngestor(
    host="https://adb-123.azuredatabricks.net",
    client_id="aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee",
    client_secret="super‑secret‑value",
    http_path="/sql/1.0/warehouses/1234abcd",
)
schemas = ingestor.list_schemas()
print(schemas)

Environment Variable Configuration

Instead of hardcoding secrets, export standard Databricks environment variables. The constructor automatically detects these values when explicit parameters are omitted.

Variable	Purpose
DATABRICKS_HOST	Workspace URL
DATABRICKS_TOKEN	Personal Access Token
DATABRICKS_HTTP_PATH	SQL warehouse HTTP path
DATABRICKS_CLIENT_ID	OAuth M2M client ID
DATABRICKS_CLIENT_SECRET	OAuth M2M client secret
DATABRICKS_CATALOG	Default Unity Catalog (defaults to main)
DATABRICKS_SCHEMA	Default schema (defaults to default)

export DATABRICKS_HOST="https://adb-123.azuredatabricks.net"
export DATABRICKS_TOKEN="dapi-EXAMPLEPAT"
export DATABRICKS_HTTP_PATH="/sql/1.0/warehouses/1234abcd"
export DATABRICKS_CATALOG="my_catalog"
export DATABRICKS_SCHEMA="my_schema"

from semantica.ingest import DatabricksIngestor

ingestor = DatabricksIngestor()   # picks up everything from env vars

catalogs = ingestor.list_catalogs()
print(catalogs)

Unity Catalog Client Initialization

Unity Catalog operations require the WorkspaceClient provided by get_workspace_client(). This method lazily initializes the client using the same credentials validated at construction, ensuring that catalog exploration and schema inspection use identical authentication to the SQL connection.

The method is invoked internally when calling:

• list_catalogs()
• list_schemas()
• list_tables()
• get_table_schema()

Connection Management with Context Managers

The ingestor supports Python's context manager protocol for automatic connection lifecycle management. This ensures connect() runs on entry and resources cleanup on exit.

from semantica.ingest import DatabricksIngestor

with DatabricksIngestor(
    host="https://adb-123.azuredatabricks.net",
    token="dapi-EXAMPLEPAT",
    http_path="/sql/1.0/warehouses/1234abcd",
) as ingestor:
    docs = ingestor.export_as_documents(
        ingestor.ingest_table("sales", limit=500),
        text_fields=["product_name", "description"]
    )
    for doc in docs[:3]:
        print(doc["metadata"]["row_data"])

Summary

• Centralized Authentication: DatabricksConnector manages a single set of credentials for both SQL connections and Unity Catalog APIs in semantica/ingest/databricks_ingestor.py.
• Two Valid Methods: Supply either a Personal Access Token (host + token) or OAuth M2M credentials (host + client_id + client_secret); mixing both or providing neither raises ValidationError.
• Environment Fallback: All authentication parameters can be sourced from standard DATABRICKS_* environment variables.
• Unity Catalog Access: The get_workspace_client() method creates a databricks-sdk client using validated credentials to support metadata operations.
• Secure Patterns: Use context managers (with statements) to ensure proper connection cleanup.

Frequently Asked Questions

Can I use both PAT and OAuth M2M authentication simultaneously?

No. The DatabricksConnector constructor explicitly validates that exactly one authentication method is provided. Supplying both token and client_id/client_secret raises a ValidationError during initialization (lines 44-58 in semantica/ingest/databricks_ingestor.py). Choose either PAT for interactive development or OAuth M2M for service principal automation.

How does the ingestor handle Unity Catalog authentication differently from SQL warehouse authentication?

It does not handle them differently. Both the SQL connection (via databricks-sql-connector) and the Unity Catalog client (via databricks-sdk WorkspaceClient) receive the same authentication material from DatabricksConnector. When you call get_workspace_client(), it uses the credentials validated during construction to create the WorkspaceClient (lines 56-66 in semantica/ingest/databricks_ingestor.py).

What happens if I do not provide authentication parameters?

If constructor arguments are omitted, the connector attempts to read from standard Databricks environment variables (DATABRICKS_HOST, DATABRICKS_TOKEN, etc.). If neither explicit parameters nor environment variables provide a valid authentication pair, the constructor raises a ValidationError indicating that authentication configuration is required.

Is the Unity Catalog client created immediately when I instantiate DatabricksIngestor?

No. The WorkspaceClient for Unity Catalog operations is created lazily via get_workspace_client() when you first invoke a Unity Catalog method such as list_catalogs() or get_table_schema(). This deferred initialization ensures resources are only allocated when metadata operations are actually requested.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}