# How to Authenticate DatabricksIngestor with Unity Catalog: PAT and OAuth M2M Guide

> Authenticate DatabricksIngestor with Unity Catalog using PAT or OAuth M2M. Learn how to secure your data pipelines with this essential guide.

- Repository: [Semantica /semantica](https://github.com/semantica-agi/semantica)
- Tags: how-to-guide
- Published: 2026-08-17

---

**Authenticate DatabricksIngestor with Unity Catalog by supplying either a Personal Access Token (PAT) or OAuth M2M service principal credentials to the `DatabricksConnector`, which centrally manages authentication for both SQL queries and Unity Catalog metadata operations.**

The `semantica-agi/semantica` library provides the **DatabricksIngestor** class to streamline data ingestion from Databricks Unity Catalog. Authentication is handled transparently through the internal `DatabricksConnector` component, ensuring that both SQL warehouse connections and Unity Catalog API calls use the same validated credentials.

## Authentication Architecture

The ingestor relies on two distinct clients that share authentication material from `DatabricksConnector`:

- **SQL Connection**: Built using `databricks-sql-connector` for warehouse queries
- **Unity Catalog Client**: Implemented via `databricks-sdk` `WorkspaceClient` for metadata operations

Both clients receive credentials from the same source, ensuring consistent access control across SQL and Unity Catalog operations as implemented in [`semantica/ingest/databricks_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/databricks_ingestor.py).

## Supported Authentication Methods

`DatabricksConnector` enforces that exactly one authentication method is provided during instantiation. Supplying both or neither triggers a **ValidationError** (defined in [`semantica/utils/exceptions.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/utils/exceptions.py) and raised in [`semantica/ingest/databricks_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/databricks_ingestor.py) lines 44-58).

### Personal Access Token (PAT)

Pass your Databricks workspace URL and personal access token directly to the constructor.

When `connect()` is invoked, the token is added to the SQL connection parameters as `access_token` (lines 100-126 in [`semantica/ingest/databricks_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/databricks_ingestor.py)). The identical token initializes the `WorkspaceClient` for Unity Catalog operations (lines 56-66).

```python
from semantica.ingest import DatabricksIngestor

ingestor = DatabricksIngestor(
    host="https://adb-123.azuredatabricks.net",
    token="dapi-EXAMPLEPAT",
    http_path="/sql/1.0/warehouses/1234abcd",
    catalog="my_catalog",
    schema="my_schema",
)
data = ingestor.ingest_table("customers", limit=100)
print(data.row_count)

```

### OAuth M2M Service Principal

For production workloads, use **OAuth machine-to-machine** authentication with a service principal's client ID and secret.

In [`semantica/ingest/databricks_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/databricks_ingestor.py), the `connect()` method constructs a `credentials_provider` callable using `databricks.sdk.core.oauth_service_principal` (lines 106-124). These same credentials instantiate the `WorkspaceClient` for Unity Catalog access (lines 56-66).

```python
from semantica.ingest import DatabricksIngestor

ingestor = DatabricksIngestor(
    host="https://adb-123.azuredatabricks.net",
    client_id="aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee",
    client_secret="super‑secret‑value",
    http_path="/sql/1.0/warehouses/1234abcd",
)
schemas = ingestor.list_schemas()
print(schemas)

```

## Environment Variable Configuration

Instead of hardcoding secrets, export standard Databricks environment variables. The constructor automatically detects these values when explicit parameters are omitted.

| Variable | Purpose |
|----------|---------|
| `DATABRICKS_HOST` | Workspace URL |
| `DATABRICKS_TOKEN` | Personal Access Token |
| `DATABRICKS_HTTP_PATH` | SQL warehouse HTTP path |
| `DATABRICKS_CLIENT_ID` | OAuth M2M client ID |
| `DATABRICKS_CLIENT_SECRET` | OAuth M2M client secret |
| `DATABRICKS_CATALOG` | Default Unity Catalog (defaults to `main`) |
| `DATABRICKS_SCHEMA` | Default schema (defaults to `default`) |

```bash
export DATABRICKS_HOST="https://adb-123.azuredatabricks.net"
export DATABRICKS_TOKEN="dapi-EXAMPLEPAT"
export DATABRICKS_HTTP_PATH="/sql/1.0/warehouses/1234abcd"
export DATABRICKS_CATALOG="my_catalog"
export DATABRICKS_SCHEMA="my_schema"

```

```python
from semantica.ingest import DatabricksIngestor

ingestor = DatabricksIngestor()   # picks up everything from env vars

catalogs = ingestor.list_catalogs()
print(catalogs)

```

## Unity Catalog Client Initialization

Unity Catalog operations require the `WorkspaceClient` provided by `get_workspace_client()`. This method lazily initializes the client using the same credentials validated at construction, ensuring that catalog exploration and schema inspection use identical authentication to the SQL connection.

The method is invoked internally when calling:

- `list_catalogs()`
- `list_schemas()`
- `list_tables()`
- `get_table_schema()`

## Connection Management with Context Managers

The ingestor supports Python's context manager protocol for automatic connection lifecycle management. This ensures `connect()` runs on entry and resources cleanup on exit.

```python
from semantica.ingest import DatabricksIngestor

with DatabricksIngestor(
    host="https://adb-123.azuredatabricks.net",
    token="dapi-EXAMPLEPAT",
    http_path="/sql/1.0/warehouses/1234abcd",
) as ingestor:
    docs = ingestor.export_as_documents(
        ingestor.ingest_table("sales", limit=500),
        text_fields=["product_name", "description"]
    )
    for doc in docs[:3]:
        print(doc["metadata"]["row_data"])

```

## Summary

- **Centralized Authentication**: `DatabricksConnector` manages a single set of credentials for both SQL connections and Unity Catalog APIs in [`semantica/ingest/databricks_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/databricks_ingestor.py).
- **Two Valid Methods**: Supply either a **Personal Access Token** (`host` + `token`) or **OAuth M2M credentials** (`host` + `client_id` + `client_secret`); mixing both or providing neither raises `ValidationError`.
- **Environment Fallback**: All authentication parameters can be sourced from standard `DATABRICKS_*` environment variables.
- **Unity Catalog Access**: The `get_workspace_client()` method creates a `databricks-sdk` client using validated credentials to support metadata operations.
- **Secure Patterns**: Use context managers (`with` statements) to ensure proper connection cleanup.

## Frequently Asked Questions

### Can I use both PAT and OAuth M2M authentication simultaneously?

No. The `DatabricksConnector` constructor explicitly validates that exactly one authentication method is provided. Supplying both `token` and `client_id`/`client_secret` raises a `ValidationError` during initialization (lines 44-58 in [`semantica/ingest/databricks_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/databricks_ingestor.py)). Choose either PAT for interactive development or OAuth M2M for service principal automation.

### How does the ingestor handle Unity Catalog authentication differently from SQL warehouse authentication?

It does not handle them differently. Both the SQL connection (via `databricks-sql-connector`) and the Unity Catalog client (via `databricks-sdk` `WorkspaceClient`) receive the same authentication material from `DatabricksConnector`. When you call `get_workspace_client()`, it uses the credentials validated during construction to create the WorkspaceClient (lines 56-66 in [`semantica/ingest/databricks_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/databricks_ingestor.py)).

### What happens if I do not provide authentication parameters?

If constructor arguments are omitted, the connector attempts to read from standard Databricks environment variables (`DATABRICKS_HOST`, `DATABRICKS_TOKEN`, etc.). If neither explicit parameters nor environment variables provide a valid authentication pair, the constructor raises a `ValidationError` indicating that authentication configuration is required.

### Is the Unity Catalog client created immediately when I instantiate DatabricksIngestor?

No. The `WorkspaceClient` for Unity Catalog operations is created lazily via `get_workspace_client()` when you first invoke a Unity Catalog method such as `list_catalogs()` or `get_table_schema()`. This deferred initialization ensures resources are only allocated when metadata operations are actually requested.