# Integrating Semantica with Databricks and Unity Catalog: A Complete Implementation Guide

> Implement Semantica with Databricks Unity Catalog using the native DatabricksIngestor module. Extract tables, schemas, and lineage data for LLMs with PAT or OAuth authentication.

- Repository: [Semantica /semantica](https://github.com/semantica-agi/semantica)
- Tags: how-to-guide
- Published: 2026-08-18

---

**Semantica provides a native `DatabricksIngestor` module that connects to Databricks Unity Catalog via the Databricks SDK and SQL connector, supporting both Personal Access Token and OAuth M2M service-principal authentication to extract tables, schemas, and lineage data into LLM-ready documents.**

Integrating Semantica with Databricks and Unity Catalog enables enterprise teams to ingest governed data assets directly into knowledge-graph pipelines. The `semantica-agi/semantica` repository ships a production-grade ingestion layer that handles authentication, SQL execution, and metadata extraction through a unified Python interface.

## Architecture and Core Components

The integration operates through three distinct layers defined in [`semantica/ingest/databricks_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/databricks_ingestor.py):

- **`DatabricksConnector`** (line 79): Manages low-level SQL connections via `databricks-sql-connector` and instantiates a `WorkspaceClient` from the `databricks-sdk` for Unity Catalog API operations.
- **`DatabricksIngestor`** (line 103): Orchestrates table and query ingestion, pagination, progress tracking, and converts raw results into structured dataclasses.
- **`DatabricksData`** (line 64): A container dataclass storing `data`, `row_count`, `columns`, and `metadata`, providing an `export_as_documents()` method for downstream LLM consumption.

## Prerequisites and Installation

The module requires two external libraries that are checked at runtime with clear `ImportError` messages if absent.

```bash
pip install databricks-sdk databricks-sql-connector

```

Configure your environment variables or pass credentials explicitly. The connector reads `DATABRICKS_HOST`, `DATABRICKS_TOKEN`, `DATABRICKS_HTTP_PATH`, `DATABRICKS_CLIENT_ID`, and `DATABRICKS_CLIENT_SECRET` automatically when available.

```bash
export DATABRICKS_HOST="https://adb-<workspace>.azuredatabricks.net"
export DATABRICKS_HTTP_PATH="/sql/1.0/warehouses/<warehouse-id>"
export DATABRICKS_CATALOG="main"
export DATABRICKS_SCHEMA="default"

```

## Authentication Methods

### Personal Access Token (PAT)

The simplest authentication method passes a token directly to `DatabricksConnector.connect()`, which invokes `databricks_sql.connect` with the `access_token` parameter.

```python
from semantica.ingest import DatabricksIngestor

ingestor = DatabricksIngestor(
    host="https://adb-xyz.azuredatabricks.net",
    token="dapi-abcdef123456",
    http_path="/sql/1.0/warehouses/1234",
)

```

### OAuth M2M Service Principal

For production workloads, use OAuth machine-to-machine authentication. The connector builds a `credentials_provider` callable using `oauth_service_principal(Config(...))`, compatible with `databricks-sql-connector >= 2.5`.

```python
ingestor = DatabricksIngestor(
    host="https://adb-xyz.azuredatabricks.net",
    client_id="sp-client-id",
    client_secret="sp-client-secret",
    http_path="/sql/1.0/warehouses/1234",
)

```

Verify connectivity before ingestion using the `test_connection()` method, which returns a boolean.

## Ingesting Data from Unity Catalog

### Table Ingestion with Safety Controls

The `ingest_table()` method constructs fully-qualified, back-ticked references (`catalog.schema.table`) via `_escaped_table_ref` and validates `WHERE` and `ORDER BY` clauses before execution.

```python

# Simple ingestion with row limit

data = ingestor.ingest_table("customers", limit=5000)

# Filtered ingestion with ordering

data = ingestor.ingest_table(
    "orders",
    where="order_date >= '2024-01-01'",
    order_by="order_date DESC",
)

```

### Arbitrary Query Execution

For complex transformations, use `ingest_query()` with optional parameter binding and batched fetching via `fetchmany`.

```python
query_data = ingestor.ingest_query(
    "SELECT * FROM sales WHERE region = :region",
    parameters={"region": "EMEA"},
    batch_size=1000,
)

```

## Unity Catalog Discovery and Metadata

### Schema and Object Enumeration

The `DatabricksConnector.get_workspace_client()` lazily initializes a `WorkspaceClient` to interact with Unity Catalog endpoints.

```python
catalogs = ingestor.list_catalogs()
schemas = ingestor.list_schemas(catalog="main")
tables = ingestor.list_tables(catalog="main", schema="default")

```

### Lineage Extraction

Extract table-level and column-level lineage by calling the Unity Catalog REST API endpoints `/api/2.0/lineage-tracking/table-lineage` and `/api/2.0/lineage-tracking/column-lineage`.

```python
schema = ingestor.get_table_schema("customers")
lineage = ingestor.get_table_lineage(
    "customers", 
    include_column_lineage=True
)

```

## Exporting to Semantica Documents

Convert `DatabricksData` instances into Semantica-compatible document dictionaries using `export_as_documents()`. This method maps database fields to `id`, `text`, and `metadata` keys required by vector stores.

```python
docs = ingestor.export_as_documents(
    data,
    id_field="customer_id",
    text_fields=["first_name", "last_name", "notes"],
)

# Feed directly to your vector store

# vector_store.add_documents(docs)

```

## Production Patterns and Connection Management

### Context Manager Usage

The `DatabricksIngestor` supports Python context managers to ensure connections are opened once, reused across multiple operations, and cleanly closed on exit. This pattern is validated in [`tests/test_databricks_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/tests/test_databricks_ingestor.py) (see `test_context_manager` and `test_context_manager_reuses_connection_across_calls`).

```python
with DatabricksIngestor(
    host="https://adb-abc.azuredatabricks.net",
    token="dapi-xyz123",
    http_path="/sql/1.0/warehouses/5678",
) as ingestor:
    
    # Connection opened here

    catalogs = ingestor.list_catalogs()
    customers = ingestor.ingest_table("customers", limit=1000)
    docs = ingestor.export_as_documents(customers, id_field="id")
    

# Connection automatically closed

```

### Manual Connection Lifecycle

For long-running processes, explicitly manage the connection lifecycle using `connect()` and `close()`.

```python
ingestor = DatabricksIngestor(...)
ingestor.connector.connect()

# ... perform operations ...

ingestor.close()

```

## Summary

- **Dual Authentication**: The connector supports both Personal Access Tokens and OAuth M2M service principals via environment variables or constructor arguments.
- **Unified Interface**: `DatabricksIngestor` combines SQL execution, Unity Catalog SDK operations, and document export in a single class.
- **Safety Features**: Automatic SQL escaping, `WHERE`/`ORDER BY` validation, and parameterized queries prevent injection attacks.
- **Metadata Richness**: Native support for schema discovery and lineage extraction through the Databricks Unity Catalog REST API.
- **Production Ready**: Context manager support ensures efficient connection pooling and cleanup, with progress tracking for large ingestion jobs.

## Frequently Asked Questions

### What authentication methods does Semantica support for Databricks?

Semantica supports **Personal Access Token (PAT)** authentication and **OAuth M2M service principal** authentication. PAT authentication passes the token directly to `databricks_sql.connect`, while OAuth M2M uses a `credentials_provider` callable compatible with `databricks-sql-connector >= 2.5`. Both methods can be configured via environment variables or passed explicitly to the `DatabricksIngestor` constructor.

### How does the integration handle Unity Catalog metadata like lineage?

The `DatabricksConnector` lazily instantiates a `WorkspaceClient` from the `databricks-sdk` to call Unity Catalog REST endpoints. The `get_table_lineage()` method invokes `/api/2.0/lineage-tracking/table-lineage` and optionally `/api/2.0/lineage-tracking/column-lineage` to retrieve upstream and downstream dependencies. This metadata can be used for data governance audits or knowledge-graph construction.

### Can I ingest data from arbitrary SQL queries rather than full tables?

Yes. The `ingest_query()` method accepts any valid SQL string with optional parameter binding and batched fetching. This allows you to ingest filtered views, joins, or aggregated results without creating temporary tables in Unity Catalog. Set `batch_size` to handle large result sets efficiently.

### Is connection pooling supported for high-throughput scenarios?

Yes. The `DatabricksConnector` caches the SQL connection on the instance and reuses it across multiple ingestion calls. Use the context manager pattern (`with DatabricksIngestor(...) as ingestor:`) to ensure the connection remains open for batch operations and is cleanly closed afterward, as verified in the test suite's `test_context_manager_reuses_connection_across_calls`.