# How to Integrate Semantica with Databricks: A Complete Guide to Lakehouse Ingestion

> Integrate Semantica with Databricks for seamless lakehouse ingestion. Easily pull structured data into knowledge graphs using Unity Catalog and Delta Lake with PAT or OAuth M2M authentication.

- Repository: [Semantica /semantica](https://github.com/semantica-agi/semantica)
- Tags: how-to-guide
- Published: 2026-09-05

---

**Semantica provides a dedicated `DatabricksIngestor` class that connects directly to Unity Catalog and Delta Lake, enabling you to pull structured data into knowledge-graph pipelines with PAT or OAuth M2M authentication.**

The `semantica-agi/semantica` open-source repository ships with a first-class Databricks integration package. Whether you're building RAG applications or semantic knowledge graphs, this guide shows you how to authenticate, ingest tables, run custom queries, and extract lineage metadata—all using the actual source implementation in [`semantica/ingest/databricks_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/databricks_ingestor.py).

## Core Components of the Databricks Integration

The integration architecture consists of three tightly-coupled classes:

- **`DatabricksConnector`** – handles low-level SQL connections and Unity Catalog workspace clients
- **`DatabricksIngestor`** – the high-level API for application developers
- **`DatabricksData`** – a dataclass holding fetched rows, column metadata, and provenance

All three are implemented in [`semantica/ingest/databricks_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/databricks_ingestor.py), with initialization logic starting at **line 33** and connection management at **lines 68-84**.

### DatabricksConnector: Authentication and Connection Management

The `DatabricksConnector` class manages two authentication flows:

| Authentication | Environment Variables | Constructor Parameters |
|---|---|---|
| **Personal Access Token (PAT)** | `DATABRICKS_HOST`, `DATABRICKS_TOKEN`, `DATABRICKS_HTTP_PATH` | `host`, `token`, `http_path` |
| **OAuth M2M** | `DATABRICKS_HOST`, `DATABRICKS_CLIENT_ID`, `DATABRICKS_CLIENT_SECRET`, `DATABRICKS_HTTP_PATH` | `host`, `client_id`, `client_secret`, `http_path` |

If you omit parameters during `DatabricksIngestor` instantiation, the connector automatically falls back to environment variables defined at **lines 34-42** in [`databricks_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/databricks_ingestor.py). The actual TCP connection is established lazily in `connect()` at **lines 68-84**.

## Step-by-Step: Integrating Semantica with Databricks

### Step 1: Install Optional Dependencies

The module uses lazy imports for `databricks-sql-connector` and `databricks-sdk`. If missing, `DatabricksConnector` raises an `ImportError` with the installation command at **lines 25-30**:

```bash
pip install databricks-sql-connector databricks-sdk

```

### Step 2: Configure Authentication

Choose either PAT or OAuth M2M. For PAT, set three environment variables:

```bash
export DATABRICKS_HOST=https://adb-12345.azuredatabricks.net
export DATABRICKS_TOKEN=dapi-xxxxxxxxxxxxxxxx
export DATABRICKS_HTTP_PATH=/sql/1.0/warehouses/1111-2222-3333

```

For OAuth M2M service principals:

```bash
export DATABRICKS_CLIENT_ID=abcd-1234-efgh-5678
export DATABRICKS_CLIENT_SECRET=super-secret-value

```

### Step 3: Instantiate DatabricksIngestor

The ingestor builds an internal `DatabricksConnector` at **lines 63-73**. Override any environment defaults via constructor parameters:

```python
from semantica.ingest import DatabricksIngestor

ingestor = DatabricksIngestor(
    host="https://adb-12345.azuredatabricks.net",
    token="dapi-xxxxxxxxxxxxxxxx",
    http_path="/sql/1.0/warehouses/1111-2222-3333",
    catalog="analytics",
    schema="sales",
)

```

### Step 4: Ingest Delta Tables or Run Custom Queries

**Table ingestion** via `ingest_table()` at **lines 215-260**:

```python

# Pull first 5,000 rows with optional filtering

orders = ingestor.ingest_table(
    "orders",
    where="status = 'completed'",
    order_by="created_at DESC",
    limit=5000,
    offset=0,
)

```

**Custom SQL queries** via `ingest_query()` at **lines 52-63**:

```python
sales_qry = ingestor.ingest_query(
    "SELECT * FROM analytics.sales WHERE sale_date > :date",
    parameters={"date": "2024-01-01"},
    batch_size=2000,
)

```

Both methods build escaped SQL references, stream results to manage memory, and wrap output in a `DatabricksData` object defined at **lines 64-76**.

### Step 5: Convert to Semantica Documents

Use `export_as_documents()` to transform rows into the generic document format expected by downstream modules:

```python
documents = ingestor.export_as_documents(
    orders,
    id_field="order_id",
    text_fields=["customer_name", "product_name", "notes"],
)

```

This method is implemented at **lines 75-84** and prepares data for chunking, embedding, and graph construction.

## Advanced Features: Schema Discovery and Lineage

### Query Unity Catalog Metadata

Retrieve table schemas without ingestion using `get_table_schema()`:

```python
schema = ingestor.get_table_schema("customers")

```

List catalogs, schemas, and tables through the underlying connector methods.

### Extract Data Lineage

The `get_table_lineage()` method at **lines 124-138** retrieves upstream and downstream dependencies:

```python
lineage = ingestor.get_table_lineage(
    table_name="customers",
    catalog="marketing",
    schema="raw",
    include_column_lineage=True,
)

print("Upstream tables:", lineage["upstream"])
print("Downstream tables:", lineage["downstream"])

```

This provenance enriches knowledge-graph nodes with source tracking, visualized downstream in modules like [`semantica/visualization/visualization_provenance.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/visualization/visualization_provenance.py).

## Progress Tracking and Error Handling

Every ingestion step wraps the internal `self.progress_tracker` for real-time status updates. The tracker starts at `self.progress_tracker.start_tracking()` within `ingest_table()` at **lines 61-66**, providing:

- Streaming progress for large tables
- Batch-level error recovery
- Structured exceptions via `ProcessingError` and `ValidationError` from [`semantica/utils/exceptions.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/utils/exceptions.py)

## Complete Integration Example: OAuth M2M with Lineage

```python
from semantica.ingest import DatabricksIngestor

# OAuth M2M authentication

ingestor = DatabricksIngestor(
    host="https://adb-98765.azuredatabricks.net",
    client_id="abcd-1234-efgh-5678",
    client_secret="super-secret-value",
    http_path="/sql/1.0/warehouses/9999-8888-7777",
)

# Parameterized query with batching

results = ingestor.ingest_query(
    "SELECT * FROM analytics.sales WHERE sale_date > :date",
    parameters={"date": "2024-01-01"},
    batch_size=2000,
)

# Export with auto-detected text fields

documents = ingestor.export_as_documents(results)

# Enrich with lineage metadata

lineage = ingestor.get_table_lineage(
    table_name="sales",
    catalog="analytics",
    include_column_lineage=True,
)

```

## Summary

- **Secure authentication**: PAT or OAuth M2M via environment variables or explicit parameters in `DatabricksIngestor`
- **Flexible ingestion**: `ingest_table()` for full Delta tables, `ingest_query()` for arbitrary SQL with parameterization
- **Memory-efficient streaming**: Results batched and wrapped in `DatabricksData` objects
- **Native document conversion**: `export_as_documents()` bridges to Semantica's pipeline
- **Metadata enrichment**: Unity Catalog schema discovery and lineage extraction built-in
- **Production observability**: Integrated progress tracking and structured error handling

## Frequently Asked Questions

### What authentication methods does Semantica's Databricks integration support?

Semantica supports **Personal Access Tokens (PAT)** and **OAuth M2M** authentication. PAT uses a single token string, while OAuth M2M requires a client ID and client secret pair. Both methods authenticate against SQL warehouses or clusters via the `databricks-sql-connector`.

### How does Semantica handle large Delta tables without running out of memory?

The `ingest_table()` and `ingest_query()` methods stream results in configurable batches. The `batch_size` parameter controls row chunks, and `DatabricksData` holds metadata separately from row data. This design prevents full table materialization in memory.

### Can I use Semantica with Databricks without installing all dependencies?

Yes. The `databricks-sql-connector` and `databricks-sdk` are **lazy imports** at **lines 25-30** of [`databricks_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/databricks_ingestor.py). If absent, you receive a clear `ImportError` with the exact `pip install` command needed, keeping the base Semantica installation lightweight.

### What provenance information is captured during Databricks ingestion?

Every `DatabricksData` object tracks **source catalog, schema, table name, query text, and execution timestamp**. When combined with `get_table_lineage()`, you capture upstream/downstream table dependencies and column-level lineage for full dataflow visibility in your knowledge graph.