# How to Connect Semantica to Databricks Unity Catalog for Data Ingestion

> Connect Semantica to Databricks Unity Catalog for seamless data ingestion. Learn how to use the DatabricksIngestor class and official SDK for efficient table extraction.

- Repository: [Semantica /semantica](https://github.com/semantica-agi/semantica)
- Tags: how-to-guide
- Published: 2026-09-04

---

**Semantica connects to Databricks Unity Catalog through the `DatabricksIngestor` class, which authenticates via environment variables and extracts tables as `DocumentSet` objects using the official Databricks SDK.**

The `semantica-agi/semantica` repository provides a native integration that bridges Databricks Unity Catalog with Semantica's knowledge graph and vector store pipelines. By leveraging the `databricks-sdk` and `databricks-sql-connector` libraries, you can ingest structured data directly from Unity Catalog tables into downstream AI workflows without manual ETL processes.

## Installing the Databricks Integration

Before connecting to Unity Catalog, you must install the optional Databricks dependencies bundled with Semantica.

### Prerequisites and Package Installation

Install the Databricks extra to pull in the required client libraries:

```bash
pip install "semantica[databricks]"

```

This command installs `databricks-sdk` and `databricks-sql-connector`, which handle authentication, connection pooling, and SQL-based data extraction against Unity Catalog endpoints.

## Configuring Authentication Credentials

Semantica reads Databricks connection parameters from environment variables rather than hardcoded configuration files. This approach supports secure credential management in cloud environments and CI/CD pipelines.

Set the following variables before instantiating the ingestor:

- **`DATABRICKS_HOST`**: The workspace URL (e.g., `https://adb-1234567890123456.azuredatabricks.net`)
- **`DATABRICKS_TOKEN`**: A personal access token for token-based authentication
- **Or for service-principal authentication**:
  - **`DATABRICKS_CLIENT_ID`**
  - **`DATABRICKS_CLIENT_SECRET`**

The `DatabricksConnector` class, defined in [`semantica/ingest/databricks_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/databricks_ingestor.py), abstracts these credentials and manages connection lifecycle automatically.

## Ingesting Tables from Unity Catalog

The core interface for data extraction is the `DatabricksIngestor` class, which translates Unity Catalog tables into Semantica's internal `DocumentSet` representation.

### Understanding the DatabricksIngestor Class

Located in [`semantica/ingest/databricks_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/databricks_ingestor.py), this class wraps the low-level `DatabricksConnector` and exposes a high-level API for table ingestion. When initialized without arguments, it automatically constructs a connector from the environment variables described above.

### Using ingest_table() to Load Data

The `ingest_table()` method accepts a fully-qualified Unity Catalog identifier in the format `catalog.schema.table` and returns a `DocumentSet` containing row-level documents.

```python
import os
from semantica.ingest.databricks_ingestor import DatabricksIngestor

# Configure credentials via environment

os.environ["DATABRICKS_HOST"] = "https://adb-1234567890123456.azuredatabricks.net"
os.environ["DATABRICKS_TOKEN"] = "dapiXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"

# Initialize the ingestor (connector auto-loads from env vars)

ingestor = DatabricksIngestor()

# Ingest a specific Unity Catalog table

catalog_table = "my_catalog.sales.orders"
doc_set = ingestor.ingest_table(catalog_table)

print(f"Ingested {len(doc_set)} documents from {catalog_table}")

```

Each row in the source table becomes a document within the `DocumentSet`, preserving column names as metadata fields for downstream graph construction or vector indexing.

## End-to-End Pipeline Integration

Once ingested, the `DocumentSet` flows directly into Semantica's pipeline architecture. You can chain ingestion with knowledge graph construction and vector store persistence in a single execution block.

```python
from semantica.pipeline import Pipeline
from semantica.vector_store.pgvector_store import PGVectorStore

# Ingest from Unity Catalog

ingestor = DatabricksIngestor()
documents = ingestor.ingest_table("production.analytics.events")

# Configure downstream processing

pipeline = Pipeline() \
    .add_step("graph_builder") \
    .add_step(
        "vector_indexer", 
        store=PGVectorStore(connection_string="postgresql://user:pass@localhost/db")
    )

# Execute the full workflow

pipeline.run(documents)

```

This pattern enables real-time knowledge base construction from Databricks tables without intermediate file exports or manual schema mapping.

## Key Implementation Details

The Databricks integration is implemented across several key files in the `semantica-agi/semantica` repository:

- **[`semantica/ingest/databricks_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/databricks_ingestor.py)**: Contains both the `DatabricksConnector` class (handling authentication and connection pooling) and the `DatabricksIngestor` class (providing the `ingest_table()` API)
- **[`docs/integrations/databricks.md`](https://github.com/semantica-agi/semantica/blob/main/docs/integrations/databricks.md)**: Comprehensive integration guide covering advanced configuration and troubleshooting
- **[`tests/test_databricks_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/tests/test_databricks_ingestor.py)**: Unit tests verifying connector behavior, authentication modes, and table-level ingestion logic

The connector supports both token-based and OAuth service-principal authentication, with automatic retry logic and connection pooling managed internally by the underlying Databricks SDK.

## Summary

- **Install** the integration using `pip install "semantica[databricks]"` to acquire the required SDK dependencies
- **Configure** authentication via `DATABRICKS_HOST` and either `DATABRICKS_TOKEN` or service-principal credentials
- **Instantiate** `DatabricksIngestor` from [`semantica/ingest/databricks_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/databricks_ingestor.py) to create a connection handler
- **Ingest** tables using `ingest_table("catalog.schema.table")`, which returns a `DocumentSet` for downstream processing
- **Integrate** the output directly into Semantica pipelines for knowledge graph construction or vector store indexing

## Frequently Asked Questions

### What authentication methods does Semantica support for Databricks?

Semantica supports two authentication patterns as implemented in the `DatabricksConnector` class: personal access tokens via the `DATABRICKS_TOKEN` environment variable, and service-principal OAuth using `DATABRICKS_CLIENT_ID` and `DATABRICKS_CLIENT_SECRET`. Both methods require the `DATABRICKS_HOST` variable to be set to your workspace URL.

### How do I ingest a specific Unity Catalog table into Semantica?

Use the `ingest_table()` method of the `DatabricksIngestor` class, passing the fully-qualified table name as a string in the format `catalog_name.schema_name.table_name`. This method queries the table via the Databricks SQL connector and converts each row into a document within a Semantica `DocumentSet` object.

### Where can I find the source code for the Databricks integration?

The primary implementation resides in [`semantica/ingest/databricks_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/databricks_ingestor.py) within the `semantica-agi/semantica` repository. This file contains both the connection management logic and the high-level ingestion API. Additional documentation is available in [`docs/integrations/databricks.md`](https://github.com/semantica-agi/semantica/blob/main/docs/integrations/databricks.md), and test coverage is provided in [`tests/test_databricks_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/tests/test_databricks_ingestor.py).

### Does Semantica support incremental ingestion from Databricks?

The current `ingest_table()` implementation performs full-table reads via the `databricks-sql-connector`. For incremental loads, you would implement filtering logic at the SQL level by extending the ingestion query or processing the resulting `DocumentSet` through Semantica's deduplication pipeline steps before vector store insertion.