How to Connect Semantica to Databricks Unity Catalog for Data Ingestion

Semantica connects to Databricks Unity Catalog through the DatabricksIngestor class, which authenticates via environment variables and extracts tables as DocumentSet objects using the official Databricks SDK.

The semantica-agi/semantica repository provides a native integration that bridges Databricks Unity Catalog with Semantica's knowledge graph and vector store pipelines. By leveraging the databricks-sdk and databricks-sql-connector libraries, you can ingest structured data directly from Unity Catalog tables into downstream AI workflows without manual ETL processes.

Installing the Databricks Integration

Before connecting to Unity Catalog, you must install the optional Databricks dependencies bundled with Semantica.

Prerequisites and Package Installation

Install the Databricks extra to pull in the required client libraries:

pip install "semantica[databricks]"

This command installs databricks-sdk and databricks-sql-connector, which handle authentication, connection pooling, and SQL-based data extraction against Unity Catalog endpoints.

Configuring Authentication Credentials

Semantica reads Databricks connection parameters from environment variables rather than hardcoded configuration files. This approach supports secure credential management in cloud environments and CI/CD pipelines.

Set the following variables before instantiating the ingestor:

• DATABRICKS_HOST: The workspace URL (e.g., https://adb-1234567890123456.azuredatabricks.net)
• DATABRICKS_TOKEN: A personal access token for token-based authentication
• Or for service-principal authentication:
  • DATABRICKS_CLIENT_ID
  • DATABRICKS_CLIENT_SECRET

The DatabricksConnector class, defined in semantica/ingest/databricks_ingestor.py, abstracts these credentials and manages connection lifecycle automatically.

Ingesting Tables from Unity Catalog

The core interface for data extraction is the DatabricksIngestor class, which translates Unity Catalog tables into Semantica's internal DocumentSet representation.

Understanding the DatabricksIngestor Class

Located in semantica/ingest/databricks_ingestor.py, this class wraps the low-level DatabricksConnector and exposes a high-level API for table ingestion. When initialized without arguments, it automatically constructs a connector from the environment variables described above.

Using ingest_table() to Load Data

The ingest_table() method accepts a fully-qualified Unity Catalog identifier in the format catalog.schema.table and returns a DocumentSet containing row-level documents.

import os
from semantica.ingest.databricks_ingestor import DatabricksIngestor

# Configure credentials via environment

os.environ["DATABRICKS_HOST"] = "https://adb-1234567890123456.azuredatabricks.net"
os.environ["DATABRICKS_TOKEN"] = "dapiXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"

# Initialize the ingestor (connector auto-loads from env vars)

ingestor = DatabricksIngestor()

# Ingest a specific Unity Catalog table

catalog_table = "my_catalog.sales.orders"
doc_set = ingestor.ingest_table(catalog_table)

print(f"Ingested {len(doc_set)} documents from {catalog_table}")

Each row in the source table becomes a document within the DocumentSet, preserving column names as metadata fields for downstream graph construction or vector indexing.

End-to-End Pipeline Integration

Once ingested, the DocumentSet flows directly into Semantica's pipeline architecture. You can chain ingestion with knowledge graph construction and vector store persistence in a single execution block.

from semantica.pipeline import Pipeline
from semantica.vector_store.pgvector_store import PGVectorStore

# Ingest from Unity Catalog

ingestor = DatabricksIngestor()
documents = ingestor.ingest_table("production.analytics.events")

# Configure downstream processing

pipeline = Pipeline() \
    .add_step("graph_builder") \
    .add_step(
        "vector_indexer", 
        store=PGVectorStore(connection_string="postgresql://user:pass@localhost/db")
    )

# Execute the full workflow

pipeline.run(documents)

This pattern enables real-time knowledge base construction from Databricks tables without intermediate file exports or manual schema mapping.

Key Implementation Details

The Databricks integration is implemented across several key files in the semantica-agi/semantica repository:

• semantica/ingest/databricks_ingestor.py: Contains both the DatabricksConnector class (handling authentication and connection pooling) and the DatabricksIngestor class (providing the ingest_table() API)
• docs/integrations/databricks.md: Comprehensive integration guide covering advanced configuration and troubleshooting
• tests/test_databricks_ingestor.py: Unit tests verifying connector behavior, authentication modes, and table-level ingestion logic

The connector supports both token-based and OAuth service-principal authentication, with automatic retry logic and connection pooling managed internally by the underlying Databricks SDK.

Summary

• Install the integration using pip install "semantica[databricks]" to acquire the required SDK dependencies
• Configure authentication via DATABRICKS_HOST and either DATABRICKS_TOKEN or service-principal credentials
• Instantiate DatabricksIngestor from semantica/ingest/databricks_ingestor.py to create a connection handler
• Ingest tables using ingest_table("catalog.schema.table"), which returns a DocumentSet for downstream processing
• Integrate the output directly into Semantica pipelines for knowledge graph construction or vector store indexing

Frequently Asked Questions

What authentication methods does Semantica support for Databricks?

Semantica supports two authentication patterns as implemented in the DatabricksConnector class: personal access tokens via the DATABRICKS_TOKEN environment variable, and service-principal OAuth using DATABRICKS_CLIENT_ID and DATABRICKS_CLIENT_SECRET. Both methods require the DATABRICKS_HOST variable to be set to your workspace URL.

How do I ingest a specific Unity Catalog table into Semantica?

Use the ingest_table() method of the DatabricksIngestor class, passing the fully-qualified table name as a string in the format catalog_name.schema_name.table_name. This method queries the table via the Databricks SQL connector and converts each row into a document within a Semantica DocumentSet object.

Where can I find the source code for the Databricks integration?

The primary implementation resides in semantica/ingest/databricks_ingestor.py within the semantica-agi/semantica repository. This file contains both the connection management logic and the high-level ingestion API. Additional documentation is available in docs/integrations/databricks.md, and test coverage is provided in tests/test_databricks_ingestor.py.

Does Semantica support incremental ingestion from Databricks?

The current ingest_table() implementation performs full-table reads via the databricks-sql-connector. For incremental loads, you would implement filtering logic at the SQL level by extending the ingestion query or processing the resulting DocumentSet through Semantica's deduplication pipeline steps before vector store insertion.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}