Integrating Semantica with Databricks and Unity Catalog: A Complete Implementation Guide

Semantica provides a native DatabricksIngestor module that connects to Databricks Unity Catalog via the Databricks SDK and SQL connector, supporting both Personal Access Token and OAuth M2M service-principal authentication to extract tables, schemas, and lineage data into LLM-ready documents.

Integrating Semantica with Databricks and Unity Catalog enables enterprise teams to ingest governed data assets directly into knowledge-graph pipelines. The semantica-agi/semantica repository ships a production-grade ingestion layer that handles authentication, SQL execution, and metadata extraction through a unified Python interface.

Architecture and Core Components

The integration operates through three distinct layers defined in semantica/ingest/databricks_ingestor.py:

• DatabricksConnector (line 79): Manages low-level SQL connections via databricks-sql-connector and instantiates a WorkspaceClient from the databricks-sdk for Unity Catalog API operations.
• DatabricksIngestor (line 103): Orchestrates table and query ingestion, pagination, progress tracking, and converts raw results into structured dataclasses.
• DatabricksData (line 64): A container dataclass storing data, row_count, columns, and metadata, providing an export_as_documents() method for downstream LLM consumption.

Prerequisites and Installation

The module requires two external libraries that are checked at runtime with clear ImportError messages if absent.

pip install databricks-sdk databricks-sql-connector

Configure your environment variables or pass credentials explicitly. The connector reads DATABRICKS_HOST, DATABRICKS_TOKEN, DATABRICKS_HTTP_PATH, DATABRICKS_CLIENT_ID, and DATABRICKS_CLIENT_SECRET automatically when available.

export DATABRICKS_HOST="https://adb-<workspace>.azuredatabricks.net"
export DATABRICKS_HTTP_PATH="/sql/1.0/warehouses/<warehouse-id>"
export DATABRICKS_CATALOG="main"
export DATABRICKS_SCHEMA="default"

Authentication Methods

Personal Access Token (PAT)

The simplest authentication method passes a token directly to DatabricksConnector.connect(), which invokes databricks_sql.connect with the access_token parameter.

from semantica.ingest import DatabricksIngestor

ingestor = DatabricksIngestor(
    host="https://adb-xyz.azuredatabricks.net",
    token="dapi-abcdef123456",
    http_path="/sql/1.0/warehouses/1234",
)

OAuth M2M Service Principal

For production workloads, use OAuth machine-to-machine authentication. The connector builds a credentials_provider callable using oauth_service_principal(Config(...)), compatible with databricks-sql-connector >= 2.5.

ingestor = DatabricksIngestor(
    host="https://adb-xyz.azuredatabricks.net",
    client_id="sp-client-id",
    client_secret="sp-client-secret",
    http_path="/sql/1.0/warehouses/1234",
)

Verify connectivity before ingestion using the test_connection() method, which returns a boolean.

Ingesting Data from Unity Catalog

Table Ingestion with Safety Controls

The ingest_table() method constructs fully-qualified, back-ticked references (catalog.schema.table) via _escaped_table_ref and validates WHERE and ORDER BY clauses before execution.

# Simple ingestion with row limit

data = ingestor.ingest_table("customers", limit=5000)

# Filtered ingestion with ordering

data = ingestor.ingest_table(
    "orders",
    where="order_date >= '2024-01-01'",
    order_by="order_date DESC",
)

Arbitrary Query Execution

For complex transformations, use ingest_query() with optional parameter binding and batched fetching via fetchmany.

query_data = ingestor.ingest_query(
    "SELECT * FROM sales WHERE region = :region",
    parameters={"region": "EMEA"},
    batch_size=1000,
)

Unity Catalog Discovery and Metadata

Schema and Object Enumeration

The DatabricksConnector.get_workspace_client() lazily initializes a WorkspaceClient to interact with Unity Catalog endpoints.

catalogs = ingestor.list_catalogs()
schemas = ingestor.list_schemas(catalog="main")
tables = ingestor.list_tables(catalog="main", schema="default")

Lineage Extraction

Extract table-level and column-level lineage by calling the Unity Catalog REST API endpoints /api/2.0/lineage-tracking/table-lineage and /api/2.0/lineage-tracking/column-lineage.

schema = ingestor.get_table_schema("customers")
lineage = ingestor.get_table_lineage(
    "customers", 
    include_column_lineage=True
)

Exporting to Semantica Documents

Convert DatabricksData instances into Semantica-compatible document dictionaries using export_as_documents(). This method maps database fields to id, text, and metadata keys required by vector stores.

docs = ingestor.export_as_documents(
    data,
    id_field="customer_id",
    text_fields=["first_name", "last_name", "notes"],
)

# Feed directly to your vector store

# vector_store.add_documents(docs)

Production Patterns and Connection Management

Context Manager Usage

The DatabricksIngestor supports Python context managers to ensure connections are opened once, reused across multiple operations, and cleanly closed on exit. This pattern is validated in tests/test_databricks_ingestor.py (see test_context_manager and test_context_manager_reuses_connection_across_calls).

with DatabricksIngestor(
    host="https://adb-abc.azuredatabricks.net",
    token="dapi-xyz123",
    http_path="/sql/1.0/warehouses/5678",
) as ingestor:
    
    # Connection opened here

    catalogs = ingestor.list_catalogs()
    customers = ingestor.ingest_table("customers", limit=1000)
    docs = ingestor.export_as_documents(customers, id_field="id")
    

# Connection automatically closed

Manual Connection Lifecycle

For long-running processes, explicitly manage the connection lifecycle using connect() and close().

ingestor = DatabricksIngestor(...)
ingestor.connector.connect()

# ... perform operations ...

ingestor.close()

Summary

• Dual Authentication: The connector supports both Personal Access Tokens and OAuth M2M service principals via environment variables or constructor arguments.
• Unified Interface: DatabricksIngestor combines SQL execution, Unity Catalog SDK operations, and document export in a single class.
• Safety Features: Automatic SQL escaping, WHERE/ORDER BY validation, and parameterized queries prevent injection attacks.
• Metadata Richness: Native support for schema discovery and lineage extraction through the Databricks Unity Catalog REST API.
• Production Ready: Context manager support ensures efficient connection pooling and cleanup, with progress tracking for large ingestion jobs.

Frequently Asked Questions

What authentication methods does Semantica support for Databricks?

Semantica supports Personal Access Token (PAT) authentication and OAuth M2M service principal authentication. PAT authentication passes the token directly to databricks_sql.connect, while OAuth M2M uses a credentials_provider callable compatible with databricks-sql-connector >= 2.5. Both methods can be configured via environment variables or passed explicitly to the DatabricksIngestor constructor.

How does the integration handle Unity Catalog metadata like lineage?

The DatabricksConnector lazily instantiates a WorkspaceClient from the databricks-sdk to call Unity Catalog REST endpoints. The get_table_lineage() method invokes /api/2.0/lineage-tracking/table-lineage and optionally /api/2.0/lineage-tracking/column-lineage to retrieve upstream and downstream dependencies. This metadata can be used for data governance audits or knowledge-graph construction.

Can I ingest data from arbitrary SQL queries rather than full tables?

Yes. The ingest_query() method accepts any valid SQL string with optional parameter binding and batched fetching. This allows you to ingest filtered views, joins, or aggregated results without creating temporary tables in Unity Catalog. Set batch_size to handle large result sets efficiently.

Is connection pooling supported for high-throughput scenarios?

Yes. The DatabricksConnector caches the SQL connection on the instance and reuses it across multiple ingestion calls. Use the context manager pattern (with DatabricksIngestor(...) as ingestor:) to ensure the connection remains open for batch operations and is cleanly closed afterward, as verified in the test suite's test_context_manager_reuses_connection_across_calls.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}