How to Integrate Semantica with Databricks: A Complete Guide to Lakehouse Ingestion

Semantica provides a dedicated DatabricksIngestor class that connects directly to Unity Catalog and Delta Lake, enabling you to pull structured data into knowledge-graph pipelines with PAT or OAuth M2M authentication.

The semantica-agi/semantica open-source repository ships with a first-class Databricks integration package. Whether you're building RAG applications or semantic knowledge graphs, this guide shows you how to authenticate, ingest tables, run custom queries, and extract lineage metadata—all using the actual source implementation in semantica/ingest/databricks_ingestor.py.

Core Components of the Databricks Integration

The integration architecture consists of three tightly-coupled classes:

• DatabricksConnector – handles low-level SQL connections and Unity Catalog workspace clients
• DatabricksIngestor – the high-level API for application developers
• DatabricksData – a dataclass holding fetched rows, column metadata, and provenance

All three are implemented in semantica/ingest/databricks_ingestor.py, with initialization logic starting at line 33 and connection management at lines 68-84.

DatabricksConnector: Authentication and Connection Management

The DatabricksConnector class manages two authentication flows:

Authentication	Environment Variables	Constructor Parameters
Personal Access Token (PAT)	DATABRICKS_HOST, DATABRICKS_TOKEN, DATABRICKS_HTTP_PATH	host, token, http_path
OAuth M2M	DATABRICKS_HOST, DATABRICKS_CLIENT_ID, DATABRICKS_CLIENT_SECRET, DATABRICKS_HTTP_PATH	host, client_id, client_secret, http_path

If you omit parameters during DatabricksIngestor instantiation, the connector automatically falls back to environment variables defined at lines 34-42 in databricks_ingestor.py. The actual TCP connection is established lazily in connect() at lines 68-84.

Step-by-Step: Integrating Semantica with Databricks

Step 1: Install Optional Dependencies

The module uses lazy imports for databricks-sql-connector and databricks-sdk. If missing, DatabricksConnector raises an ImportError with the installation command at lines 25-30:

pip install databricks-sql-connector databricks-sdk

Step 2: Configure Authentication

Choose either PAT or OAuth M2M. For PAT, set three environment variables:

export DATABRICKS_HOST=https://adb-12345.azuredatabricks.net
export DATABRICKS_TOKEN=dapi-xxxxxxxxxxxxxxxx
export DATABRICKS_HTTP_PATH=/sql/1.0/warehouses/1111-2222-3333

For OAuth M2M service principals:

export DATABRICKS_CLIENT_ID=abcd-1234-efgh-5678
export DATABRICKS_CLIENT_SECRET=super-secret-value

Step 3: Instantiate DatabricksIngestor

The ingestor builds an internal DatabricksConnector at lines 63-73. Override any environment defaults via constructor parameters:

from semantica.ingest import DatabricksIngestor

ingestor = DatabricksIngestor(
    host="https://adb-12345.azuredatabricks.net",
    token="dapi-xxxxxxxxxxxxxxxx",
    http_path="/sql/1.0/warehouses/1111-2222-3333",
    catalog="analytics",
    schema="sales",
)

Step 4: Ingest Delta Tables or Run Custom Queries

Table ingestion via ingest_table() at lines 215-260:

# Pull first 5,000 rows with optional filtering

orders = ingestor.ingest_table(
    "orders",
    where="status = 'completed'",
    order_by="created_at DESC",
    limit=5000,
    offset=0,
)

Custom SQL queries via ingest_query() at lines 52-63:

sales_qry = ingestor.ingest_query(
    "SELECT * FROM analytics.sales WHERE sale_date > :date",
    parameters={"date": "2024-01-01"},
    batch_size=2000,
)

Both methods build escaped SQL references, stream results to manage memory, and wrap output in a DatabricksData object defined at lines 64-76.

Step 5: Convert to Semantica Documents

Use export_as_documents() to transform rows into the generic document format expected by downstream modules:

documents = ingestor.export_as_documents(
    orders,
    id_field="order_id",
    text_fields=["customer_name", "product_name", "notes"],
)

This method is implemented at lines 75-84 and prepares data for chunking, embedding, and graph construction.

Advanced Features: Schema Discovery and Lineage

Query Unity Catalog Metadata

Retrieve table schemas without ingestion using get_table_schema():

schema = ingestor.get_table_schema("customers")

List catalogs, schemas, and tables through the underlying connector methods.

Extract Data Lineage

The get_table_lineage() method at lines 124-138 retrieves upstream and downstream dependencies:

lineage = ingestor.get_table_lineage(
    table_name="customers",
    catalog="marketing",
    schema="raw",
    include_column_lineage=True,
)

print("Upstream tables:", lineage["upstream"])
print("Downstream tables:", lineage["downstream"])

This provenance enriches knowledge-graph nodes with source tracking, visualized downstream in modules like semantica/visualization/visualization_provenance.py.

Progress Tracking and Error Handling

Every ingestion step wraps the internal self.progress_tracker for real-time status updates. The tracker starts at self.progress_tracker.start_tracking() within ingest_table() at lines 61-66, providing:

• Streaming progress for large tables
• Batch-level error recovery
• Structured exceptions via ProcessingError and ValidationError from semantica/utils/exceptions.py

Complete Integration Example: OAuth M2M with Lineage

from semantica.ingest import DatabricksIngestor

# OAuth M2M authentication

ingestor = DatabricksIngestor(
    host="https://adb-98765.azuredatabricks.net",
    client_id="abcd-1234-efgh-5678",
    client_secret="super-secret-value",
    http_path="/sql/1.0/warehouses/9999-8888-7777",
)

# Parameterized query with batching

results = ingestor.ingest_query(
    "SELECT * FROM analytics.sales WHERE sale_date > :date",
    parameters={"date": "2024-01-01"},
    batch_size=2000,
)

# Export with auto-detected text fields

documents = ingestor.export_as_documents(results)

# Enrich with lineage metadata

lineage = ingestor.get_table_lineage(
    table_name="sales",
    catalog="analytics",
    include_column_lineage=True,
)

Summary

• Secure authentication: PAT or OAuth M2M via environment variables or explicit parameters in DatabricksIngestor
• Flexible ingestion: ingest_table() for full Delta tables, ingest_query() for arbitrary SQL with parameterization
• Memory-efficient streaming: Results batched and wrapped in DatabricksData objects
• Native document conversion: export_as_documents() bridges to Semantica's pipeline
• Metadata enrichment: Unity Catalog schema discovery and lineage extraction built-in
• Production observability: Integrated progress tracking and structured error handling

Frequently Asked Questions

What authentication methods does Semantica's Databricks integration support?

Semantica supports Personal Access Tokens (PAT) and OAuth M2M authentication. PAT uses a single token string, while OAuth M2M requires a client ID and client secret pair. Both methods authenticate against SQL warehouses or clusters via the databricks-sql-connector.

How does Semantica handle large Delta tables without running out of memory?

The ingest_table() and ingest_query() methods stream results in configurable batches. The batch_size parameter controls row chunks, and DatabricksData holds metadata separately from row data. This design prevents full table materialization in memory.

Can I use Semantica with Databricks without installing all dependencies?

Yes. The databricks-sql-connector and databricks-sdk are lazy imports at lines 25-30 of databricks_ingestor.py. If absent, you receive a clear ImportError with the exact pip install command needed, keeping the base Semantica installation lightweight.

What provenance information is captured during Databricks ingestion?

Every DatabricksData object tracks source catalog, schema, table name, query text, and execution timestamp. When combined with get_table_lineage(), you capture upstream/downstream table dependencies and column-level lineage for full dataflow visibility in your knowledge graph.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}