# How to Integrate Semantica with Databricks or Snowflake: A Complete Guide to Data Warehouse Ingestors

> Seamlessly integrate Semantica with Databricks or Snowflake. Our guide shows you how to use Python APIs to query warehouse data and convert it into Document objects for LLM and knowledge-graph pipelines.

- Repository: [Semantica /semantica](https://github.com/semantica-agi/semantica)
- Tags: how-to-guide
- Published: 2026-08-22

---

**Semantica provides dedicated ingestors for Databricks and Snowflake that expose a unified Python API to authenticate, query, and convert warehouse data into `Document` objects for downstream LLM and knowledge-graph pipelines.**

Semantica is an open-source framework for building knowledge graphs and LLM pipelines. To integrate Semantica with Databricks or Snowflake, you use the platform-specific ingestor classes—`DatabricksIngestor` and `SnowflakeIngestor`—which wrap the native SDKs and handle authentication, pagination, and schema introspection.

## Prerequisites and Installation

Before using the warehouse integrations, install the required SDK packages in the same environment as Semantica. You only need to install the packages for the platform you intend to use.

| Platform | Required Python Packages |
|----------|--------------------------|
| **Databricks** | `databricks-sdk` and `databricks-sql-connector` |
| **Snowflake** | `snowflake-connector-python` |

Install them using pip:

```bash

# For Databricks Unity Catalog and Delta Lake

pip install databricks-sdk databricks-sql-connector

# For Snowflake

pip install snowflake-connector-python

```

## Configuration: Environment Variables vs. Constructor Arguments

Both ingestors read connection details from **environment variables** by default, but you can override them by passing explicit arguments to the constructor. Store credentials outside source control (e.g., in a `.env` file loaded with `python-dotenv`).

| Variable | Databricks | Snowflake |
|----------|------------|-----------|
| **Host / Account** | `DATABRICKS_HOST` | `SNOWFLAKE_ACCOUNT` (e.g., `myorg.snowflakecomputing.com`) |
| **Token / Password** | `DATABRICKS_TOKEN` | `SNOWFLAKE_PASSWORD` |
| **OAuth Client ID** | `DATABRICKS_CLIENT_ID` | `SNOWFLAKE_CLIENT_ID` |
| **OAuth Client Secret** | `DATABRICKS_CLIENT_SECRET` | `SNOWFLAKE_CLIENT_SECRET` |
| **HTTP Path** (Databricks only) | `DATABRICKS_HTTP_PATH` | — |
| **Default Catalog / Database** | `DATABRICKS_CATALOG` | `SNOWFLAKE_DATABASE` |
| **Default Schema** | `DATABRICKS_SCHEMA` | `SNOWFLAKE_SCHEMA` |

## Ingesting Data from Databricks

The `DatabricksIngestor` class in [`semantica/ingest/databricks_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/databricks_ingestor.py) wraps the `DatabricksConnector` to manage connections to Unity Catalog and Delta Lake. It supports both personal access tokens and OAuth M2M authentication.

Instantiate the ingestor and pull entire tables or execute arbitrary SQL with automatic pagination:

```python
from semantica.ingest import DatabricksIngestor

# Constructor-based configuration

ingestor = DatabricksIngestor(
    host="https://adb-123.azuredatabricks.net",
    token="dapi-xxxxxxxxxxxx",
    http_path="/sql/1.0/warehouses/xxxxxxxx",
    catalog="analytics",
    schema="sales",
)

# Ingest a whole table with an optional row limit

data = ingestor.ingest_table("customers", limit=5000)

# Execute a parameterized query with batching for large results

query = "SELECT * FROM sales_revenue WHERE YEAR = %(year)s"
query_data = ingestor.ingest_query(
    query, 
    parameters={"year": 2024}, 
    batch_size=10_000
)

```

Use the context manager protocol to ensure connections close automatically:

```python
with DatabricksIngestor(...) as ingestor:
    data = ingestor.ingest_table("events")
    docs = ingestor.export_as_documents(data)

# Connection closed automatically when exiting the block

```

## Ingesting Data from Snowflake

The `SnowflakeIngestor` class in [`semantica/ingest/snowflake_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/snowflake_ingestor.py) provides an identical API surface for Snowflake, wrapping `snowflake-connector-python`. It accepts password-based or OAuth-based authentication.

```python
from semantica.ingest import SnowflakeIngestor

# Initialize with explicit credentials

sf_ingestor = SnowflakeIngestor(
    account="myorg.snowflakecomputing.com",
    user="etl_user",
    password="my_secret_password",  # Or use OAuth client_id/secret

    database="ANALYTICS",
    schema="SALES",
)

# Ingest with optional WHERE clause filtering

sf_data = sf_ingestor.ingest_table(
    "ORDERS", 
    where="ORDER_DATE > '2024-01-01'"
)

```

## Converting Warehouse Data to Semantica Documents

Both ingestors transform result sets into platform-specific data models—`DatabricksData` and `SnowflakeData`—which contain lists of row dictionaries, schema metadata, and tracking information. To feed this data into LLM or vector-store pipelines, use the `export_as_documents()` method, which maps database rows to Semantica’s `Document` format.

```python

# Databricks: Convert to documents specifying which fields contain text

db_docs = ingestor.export_as_documents(
    data, 
    text_fields=["name", "notes"]
)

# Snowflake: Same API for uniform downstream processing

sf_docs = sf_ingestor.export_as_documents(
    sf_data, 
    text_fields=["product_name", "description"]
)

```

The resulting document list contains dictionaries with `id`, `text`, and `metadata` keys, ready for ingestion into Semantica’s knowledge graph components or external vector databases.

## Advanced Features and Schema Introspection

Both ingestors expose methods for metadata discovery and progress tracking, implemented in [`semantica/ingest/databricks_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/databricks_ingestor.py) and [`semantica/ingest/snowflake_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/snowflake_ingestor.py), and integrated with [`semantica/utils/progress_tracker.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/utils/progress_tracker.py).

**Catalog and Table Discovery:**

- **Databricks**: `list_catalogs()`, `list_schemas()`, `list_tables()`
- **Snowflake**: `list_databases()`, `list_schemas()`, `list_tables()`

**Schema Introspection:**
Call `get_table_schema()` on either ingestor to retrieve column definitions and data types before querying.

**Lineage Tracking (Databricks Only):**
For tables registered in Unity Catalog, invoke `get_table_lineage(include_column_lineage=True)` to retrieve upstream and downstream dependencies. Snowflake lineage is not currently exposed but can be added by extending `SnowflakeIngestor` with calls to Snowflake’s Query History API.

**Custom Connection Configuration:**
Pass additional keyword arguments to the underlying SDK via the `config` parameter. For Databricks, these are forwarded to `databricks_sql.connect()`; for Snowflake, to `snowflake.connector.connect()`.

## Summary

- **Semantica ingestors** for Databricks and Snowflake live in [`semantica/ingest/databricks_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/databricks_ingestor.py) and [`semantica/ingest/snowflake_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/snowflake_ingestor.py), respectively.
- **Authentication** supports both token-based and OAuth M2M flows via environment variables or constructor arguments.
- **Data ingestion** uses `ingest_table()` for full tables or `ingest_query()` for custom SQL, with optional result limiting and batching.
- **Document conversion** via `export_as_documents()` standardizes warehouse rows into the `id`, `text`, `metadata` format required by downstream LLM pipelines.
- **Progress tracking** is automatic through [`semantica/utils/progress_tracker.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/utils/progress_tracker.py), logging each connection, query, and fetch operation.

## Frequently Asked Questions

### Do I need to install connectors for both Databricks and Snowflake?

No. Install only the package for the platform you use. Databricks requires `databricks-sdk` and `databricks-sql-connector`, while Snowflake requires `snowflake-connector-python`. The [`semantica/ingest/registry.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/registry.py) dynamically registers only the ingestors whose dependencies are present.

### Can I switch between authentication methods without changing code?

Yes. The ingestors automatically detect credentials based on environment variables. Set `DATABRICKS_TOKEN` or `SNOWFLAKE_PASSWORD` for personal access token authentication, or set `*CLIENT_ID` and `*CLIENT_SECRET` for OAuth M2M. The constructor picks the appropriate method without requiring code changes.

### How does the ingestor handle huge tables that do not fit in memory?

Both ingestors support chunked retrieval. In `ingest_query()`, specify `batch_size` to control page size. For Databricks, use `limit` and `offset` parameters in `ingest_table()`. The `ProgressTracker` updates after each batch, allowing you to monitor ingestion progress via the shared logging utility in [`semantica/utils/progress_tracker.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/utils/progress_tracker.py).

### Does Semantica capture data lineage for Snowflake?

Currently, lineage extraction is only implemented for Databricks Unity Catalog via `get_table_lineage()`. Snowflake lineage can be manually implemented by subclassing `SnowflakeIngestor` and querying Snowflake’s Query History or Account Usage views, then mapping the results to Semantica’s internal lineage format.