How to Integrate Semantica with Databricks or Snowflake: A Complete Guide to Data Warehouse Ingestors

Semantica provides dedicated ingestors for Databricks and Snowflake that expose a unified Python API to authenticate, query, and convert warehouse data into Document objects for downstream LLM and knowledge-graph pipelines.

Semantica is an open-source framework for building knowledge graphs and LLM pipelines. To integrate Semantica with Databricks or Snowflake, you use the platform-specific ingestor classes—DatabricksIngestor and SnowflakeIngestor—which wrap the native SDKs and handle authentication, pagination, and schema introspection.

Prerequisites and Installation

Before using the warehouse integrations, install the required SDK packages in the same environment as Semantica. You only need to install the packages for the platform you intend to use.

Platform	Required Python Packages
Databricks	databricks-sdk and databricks-sql-connector
Snowflake	snowflake-connector-python

Install them using pip:

# For Databricks Unity Catalog and Delta Lake

pip install databricks-sdk databricks-sql-connector

# For Snowflake

pip install snowflake-connector-python

Configuration: Environment Variables vs. Constructor Arguments

Both ingestors read connection details from environment variables by default, but you can override them by passing explicit arguments to the constructor. Store credentials outside source control (e.g., in a .env file loaded with python-dotenv).

Variable	Databricks	Snowflake
Host / Account	DATABRICKS_HOST	SNOWFLAKE_ACCOUNT (e.g., myorg.snowflakecomputing.com)
Token / Password	DATABRICKS_TOKEN	SNOWFLAKE_PASSWORD
OAuth Client ID	DATABRICKS_CLIENT_ID	SNOWFLAKE_CLIENT_ID
OAuth Client Secret	DATABRICKS_CLIENT_SECRET	SNOWFLAKE_CLIENT_SECRET
HTTP Path (Databricks only)	DATABRICKS_HTTP_PATH	—
Default Catalog / Database	DATABRICKS_CATALOG	SNOWFLAKE_DATABASE
Default Schema	DATABRICKS_SCHEMA	SNOWFLAKE_SCHEMA

Ingesting Data from Databricks

The DatabricksIngestor class in semantica/ingest/databricks_ingestor.py wraps the DatabricksConnector to manage connections to Unity Catalog and Delta Lake. It supports both personal access tokens and OAuth M2M authentication.

Instantiate the ingestor and pull entire tables or execute arbitrary SQL with automatic pagination:

from semantica.ingest import DatabricksIngestor

# Constructor-based configuration

ingestor = DatabricksIngestor(
    host="https://adb-123.azuredatabricks.net",
    token="dapi-xxxxxxxxxxxx",
    http_path="/sql/1.0/warehouses/xxxxxxxx",
    catalog="analytics",
    schema="sales",
)

# Ingest a whole table with an optional row limit

data = ingestor.ingest_table("customers", limit=5000)

# Execute a parameterized query with batching for large results

query = "SELECT * FROM sales_revenue WHERE YEAR = %(year)s"
query_data = ingestor.ingest_query(
    query, 
    parameters={"year": 2024}, 
    batch_size=10_000
)

Use the context manager protocol to ensure connections close automatically:

with DatabricksIngestor(...) as ingestor:
    data = ingestor.ingest_table("events")
    docs = ingestor.export_as_documents(data)

# Connection closed automatically when exiting the block

Ingesting Data from Snowflake

The SnowflakeIngestor class in semantica/ingest/snowflake_ingestor.py provides an identical API surface for Snowflake, wrapping snowflake-connector-python. It accepts password-based or OAuth-based authentication.

from semantica.ingest import SnowflakeIngestor

# Initialize with explicit credentials

sf_ingestor = SnowflakeIngestor(
    account="myorg.snowflakecomputing.com",
    user="etl_user",
    password="my_secret_password",  # Or use OAuth client_id/secret

    database="ANALYTICS",
    schema="SALES",
)

# Ingest with optional WHERE clause filtering

sf_data = sf_ingestor.ingest_table(
    "ORDERS", 
    where="ORDER_DATE > '2024-01-01'"
)

Converting Warehouse Data to Semantica Documents

Both ingestors transform result sets into platform-specific data models—DatabricksData and SnowflakeData—which contain lists of row dictionaries, schema metadata, and tracking information. To feed this data into LLM or vector-store pipelines, use the export_as_documents() method, which maps database rows to Semantica’s Document format.

# Databricks: Convert to documents specifying which fields contain text

db_docs = ingestor.export_as_documents(
    data, 
    text_fields=["name", "notes"]
)

# Snowflake: Same API for uniform downstream processing

sf_docs = sf_ingestor.export_as_documents(
    sf_data, 
    text_fields=["product_name", "description"]
)

The resulting document list contains dictionaries with id, text, and metadata keys, ready for ingestion into Semantica’s knowledge graph components or external vector databases.

Advanced Features and Schema Introspection

Both ingestors expose methods for metadata discovery and progress tracking, implemented in semantica/ingest/databricks_ingestor.py and semantica/ingest/snowflake_ingestor.py, and integrated with semantica/utils/progress_tracker.py.

Catalog and Table Discovery:

• Databricks: list_catalogs(), list_schemas(), list_tables()
• Snowflake: list_databases(), list_schemas(), list_tables()

Schema Introspection: Call get_table_schema() on either ingestor to retrieve column definitions and data types before querying.

Lineage Tracking (Databricks Only): For tables registered in Unity Catalog, invoke get_table_lineage(include_column_lineage=True) to retrieve upstream and downstream dependencies. Snowflake lineage is not currently exposed but can be added by extending SnowflakeIngestor with calls to Snowflake’s Query History API.

Custom Connection Configuration: Pass additional keyword arguments to the underlying SDK via the config parameter. For Databricks, these are forwarded to databricks_sql.connect(); for Snowflake, to snowflake.connector.connect().

Summary

• Semantica ingestors for Databricks and Snowflake live in semantica/ingest/databricks_ingestor.py and semantica/ingest/snowflake_ingestor.py, respectively.
• Authentication supports both token-based and OAuth M2M flows via environment variables or constructor arguments.
• Data ingestion uses ingest_table() for full tables or ingest_query() for custom SQL, with optional result limiting and batching.
• Document conversion via export_as_documents() standardizes warehouse rows into the id, text, metadata format required by downstream LLM pipelines.
• Progress tracking is automatic through semantica/utils/progress_tracker.py, logging each connection, query, and fetch operation.

Frequently Asked Questions

Do I need to install connectors for both Databricks and Snowflake?

No. Install only the package for the platform you use. Databricks requires databricks-sdk and databricks-sql-connector, while Snowflake requires snowflake-connector-python. The semantica/ingest/registry.py dynamically registers only the ingestors whose dependencies are present.

Can I switch between authentication methods without changing code?

Yes. The ingestors automatically detect credentials based on environment variables. Set DATABRICKS_TOKEN or SNOWFLAKE_PASSWORD for personal access token authentication, or set *CLIENT_ID and *CLIENT_SECRET for OAuth M2M. The constructor picks the appropriate method without requiring code changes.

How does the ingestor handle huge tables that do not fit in memory?

Both ingestors support chunked retrieval. In ingest_query(), specify batch_size to control page size. For Databricks, use limit and offset parameters in ingest_table(). The ProgressTracker updates after each batch, allowing you to monitor ingestion progress via the shared logging utility in semantica/utils/progress_tracker.py.

Does Semantica capture data lineage for Snowflake?

Currently, lineage extraction is only implemented for Databricks Unity Catalog via get_table_lineage(). Snowflake lineage can be manually implemented by subclassing SnowflakeIngestor and querying Snowflake’s Query History or Account Usage views, then mapping the results to Semantica’s internal lineage format.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}