Semantica Authentication Options for Databricks and Snowflake Enterprise Data Platforms

Semantica provides native support for Personal Access Tokens and OAuth M2M authentication in Databricks, alongside password, key-pair, OAuth, and SSO methods for Snowflake, enabling secure, flexible connectivity to enterprise data platforms.

The Semantica open-source framework (available at semantica-agi/semantica) abstracts connection logic for major data warehouses through a unified ingestion layer. Understanding the specific authentication options for enterprise data platforms implemented in the codebase allows developers to select the appropriate security model for their operational requirements. The following sections detail the exact mechanisms available in the Databricks and Snowflake connector implementations.

Databricks Authentication Methods

The DatabricksConnector class, defined in semantica/ingest/databricks_ingestor.py, supports two enterprise-grade authentication strategies. The constructor validates credential parameters at lines L50-L57, ensuring that either a Personal Access Token or OAuth M2M credentials are properly supplied before establishing connections.

Personal Access Token (PAT)

Personal Access Tokens provide direct authentication using tokens generated within the Databricks workspace UI (formatted as dapi-…). This method is ideal for individual user accounts and automated service accounts requiring straightforward token-based access.

When initializing a DatabricksIngestor (which utilizes the underlying DatabricksConnector), the token is passed directly to the databricks-sql-connector library:

from semantica.ingest import DatabricksIngestor

ingestor = DatabricksIngestor(
    host="https://adb-123.azuredatabricks.net",
    token="dapi-XXXXXXXXXXXXXXXX",
    http_path="/sql/1.0/warehouses/XXXXXXXXXXXX",
)
data = ingestor.ingest_table("sales")
documents = ingestor.export_as_documents(data, text_fields=["product_name", "description"])

OAuth M2M (Service Principal)

For production environments requiring service-to-service authentication, Semantica implements OAuth Machine-to-Machine (M2M) flows using client credentials. This approach requires a client_id and client_secret generated from a Databricks service principal configuration.

The source code constructs a credentials_provider callable (lines L106-L124 in databricks_ingestor.py) that interfaces with databricks.sdk.core.oauth_service_principal. This provider manages token lifecycle operations automatically, ensuring long-running ingestion jobs maintain valid credentials without manual intervention:

from semantica.ingest import DatabricksIngestor

ingestor = DatabricksIngestor(
    host="https://adb-123.azuredatabricks.net",
    client_id="my-client-id",
    client_secret="my-client-secret",
    http_path="/sql/1.0/warehouses/XXXXXXXXXXXX",
)
data = ingestor.ingest_query("SELECT * FROM analytics.transactions")

Snowflake Authentication Methods

The SnowflakeConnector class in semantica/ingest/snowflake_ingestor.py offers four distinct authentication mechanisms. The connect method (lines L56-L76) dynamically constructs connection parameters based on the specified authenticator type, supporting everything from basic password auth to enterprise SSO flows.

Password Authentication

The default authentication method uses standard username and password credentials. When no specific authenticator parameter is provided, the connector validates the presence of a password (lines L70-L76) and passes these credentials directly to the Snowflake Python connector:

from semantica.ingest import SnowflakeIngestor

ingestor = SnowflakeIngestor(
    account="myaccount",
    user="myuser",
    password="mypassword",
    warehouse="COMPUTE_WH",
    database="MYDB",
    schema="PUBLIC",
)
data = ingestor.ingest_table("CUSTOMERS", limit=5000)

Key-Pair Authentication

For organizations requiring certificate-based security, Semantica supports key-pair authentication using RSA private keys. Users may specify a private_key_path to a PEM-encoded file or provide key bytes directly, with optional private_key_passphrase for encrypted keys.

The implementation processes these parameters at lines L66-L69, passing the decrypted private key bytes to Snowflake's connector:

from semantica.ingest import SnowflakeIngestor

ingestor = SnowflakeIngestor(
    account="myaccount",
    user="myuser",
    private_key_path="/path/to/rsa_private_key.pem",
    private_key_passphrase="my_passphrase",
    warehouse="COMPUTE_WH",
    database="MYDB",
)
data = ingestor.ingest_table("ORDERS")

OAuth Token

OAuth token-based authentication enables integration with external identity providers. When authenticator="oauth" is specified, the connector requires a valid token parameter (validated at lines L60-L66) and includes it in the connection configuration:

from semantica.ingest import SnowflakeIngestor

ingestor = SnowflakeIngestor(
    account="myaccount",
    user="myuser",
    authenticator="oauth",
    token="my-oauth-token",
    warehouse="COMPUTE_WH",
    database="MYDB",
)
data = ingestor.ingest_query(
    "SELECT * FROM SALES WHERE SALE_DATE > %(date)s",
    params={"date": "2024-01-01"},
)

SSO via External Browser

For interactive development environments, Semantica supports single sign-on through an external browser flow. Setting authenticator="externalbrowser" leverages Snowflake's externalbrowser authenticator, initiating a browser-based SSO handshake that allows users to authenticate through corporate identity providers without storing credentials in source code.

Summary

• Databricks supports Personal Access Tokens (PAT) for simple token-based auth and OAuth M2M (client credentials flow) for service principal authentication, with the latter implemented via a credentials_provider callable in databricks_ingestor.py.

• Snowflake offers four methods: password authentication (default), key-pair using RSA private keys, OAuth token passthrough, and external browser SSO, all handled in the connect method of snowflake_ingestor.py.

• Both connectors validate required credential combinations during initialization, raising errors before attempting network connections if required parameters are missing.

• The framework delegates underlying protocol handling to the official Databricks and Snowflake Python SDKs while providing a unified interface for data ingestion workflows.

Frequently Asked Questions

Which authentication method should I use for production Databricks workloads?

For production and automated ETL pipelines, OAuth M2M (service principal) authentication is recommended. According to the Semantica source code in databricks_ingestor.py, this method uses a credentials_provider (lines L106-L124) that handles token refresh automatically through the Databricks SDK, eliminating the need to manually rotate Personal Access Tokens.

How does Semantica validate Snowflake authentication parameters?

The SnowflakeConnector.__init__ method parses the authenticator argument, while the connect method (lines L56-L76 in snowflake_ingestor.py) enforces specific validation rules: OAuth requires a token, key-pair requires private key bytes, and password mode requires both username and password. These checks occur before establishing the database connection.

Can I use AWS IAM or Azure AD authentication with Semantica?

Based on the current implementation in semantica/semantica/ingest/databricks_ingestor.py and snowflake_ingestor.py, the framework natively supports OAuth M2M for Databricks (which can integrate with Azure AD) and OAuth tokens or SSO for Snowflake. Direct AWS IAM authentication for Snowflake is not explicitly implemented in the analyzed source files; users should utilize OAuth tokens obtained through their identity provider instead.

Where are the authentication logic and validation implemented in the codebase?

The core authentication logic resides in two files: semantica/ingest/databricks_ingestor.py (containing DatabricksConnector with validation at lines L50-L57 and OAuth handling at L106-L124) and semantica/ingest/snowflake_ingestor.py (containing SnowflakeConnector with the connect method implementing authentication routing at lines L56-L76).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}