Data Flow from Databricks and Snowflake Ingestors to Semantica's Knowledge Graph: A Complete Guide

The data flow from Databricks/Snowflake ingestors to Semantica's knowledge graph follows five stages: Connect → Ingest → Normalise → Build KG → Store/Use, with ingestors converting raw database rows into structured documents before entity and relationship extraction.

Understanding how Semantica transforms warehouse data into actionable knowledge graphs is essential for building production-ready data pipelines. This guide traces the complete data flow from Databricks and Snowflake ingestors to Semantica's knowledge graph, using the actual source code from the semantica-agi/semantica repository.

Overview of the Ingestion Pipeline

The data flow architecture consists of five distinct stages:

Stage	Component	Function
Connect	DatabricksConnector / SnowflakeConnector	Establishes authenticated connections using PAT or OAuth M2M credentials
Ingest	DatabricksIngestor / SnowflakeIngestor	Executes table scans, custom queries, or metadata calls; streams results into dataclasses
Normalise	export_as_documents()	Converts raw rows into standardized document dictionaries with content, metadata, and id fields
Build KG	semantica.kg.KnowledgeGraph	Extracts entities, relationships, and metadata through the built-in extraction pipeline
Store/Use	KnowledgeGraph instance	Provides in-memory graph storage for persistence, visualization, or downstream querying

Stage 1: Establishing Connections

Both ingestors support industry-standard authentication mechanisms. The Databricks connector in semantica/ingest/databricks_ingestor.py validates required parameters during initialization (lines 25-49):

from semantica.ingest import DatabricksIngestor

ingestor = DatabricksIngestor(
    host="https://adb-123.azuredatabricks.net",
    token="dapi-xxxxxxxx",
    http_path="/sql/1.0/warehouses/abcd",
    catalog="main",
    schema="default",
)

Snowflake follows an identical pattern via SnowflakeIngestor in semantica/ingest/snowflake_ingestor.py. The connector layer handles connection pooling, credential validation, and warehouse-specific protocol adaptations.

Stage 2: Data Ingestion

The ingest_table() method streams rows efficiently with optional limits. In semantica/ingest/databricks_ingestor.py (lines 64-78):

data = ingestor.ingest_table("customers", limit=10_000)

This returns a DatabricksData dataclass containing:

• data: List of row dictionaries
• row_count: Total rows ingested
• columns: Column names and types
• catalog, schema, table_name: Source metadata
• ingested_at: Timestamp

The Snowflake equivalent uses SnowflakeData with identical structure.

Stage 3: Document Normalization

The export_as_documents() method—implemented on both ingestors—transforms raw warehouse data into Semantica's canonical document format:

documents = ingestor.export_as_documents(data)

Each document follows this structure:

{
    "content": json.dumps(row),  # Serialized row data

    "metadata": {
        "source": f"{data.catalog}.{data.schema}.{data.table_name}",
        "ingested_at": data.ingested_at.isoformat(),
        "query": data.query if hasattr(data, "query") else None,
    },
    "id": optional_unique_identifier,
}

This normalization decouples warehouse-specific schemas from downstream processing, enabling unified handling regardless of source system.

Stage 4: Knowledge Graph Construction

The KnowledgeGraph class in semantica/kg/knowledge_graph.py (lines 2-24) serves as the canonical in-memory representation:

from semantica.kg import KnowledgeGraph

kg = KnowledgeGraph()
kg.add_documents(documents)  # Triggers entity/relationship extraction

The KG builder performs three core operations:

• Entity extraction: Identifies and deduplicates named entities from document content
• Relationship inference: Maps connections between entities using the built-in NLP pipeline
• Metadata aggregation: Preserves provenance information for traceability

The resulting structure contains:

• entities: Dictionary of extracted entities with attributes
• relationships: List of (source, relation, target) triples
• metadata: Global ingestion parameters and statistics

Stage 5: Downstream Consumption

Knowledge graph instances integrate directly with visualization and retrieval components. The KGVisualizer in semantica/visualization/kg_visualizer.py (lines 132-166) accepts KnowledgeGraph objects:

from semantica.visualization import KGVisualizer

viz = KGVisualizer()
viz.visualize_network(kg, output="interactive")

The visualizer internally calls _convert_knowledge_graph() to transform the dataclass into rendering-appropriate formats.

Complete End-to-End Example

# Databricks pipeline (Snowflake uses identical structure)

from semantica.ingest import DatabricksIngestor
from semantica.kg import KnowledgeGraph
from semantica.visualization import KGVisualizer

# 1. Connect and ingest

ingestor = DatabricksIngestor(
    host="https://adb-123.azuredatabricks.net",
    token="dapi-xxxxxxxx",
    http_path="/sql/1.0/warehouses/abcd",
    catalog="sales",
    schema="public",
)
raw_data = ingestor.ingest_table("orders", limit=5_000)

# 2. Normalize to documents

documents = ingestor.export_as_documents(raw_data)

# 3. Build knowledge graph

kg = KnowledgeGraph()
kg.add_documents(documents)

# 4. Visualize or integrate with retrieval systems

visualizer = KGVisualizer()
visualizer.visualize_network(kg, output="interactive")

The Snowflake variant requires only substituting SnowflakeIngestor and Snowflake-specific connection parameters.

Key Source Files and Architecture

File	Purpose
semantica/ingest/databricks_ingestor.py	Databricks connector, ingestion logic, and DatabricksData dataclass
semantica/ingest/snowflake_ingestor.py	Snowflake connector and SnowflakeData dataclass
semantica/kg/knowledge_graph.py	Core KnowledgeGraph dataclass with entity/relationship storage
semantica/visualization/kg_visualizer.py	Network visualization with _convert_knowledge_graph() helper
semantica/ingest/__init__.py	Public API exports for DatabricksIngestor and SnowflakeIngestor

Summary

• Connection abstraction: Both ingestors use native drivers with PAT/OAuth authentication, validated in __init__ methods
• Streaming ingestion: ingest_table() and custom query methods return rich dataclasses preserving source metadata
• Document normalization: export_as_documents() creates portable dictionaries decoupled from warehouse schemas
• Unified KG building: KnowledgeGraph.add_documents() extracts entities and relationships regardless of source system
• Composable downstream: The in-memory graph format integrates with visualization, vector stores, and agent systems

Frequently Asked Questions

What authentication methods does Semantica support for Databricks and Snowflake?

Semantica supports personal access tokens (PAT) and OAuth M2M credentials for both platforms. The Databricks connector validates these in DatabricksConnector.__init__ (semantica/ingest/databricks_ingestor.py lines 25-49), while Snowflake uses equivalent parameter validation. OAuth flows are handled natively through the underlying Python connectors.

How does Semantica handle large table ingestion without memory issues?

The ingest_table() method accepts a limit parameter for bounded ingestion, and code analysis shows streaming row processing rather than full materialization. For production workloads, implement chunked ingestion using SQL LIMIT/OFFSET or partition-based scanning, then iteratively call KnowledgeGraph.add_documents() to build the graph incrementally.

Can I use custom SQL queries instead of full table scans?

Yes. The ingestor architecture supports custom queries—DatabricksData and SnowflakeData dataclasses include an optional query field in metadata that captures the source SQL. Call ingest_query() (implied by the query metadata handling in export_as_documents) or extend the base ingestor pattern to inject arbitrary SQL before document normalization.

What entity extraction model powers the KnowledgeGraph builder?

The source code in semantica/kg/knowledge_graph.py implements a built-in entity-extraction pipeline invoked during add_documents(). While the specific model architecture isn't detailed in the analyzed files, the extraction produces standardized entities and relationships structures suitable for downstream consumption by vector stores and visualization tools.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}