# Data Flow from Databricks and Snowflake Ingestors to Semantica's Knowledge Graph: A Complete Guide

> Understand the data flow from Databricks and Snowflake ingestors to Semantica's knowledge graph. Learn the 5 stages: Connect, Ingest, Normalise, Build KG, and Store/Use.

- Repository: [Semantica /semantica](https://github.com/semantica-agi/semantica)
- Tags: how-to-guide
- Published: 2026-08-20

---

**The data flow from Databricks/Snowflake ingestors to Semantica's knowledge graph follows five stages: Connect → Ingest → Normalise → Build KG → Store/Use, with ingestors converting raw database rows into structured documents before entity and relationship extraction.**

Understanding how Semantica transforms warehouse data into actionable knowledge graphs is essential for building production-ready data pipelines. This guide traces the complete data flow from Databricks and Snowflake ingestors to Semantica's knowledge graph, using the actual source code from the `semantica-agi/semantica` repository.

## Overview of the Ingestion Pipeline

The data flow architecture consists of five distinct stages:

| Stage | Component | Function |
|-------|-----------|----------|
| **Connect** | `DatabricksConnector` / `SnowflakeConnector` | Establishes authenticated connections using PAT or OAuth M2M credentials |
| **Ingest** | `DatabricksIngestor` / `SnowflakeIngestor` | Executes table scans, custom queries, or metadata calls; streams results into dataclasses |
| **Normalise** | `export_as_documents()` | Converts raw rows into standardized document dictionaries with `content`, `metadata`, and `id` fields |
| **Build KG** | `semantica.kg.KnowledgeGraph` | Extracts entities, relationships, and metadata through the built-in extraction pipeline |
| **Store/Use** | `KnowledgeGraph` instance | Provides in-memory graph storage for persistence, visualization, or downstream querying |

## Stage 1: Establishing Connections

Both ingestors support industry-standard authentication mechanisms. The **Databricks connector** in [`semantica/ingest/databricks_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/databricks_ingestor.py) validates required parameters during initialization (`lines 25-49`):

```python
from semantica.ingest import DatabricksIngestor

ingestor = DatabricksIngestor(
    host="https://adb-123.azuredatabricks.net",
    token="dapi-xxxxxxxx",
    http_path="/sql/1.0/warehouses/abcd",
    catalog="main",
    schema="default",
)

```

**Snowflake** follows an identical pattern via `SnowflakeIngestor` in [`semantica/ingest/snowflake_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/snowflake_ingestor.py). The connector layer handles connection pooling, credential validation, and warehouse-specific protocol adaptations.

## Stage 2: Data Ingestion

The `ingest_table()` method streams rows efficiently with optional limits. In [`semantica/ingest/databricks_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/databricks_ingestor.py) (`lines 64-78`):

```python
data = ingestor.ingest_table("customers", limit=10_000)

```

This returns a **`DatabricksData`** dataclass containing:
- `data`: List of row dictionaries
- `row_count`: Total rows ingested
- `columns`: Column names and types
- `catalog`, `schema`, `table_name`: Source metadata
- `ingested_at`: Timestamp

The **Snowflake equivalent** uses `SnowflakeData` with identical structure.

## Stage 3: Document Normalization

The `export_as_documents()` method—implemented on both ingestors—transforms raw warehouse data into Semantica's canonical document format:

```python
documents = ingestor.export_as_documents(data)

```

Each document follows this structure:

```python
{
    "content": json.dumps(row),  # Serialized row data

    "metadata": {
        "source": f"{data.catalog}.{data.schema}.{data.table_name}",
        "ingested_at": data.ingested_at.isoformat(),
        "query": data.query if hasattr(data, "query") else None,
    },
    "id": optional_unique_identifier,
}

```

This normalization decouples warehouse-specific schemas from downstream processing, enabling unified handling regardless of source system.

## Stage 4: Knowledge Graph Construction

The `KnowledgeGraph` class in [`semantica/kg/knowledge_graph.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/kg/knowledge_graph.py) (`lines 2-24`) serves as the canonical in-memory representation:

```python
from semantica.kg import KnowledgeGraph

kg = KnowledgeGraph()
kg.add_documents(documents)  # Triggers entity/relationship extraction

```

The **KG builder** performs three core operations:
- **Entity extraction**: Identifies and deduplicates named entities from document content
- **Relationship inference**: Maps connections between entities using the built-in NLP pipeline
- **Metadata aggregation**: Preserves provenance information for traceability

The resulting structure contains:
- `entities`: Dictionary of extracted entities with attributes
- `relationships`: List of `(source, relation, target)` triples
- `metadata`: Global ingestion parameters and statistics

## Stage 5: Downstream Consumption

Knowledge graph instances integrate directly with visualization and retrieval components. The `KGVisualizer` in [`semantica/visualization/kg_visualizer.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/visualization/kg_visualizer.py) (`lines 132-166`) accepts `KnowledgeGraph` objects:

```python
from semantica.visualization import KGVisualizer

viz = KGVisualizer()
viz.visualize_network(kg, output="interactive")

```

The visualizer internally calls `_convert_knowledge_graph()` to transform the dataclass into rendering-appropriate formats.

## Complete End-to-End Example

```python

# Databricks pipeline (Snowflake uses identical structure)

from semantica.ingest import DatabricksIngestor
from semantica.kg import KnowledgeGraph
from semantica.visualization import KGVisualizer

# 1. Connect and ingest

ingestor = DatabricksIngestor(
    host="https://adb-123.azuredatabricks.net",
    token="dapi-xxxxxxxx",
    http_path="/sql/1.0/warehouses/abcd",
    catalog="sales",
    schema="public",
)
raw_data = ingestor.ingest_table("orders", limit=5_000)

# 2. Normalize to documents

documents = ingestor.export_as_documents(raw_data)

# 3. Build knowledge graph

kg = KnowledgeGraph()
kg.add_documents(documents)

# 4. Visualize or integrate with retrieval systems

visualizer = KGVisualizer()
visualizer.visualize_network(kg, output="interactive")

```

The Snowflake variant requires only substituting `SnowflakeIngestor` and Snowflake-specific connection parameters.

## Key Source Files and Architecture

| File | Purpose |
|------|---------|
| [`semantica/ingest/databricks_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/databricks_ingestor.py) | Databricks connector, ingestion logic, and `DatabricksData` dataclass |
| [`semantica/ingest/snowflake_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/snowflake_ingestor.py) | Snowflake connector and `SnowflakeData` dataclass |
| [`semantica/kg/knowledge_graph.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/kg/knowledge_graph.py) | Core `KnowledgeGraph` dataclass with entity/relationship storage |
| [`semantica/visualization/kg_visualizer.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/visualization/kg_visualizer.py) | Network visualization with `_convert_knowledge_graph()` helper |
| [`semantica/ingest/__init__.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/__init__.py) | Public API exports for `DatabricksIngestor` and `SnowflakeIngestor` |

## Summary

- **Connection abstraction**: Both ingestors use native drivers with PAT/OAuth authentication, validated in `__init__` methods
- **Streaming ingestion**: `ingest_table()` and custom query methods return rich dataclasses preserving source metadata
- **Document normalization**: `export_as_documents()` creates portable dictionaries decoupled from warehouse schemas
- **Unified KG building**: `KnowledgeGraph.add_documents()` extracts entities and relationships regardless of source system
- **Composable downstream**: The in-memory graph format integrates with visualization, vector stores, and agent systems

## Frequently Asked Questions

### What authentication methods does Semantica support for Databricks and Snowflake?

Semantica supports **personal access tokens (PAT)** and **OAuth M2M credentials** for both platforms. The Databricks connector validates these in `DatabricksConnector.__init__` ([`semantica/ingest/databricks_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/databricks_ingestor.py) lines 25-49), while Snowflake uses equivalent parameter validation. OAuth flows are handled natively through the underlying Python connectors.

### How does Semantica handle large table ingestion without memory issues?

The `ingest_table()` method accepts a **`limit`** parameter for bounded ingestion, and code analysis shows streaming row processing rather than full materialization. For production workloads, implement chunked ingestion using SQL `LIMIT`/`OFFSET` or partition-based scanning, then iteratively call `KnowledgeGraph.add_documents()` to build the graph incrementally.

### Can I use custom SQL queries instead of full table scans?

Yes. The ingestor architecture supports custom queries—`DatabricksData` and `SnowflakeData` dataclasses include an optional `query` field in metadata that captures the source SQL. Call `ingest_query()` (implied by the query metadata handling in `export_as_documents`) or extend the base ingestor pattern to inject arbitrary SQL before document normalization.

### What entity extraction model powers the KnowledgeGraph builder?

The source code in [`semantica/kg/knowledge_graph.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/kg/knowledge_graph.py) implements a **built-in entity-extraction pipeline** invoked during `add_documents()`. While the specific model architecture isn't detailed in the analyzed files, the extraction produces standardized `entities` and `relationships` structures suitable for downstream consumption by vector stores and visualization tools.