# What Kind of Data Does Semantica AGI Process? Enterprise Data Ingestion Explained

> Semantica AGI processes diverse enterprise data including flat files, databases, and live streams. Learn how Semantica handles any fetchable, parsable data expressed as entities, relationships, or facts.

- Repository: [Semantica /semantica](https://github.com/semantica-agi/semantica)
- Tags: how-to-guide
- Published: 2026-08-13

---

**Semantica AGI processes any enterprise data that can be fetched, parsed, and expressed as entities, relationships, or facts—from flat files and relational databases to live streaming feeds and cloud-hosted documents.**

Semantica AGI, developed by the [semantica-agi/semantica](https://github.com/semantica-agi/semantica) repository, is designed to ingest, normalize, and reason over heterogeneous data sources through a modular four-layer pipeline. Understanding what kind of data Semantica AGI processes requires examining its ingestion capabilities, document parsing mechanisms, and the knowledge graph output it generates for deterministic reasoning and AI applications.

## The Four-Layer Data Processing Architecture

The Semantica AGI architecture follows a strict pipeline: **Ingestion → Processing → Intelligence → Application**. Each layer handles specific data transformations, ensuring that raw bytes from any source become queryable, auditable knowledge graphs.

- **Ingestion**: Pulls raw content from external systems via modular connectors
- **Processing**: Parses documents and performs **entity-aware chunking** (GraphRAG style)
- **Intelligence**: Runs deterministic NER, relation extraction, and forward-chaining reasoning using Rete/Datalog/SPARQL engines
- **Application**: Exposes enriched knowledge via vector stores, polyglot graph stores (RDF & LPG), and REST APIs

## Ingestion Layer: Supported Data Types and Sources

The ingestion layer, implemented across `semantica/ingest/`, supports both batch and streaming data sources without requiring schema pre-definition.

### Local Files and Documents

In [`semantica/ingest/file_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/file_ingestor.py), the **FileIngestor** class handles heterogeneous document collections including PDF, DOCX, HTML, TXT, CSV, JSON, XML, Parquet, and Arrow files. This module recursively scans directories and automatically selects appropriate parsers based on file extensions and MIME types.

### Enterprise Data Warehouses

For structured relational data, Semantica AGI connects directly to modern data platforms:

- **Databricks Unity Catalog**: The `DatabricksIngestor` class in [`semantica/ingest/databricks_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/databricks_ingestor.py) queries tables via ODBC/JDBC connections, respecting warehouse compute limits and Unity Catalog governance policies
- **Snowflake**: The `SnowflakeIngestor` in [`semantica/ingest/snowflake_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/snowflake_ingestor.py) ingests tables using secure credential passing and warehouse-specific routing

### Web Content and APIs

The `WebIngestor` class in [`semantica/ingest/web_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/web_ingestor.py) fetches web pages while respecting [`robots.txt`](https://github.com/semantica-agi/semantica/blob/main/robots.txt) and rate limits. It handles RSS/Atom feeds, REST APIs, and public endpoints, converting HTML and JSON responses into the internal document representation.

### Streaming and Message Queues

Semantica AGI processes live event streams through connectors for **Apache Kafka**, **Amazon Kinesis**, **Apache Pulsar**, and **RabbitMQ**. These ingestors maintain offset tracking and exactly-once semantics, feeding streaming data into the processing layer without intermediate batch files.

### Cloud Storage and Collaboration Tools

The platform integrates with **Google Drive**, **Elasticsearch**, **Git repositories**, **IMAP/POP3 email servers**, **MCP resources**, and custom connectors for **MongoDB**, **DuckDB**, and **Pandas** DataFrames. This enables ingestion from SaaS tools, version control systems, and ad-hoc analytical datasets.

## Processing Layer: From Raw Data to Structured Entities

Once ingested, data flows into the processing layer where unstructured content is normalized into machine-readable structures.

### Entity-Aware Chunking

In [`semantica/split.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/split.py), Semantica AGI implements **entity-aware chunking** algorithms that preserve semantic boundaries and relationship contexts. Unlike simple text splitters, this GraphRAG-style approach ensures that entities mentioned across chunk boundaries maintain their relational integrity, critical for downstream knowledge graph accuracy.

### Multi-Modal Parsing

The processing layer handles **unstructured text**, **tables**, **images**, and **binary blobs** through a plugin-based parser architecture. Document layouts are preserved where possible, allowing the system to distinguish between header text, tabular data, and figure captions during triplet extraction.

## Intelligence Layer: Knowledge Graph Construction

The intelligence layer transforms parsed content into queryable knowledge structures:

- **Named Entity Recognition (NER)**: Extracts typed entities from text using deterministic algorithms
- **Relation Extraction**: Identifies predicates connecting entities, stored as subject-predicate-object triplets
- **Ontology Generation**: The `OntologyGenerator` in [`semantica/ontology.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ontology.py) creates OWL schemas and performs SHACL validation
- **Reasoning**: The [`reasoning.py`](https://github.com/semantica-agi/semantica/blob/main/reasoning.py) module executes forward-chaining inference using Rete networks, Datalog rules, and SPARQL queries against the knowledge graph stored in [`semantica/kg.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/kg.py)
- **Provenance Tracking**: [`semantica/provenance.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance.py) implements W3C PROV-O standards to maintain complete audit trails of data lineage

## Practical Examples: Ingesting Data with Semantica AGI

The following examples demonstrate how to ingest diverse data sources using the Python SDK:

```python

# Example 1 – Ingest a directory of heterogeneous contracts

from semantica.ingest import FileIngestor
docs = FileIngestor().ingest_directory("./contracts/", recursive=True)

```

```python

# Example 2 – Pull a live web page respecting robots.txt

from semantica.ingest import WebIngestor
html = WebIngestor().ingest_url("https://example.com/annual-report-2024.html")

```

```python

# Example 3 – Query a Databricks Unity Catalog table (requires the optional DB‑Databricks extra)

from semantica.ingest import DatabricksIngestor
db = DatabricksIngestor(
    host="https://adb-xyz.azuredatabricks.net",
    token="DATABRICKS_TOKEN",          # best passed via env var

    http_path="/sql/1.0/warehouses/12345",
    catalog="main",
)
customers = db.ingest_table("customers", limit=5_000)

```

```python

# Example 4 – Ingest from Snowflake (optional DB‑Snowflake extra)

from semantica.ingest import SnowflakeIngestor
sf = SnowflakeIngestor(
    account="myaccount",
    user="myuser",
    password="SNOWFLAKE_PASSWORD",
    warehouse="COMPUTE_WH",
    database="MYDB",
)
orders = sf.ingest_table("ORDERS", limit=10_000)

```

```python

# Example 5 – Consume the resulting knowledge graph in a downstream agent

from semantica.context import ContextGraph, AgentContext
from semantica.vector_store import VectorStore

graph = ContextGraph(advanced_analytics=True)
vs = VectorStore(backend="faiss")
ctx = AgentContext(vector_store=vs, knowledge_graph=graph)

ctx.store("Alice approved the Acme renewal in Q1 2024", conversation_id="c1")
answer = ctx.retrieve("who approved the Acme contract?")
print(answer)   # → "Alice"

```

## Summary

- **Semantica AGI processes any enterprise data** that can be fetched and parsed, including files, databases, streams, and APIs.
- The **ingestion layer** supports 15+ source types through modular classes like `FileIngestor`, `WebIngestor`, and `DatabricksIngestor`.
- **Entity-aware chunking** in [`semantica/split.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/split.py) preserves relationship context during document processing.
- The **intelligence layer** converts all inputs into structured triplets stored in a bi-temporal knowledge graph with full W3C PROV-O lineage tracking.
- Output is exposed via **vector stores**, **RDF/OWL exports**, and **REST APIs** for downstream LLM and BI tool consumption.

## Frequently Asked Questions

### Can Semantica AGI handle real-time streaming data?

Yes. Semantica AGI includes native connectors for Apache Kafka, Amazon Kinesis, Apache Pulsar, and RabbitMQ. These streaming ingestors maintain offset tracking and exactly-once processing semantics, feeding live events directly into the entity extraction pipeline without intermediate batch files.

### How does Semantica AGI differ from traditional ETL pipelines?

Traditional ETL requires predefined schemas and targets structured relational outputs. Semantica AGI uses **schema-on-read** ingestion and produces **semantic knowledge graphs** with entity relationships. Unlike ETL, it performs deterministic reasoning, conflict detection, and ontology validation through the modules in [`semantica/reasoning.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/reasoning.py) and [`semantica/ontology.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ontology.py).

### Can I add support for proprietary or custom data formats?

Yes. The ingestion architecture is plugin-based. You can extend the base ingestor classes in `semantica/ingest/` to handle proprietary APIs or binary formats, then register them in the ingestion registry. The processing layer in [`semantica/split.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/split.py) and [`semantica/semantic_extract.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/semantic_extract.py) will automatically apply entity-aware chunking and NER to your custom parser's output.

### How does the system track data lineage and provenance?

Semantica AGI implements **W3C PROV-O** standards through [`semantica/provenance.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/provenance.py), creating immutable audit trails for every data transformation. Each triplet in the knowledge graph maintains bi-temporal timestamps (valid time and transaction time) and references its original source URI, enabling complete downstream traceability for compliance and debugging purposes.