What Data Sources Can Semantica Ingest? A Complete Guide to 15+ Supported Types

Semantica can ingest data from more than 15 distinct source categories including local files (PDF, DOCX, CSV, JSON), web pages, databases (PostgreSQL, MySQL, Snowflake), message streams (Kafka, RabbitMQ), Git repositories, and enterprise systems like SAP and Databricks.

The ingestion framework in the semantica-agi/semantica repository provides dedicated Ingestor classes for each source type, all implemented under the semantica.ingest package. Every ingestor converts raw inputs into semantically-rich entities ready for the knowledge graph pipeline, with built-in provenance tracking and flexible authentication options.

Files and Documents

Semantica handles virtually any document format through FileIngestor in [semantica/ingest/file_ingestor.py](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/file_ingestor.py). The FileTypeDetector subsystem identifies formats via extension, MIME type, or magic-number analysis.

Supported file types include:

• PDF, DOCX, PPTX, HTML, TXT — standard office and web documents
• CSV, JSON, YAML, XML — structured data formats
• Excel — spreadsheet files

from semantica.ingest import FileIngestor

ingestor = FileIngestor()
documents = ingestor.ingest_directory("./contracts/", recursive=True)

print(f"Ingested {len(documents)} documents")

Web and API Sources

For external data, Semantica offers multiple specialized ingestors:

Web Pages

WebIngestor ([web_ingestor.py](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/web_ingestor.py)) fetches publicly reachable URLs with configurable crawl behavior:

from semantica.ingest import WebIngestor

ingestor = WebIngestor(delay=1.0, respect_robots=True)
page = ingestor.ingest_url("https://example.com/annual-report.html")

print(page.text[:200])

RSS and Atom Feeds

FeedIngestor handles syndicated content streams via ([feed_ingestor.py](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/feed_ingestor.py)).

REST APIs

RESTIngestor and PublicAPIIngestor in ([api_ingestor.py](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/api_ingestor.py)) support generic HTTP/HTTPS endpoints with optional authentication headers.

Databases and Data Warehouses

Relational Databases

DBIngestor ([db_ingestor.py](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/db_ingestor.py)) supports PostgreSQL, MySQL, SQLite, Oracle, and SQL Server:

from semantica.ingest import DBIngestor

ingestor = DBIngestor()
rows = ingestor.ingest_database(
    dialect="postgresql",
    host="db.example.com",
    port=5432,
    database="sales",
    username="alice",
    password="••••••"
)

Cloud Data Warehouses

• Snowflake — via SnowflakeIngestor in ([snowflake_ingestor.py](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/snowflake_ingestor.py))
• Databricks — Unity Catalog and Delta Lake via DatabricksIngestor in ([databricks_ingestor.py](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/databricks_ingestor.py))

Columnar and Arrow Formats

Parquet Datasets

ParquetIngestor ([parquet_ingestor.py](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/parquet_ingestor.py)) handles high-performance columnar storage:

from semantica.ingest import ParquetIngestor

ingestor = ParquetIngestor()
records = ingestor.ingest("./data/transactions.parquet")

Apache Arrow

ArrowIngestor ([arrow_ingestor.py](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/arrow_ingestor.py)) supports Arrow, Feather, and IPC files:

from semantica.ingest import ArrowIngestor

arrow_ingestor = ArrowIngestor()
table = arrow_ingestor.ingest("./datasets/sample.arrow")

Enterprise and Specialized Sources

SAP ERP

SAPIngestor ([sap_ingestor.py](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/sap_ingestor.py)) connects to SAP systems via OData v2/v4 protocols.

Version Control

RepoIngestor ([repo_ingestor.py](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/repo_ingestor.py)) clones and ingests Git repositories:

from semantica.ingest import RepoIngestor

repo_ingestor = RepoIngestor()
repo_data = repo_ingestor.ingest_repository(
    url="https://github.com/example/project.git",
    branch="main"
)

Email Systems

EmailIngestor ([email_ingestor.py](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/email_ingestor.py)) connects to IMAP and POP3 mailboxes.

Message Streams and Real-Time Data

For continuous data flows, Semantica implements StreamIngestor and MessageIngestor supporting:

• Kafka
• RabbitMQ
• Amazon Kinesis
• Apache Pulsar

These classes enable real-time ingestion into the knowledge graph pipeline.

MCP Resources

MCPIngestor ([mcp_ingestor.py](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/mcp_ingestor.py)) handles custom "MCP" (Model Context Protocol) assets for extended integration scenarios.

How Ingestion Architecture Works

The semantica.ingest package follows a consistent five-stage pattern across all ingestors:

1. Discovery & Loading — Top-level functions (ingest_file, ingest_web, ingest_database) instantiate the appropriate ingestor
2. Type Detection — FileTypeDetector identifies formats automatically
3. Parsing & Normalisation — Sources convert to canonical triplets (source, target, type)
4. Provenance Tracking — IngestProvenanceMixin adds source lineage metadata
5. Streaming & Batching — StreamIngestor handles continuous flows; FileIngestor supports recursive directory processing

Summary

• 15+ source categories are supported in Semantica's ingestion framework
• Dedicated ingestor classes in semantica.ingest handle each source type with consistent interfaces
• File formats span documents, structured data, columnar storage, and Apache Arrow
• Databases include PostgreSQL, MySQL, SQLite, Oracle, SQL Server, Snowflake, and Databricks Delta Lake
• Real-time streams via Kafka, RabbitMQ, Kinesis, and Pulsar use StreamIngestor and MessageIngestor
• Enterprise integration covers SAP OData, Git repositories, IMAP/POP3 email, and custom MCP resources
• Provenance tracking and batch/recursive processing are built into the core architecture

Frequently Asked Questions

Does Semantica support Microsoft Office documents?

Yes. FileIngestor in [file_ingestor.py](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/file_ingestor.py) natively handles DOCX, PPTX, and Excel formats through automatic type detection.

Can Semantica ingest from private APIs requiring authentication?

Yes. RESTIngestor and PublicAPIIngestor in [api_ingestor.py](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/api_ingestor.py) support custom headers and authentication tokens passed via environment variables or explicit arguments.

Is streaming data ingestion supported for real-time pipelines?

Yes. StreamIngestor and MessageIngestor enable continuous ingestion from Kafka, RabbitMQ, Kinesis, and Pulsar, integrating streaming sources directly with the knowledge graph pipeline.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}