How to Use Semantica's Ingest Module for Multi-Source Data: A Complete Guide

Semantica's ingest module provides a plug-in-style framework that unifies data ingestion from web pages, Snowflake warehouses, XML files, and more through a common configuration and progress-tracking interface.

The semantica-agi/semantica repository offers a production-ready ingestion system designed for heterogeneous data pipelines. Whether you're building a knowledge graph, vector store, or analytical warehouse, the ingest module lets you pull from multiple sources with consistent tooling and output formats.

Core Architecture of the Ingest Module

Semantica's ingest system is built on four interconnected components that work together to provide safe, observable, and configurable data ingestion.

IngestConfig: Centralized Configuration Management

The IngestConfig class in semantica/ingest/config.py serves as the global configuration hub. It reads environment variables, optional YAML/JSON/TOML files, and allows programmatic overrides. Key settings include default_source_type and rate_limit_delay.

from semantica.ingest.config import ingest_config

ingest_config.set("default_source_type", "web")
ingest_config.set("rate_limit_delay", 0.5)  # 500ms between requests

Progress Tracking Across All Sources

The get_progress_tracker singleton provides visibility into long-running operations. Every ingestor integrates this automatically, recording start/stop events and optional status messages. Toggle tracking on or off via configuration.

Security and Utility Helpers

Common functionality is factored into reusable utilities:

• SSRF protection: request_with_ssrf_guard in semantica/ingest/ssrf.py
• Rate limiting: RateLimiter class in web_ingestor.py (lines 80-90)
• Robots.txt compliance: RobotsChecker (lines 20-31)
• Content extraction: ContentExtractor (lines 88-99)

Ingesting Web Data with WebIngestor

The WebIngestor class in semantica/ingest/web_ingestor.py handles single URLs, sitemaps, and domain-wide crawling with built-in politeness and safety controls.

Basic Web Page Ingestion

from semantica.ingest import WebIngestor

web = WebIngestor(delay=1.0, respect_robots=True)

page = web.ingest_url("https://example.com/article")
print(page.title, page.text[:200])

Sitemap and Domain Crawling

# Crawl a sitemap (returns list of WebContent)

pages = web.crawl_sitemap("https://example.com/sitemap.xml", max_urls=10)

# Breadth-first domain crawl, same-domain links only

domain_pages = web.crawl_domain("example.com", max_pages=20)

Each WebContent object contains extracted text, metadata, and discovered links ready for downstream processing.

Ingesting Snowflake Data with SnowflakeIngestor

The SnowflakeIngestor in semantica/ingest/snowflake_ingestor.py provides table-level and query-based ingestion with automatic document conversion.

Connecting and Ingesting Tables

from semantica.ingest import SnowflakeIngestor

sf = SnowflakeIngestor(
    account="myaccount",
    user="myuser",
    password="mypassword",
    warehouse="COMPUTE_WH",
    database="MYDB",
)

data = sf.ingest_table("CUSTOMERS", limit=5000)

The _escape_identifier method (lines 25-36) safeguards against SQL injection by properly quoting identifiers.

Custom Queries and Document Export

query_data = sf.ingest_query(
    "SELECT * FROM SALES WHERE DATE > %(date)s",
    params={"date": "2024-01-01"},
)

docs = sf.export_as_documents(data, text_fields=["NAME", "DESCRIPTION"])

The export_as_documents method (lines 26-38) produces Semantica-compatible dictionaries with id, text, and metadata fields.

Ingesting XML Data with XMLIngestor

The XMLIngestor in semantica/ingest/xml_ingestor.py handles files, directories, and raw strings with optional schema validation.

File and Directory Ingestion

from semantica.ingest import XMLIngestor

xml_ingestor = XMLIngestor()

xml_data = xml_ingestor.ingest_file("data/catalog.xml")
print(xml_data.root_tag, xml_data.metadata["element_count"])

all_xml = xml_ingestor.ingest_directory("data/xml_reports", recursive=True)

String-Based Ingestion

xml_string = "<root><item id='1'>Hello</item></root>"
xml_obj = xml_ingestor.ingest_string(xml_string)

The _parse_tree method (lines 48-70) builds an lxml tree with configurable parser options, while _validate_tree (lines 84-110) performs optional XSD/DTD validation.

Building Multi-Source Ingestion Pipelines

Combine multiple ingestors to create unified document collections from heterogeneous sources:

from semantica.ingest import WebIngestor, SnowflakeIngestor, XMLIngestor

web = WebIngestor()
sf = SnowflakeIngestor(account="...", user="...", password="...")
xml = XMLIngestor()

documents = []

# Web pages

for page in web.crawl_sitemap("https://blog.example.com/sitemap.xml", max_urls=5):
    documents.append({
        "id": page.url,
        "text": page.text,
        "metadata": {"source": "web", **page.metadata}
    })

# Snowflake table

sf_data = sf.ingest_table("PRODUCTS", limit=100)
documents.extend(sf.export_as_documents(sf_data, text_fields=["NAME", "DESCRIPTION"]))

# XML files

for xml_doc in xml.ingest_directory("configs/xml"):
    documents.append({
        "id": xml_doc.source_path,
        "text": xml_doc.root.get("text", ""),
        "metadata": {"source": "xml", **xml_doc.metadata}
    })

All documents share a common schema compatible with Semantica's vector-store and knowledge-graph pipelines.

Key Implementation Files

File	Purpose
semantica/ingest/config.py	Global IngestConfig manager
semantica/ingest/web_ingestor.py	Web crawling, rate limiting, robots.txt compliance
semantica/ingest/snowflake_ingestor.py	Snowflake connection, table/query ingestion
semantica/ingest/xml_ingestor.py	XML parsing, validation, directory ingestion
semantica/utils/progress_tracker.py	Progress tracking singleton
semantica/utils/logging.py	Consistent logging utilities
semantica/ingest/ssrf.py	SSRF-guarded HTTP requests

Summary

• Semantica's ingest module provides a unified framework for multi-source data ingestion through pluggable source-specific classes.
• IngestConfig centralizes settings across all ingestors with environment variable, file, and programmatic configuration support.
• WebIngestor, SnowflakeIngestor, and XMLIngestor each implement source-optimized ingestion with shared utilities for safety and observability.
• Progress tracking is built into every ingestor via the get_progress_tracker singleton.
• Output normalization through methods like export_as_documents ensures heterogeneous sources produce compatible data for downstream pipelines.

Frequently Asked Questions

What file formats does Semantica's XMLIngestor support?

XMLIngestor supports standard XML files, compressed XML, and raw XML strings. It uses lxml for parsing with configurable options and optional XSD/DTD validation via _validate_tree. Directory ingestion with recursive=True processes nested folder structures.

How does WebIngestor handle rate limiting and robots.txt?

WebIngestor uses the RateLimiter class (lines 80-90) to enforce delays between requests and RobotsChecker (lines 20-31) to fetch and parse robots.txt files. Set respect_robots=True and delay in seconds during instantiation to ensure polite crawling.

Can I extend the ingest module with custom data sources?

Yes. The plug-in architecture expects new sources to implement a small focused API (methods like ingest_url, ingest_table, ingest_file) and integrate with IngestConfig for settings and get_progress_tracker for observability. Follow the patterns in web_ingestor.py, snowflake_ingestor.py, or xml_ingestor.py.

How do I convert Snowflake query results to documents for vector storage?

Use SnowflakeIngestor.export_as_documents(data, text_fields=[...]) (lines 26-38). Specify which columns contain text content; the method returns a list of dictionaries with id, text, and metadata keys ready for vector-store indexing.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}