# How to Use Semantica's Ingest Module for Multi-Source Data: A Complete Guide

> Learn how to use Semantica's ingest module to unify data from web pages, Snowflake, XML, and more with its plug-in framework. Streamline your data ingestion process.

- Repository: [Semantica /semantica](https://github.com/semantica-agi/semantica)
- Tags: how-to-guide
- Published: 2026-09-05

---

**Semantica's ingest module provides a plug-in-style framework that unifies data ingestion from web pages, Snowflake warehouses, XML files, and more through a common configuration and progress-tracking interface.**

The `semantica-agi/semantica` repository offers a production-ready ingestion system designed for heterogeneous data pipelines. Whether you're building a knowledge graph, vector store, or analytical warehouse, the ingest module lets you pull from multiple sources with consistent tooling and output formats.

## Core Architecture of the Ingest Module

Semantica's ingest system is built on four interconnected components that work together to provide safe, observable, and configurable data ingestion.

### IngestConfig: Centralized Configuration Management

The `IngestConfig` class in [`semantica/ingest/config.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/config.py) serves as the global configuration hub. It reads environment variables, optional YAML/JSON/TOML files, and allows programmatic overrides. Key settings include `default_source_type` and `rate_limit_delay`.

```python
from semantica.ingest.config import ingest_config

ingest_config.set("default_source_type", "web")
ingest_config.set("rate_limit_delay", 0.5)  # 500ms between requests

```

### Progress Tracking Across All Sources

The `get_progress_tracker` singleton provides visibility into long-running operations. Every ingestor integrates this automatically, recording start/stop events and optional status messages. Toggle tracking on or off via configuration.

### Security and Utility Helpers

Common functionality is factored into reusable utilities:
- **SSRF protection**: `request_with_ssrf_guard` in [`semantica/ingest/ssrf.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/ssrf.py)
- **Rate limiting**: `RateLimiter` class in [`web_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/web_ingestor.py) (lines 80-90)
- **Robots.txt compliance**: `RobotsChecker` (lines 20-31)
- **Content extraction**: `ContentExtractor` (lines 88-99)

## Ingesting Web Data with WebIngestor

The `WebIngestor` class in [`semantica/ingest/web_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/web_ingestor.py) handles single URLs, sitemaps, and domain-wide crawling with built-in politeness and safety controls.

### Basic Web Page Ingestion

```python
from semantica.ingest import WebIngestor

web = WebIngestor(delay=1.0, respect_robots=True)

page = web.ingest_url("https://example.com/article")
print(page.title, page.text[:200])

```

### Sitemap and Domain Crawling

```python

# Crawl a sitemap (returns list of WebContent)

pages = web.crawl_sitemap("https://example.com/sitemap.xml", max_urls=10)

# Breadth-first domain crawl, same-domain links only

domain_pages = web.crawl_domain("example.com", max_pages=20)

```

Each `WebContent` object contains extracted text, metadata, and discovered links ready for downstream processing.

## Ingesting Snowflake Data with SnowflakeIngestor

The `SnowflakeIngestor` in [`semantica/ingest/snowflake_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/snowflake_ingestor.py) provides table-level and query-based ingestion with automatic document conversion.

### Connecting and Ingesting Tables

```python
from semantica.ingest import SnowflakeIngestor

sf = SnowflakeIngestor(
    account="myaccount",
    user="myuser",
    password="mypassword",
    warehouse="COMPUTE_WH",
    database="MYDB",
)

data = sf.ingest_table("CUSTOMERS", limit=5000)

```

The `_escape_identifier` method (lines 25-36) safeguards against SQL injection by properly quoting identifiers.

### Custom Queries and Document Export

```python
query_data = sf.ingest_query(
    "SELECT * FROM SALES WHERE DATE > %(date)s",
    params={"date": "2024-01-01"},
)

docs = sf.export_as_documents(data, text_fields=["NAME", "DESCRIPTION"])

```

The `export_as_documents` method (lines 26-38) produces Semantica-compatible dictionaries with `id`, `text`, and `metadata` fields.

## Ingesting XML Data with XMLIngestor

The `XMLIngestor` in [`semantica/ingest/xml_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/xml_ingestor.py) handles files, directories, and raw strings with optional schema validation.

### File and Directory Ingestion

```python
from semantica.ingest import XMLIngestor

xml_ingestor = XMLIngestor()

xml_data = xml_ingestor.ingest_file("data/catalog.xml")
print(xml_data.root_tag, xml_data.metadata["element_count"])

all_xml = xml_ingestor.ingest_directory("data/xml_reports", recursive=True)

```

### String-Based Ingestion

```python
xml_string = "<root><item id='1'>Hello</item></root>"
xml_obj = xml_ingestor.ingest_string(xml_string)

```

The `_parse_tree` method (lines 48-70) builds an `lxml` tree with configurable parser options, while `_validate_tree` (lines 84-110) performs optional XSD/DTD validation.

## Building Multi-Source Ingestion Pipelines

Combine multiple ingestors to create unified document collections from heterogeneous sources:

```python
from semantica.ingest import WebIngestor, SnowflakeIngestor, XMLIngestor

web = WebIngestor()
sf = SnowflakeIngestor(account="...", user="...", password="...")
xml = XMLIngestor()

documents = []

# Web pages

for page in web.crawl_sitemap("https://blog.example.com/sitemap.xml", max_urls=5):
    documents.append({
        "id": page.url,
        "text": page.text,
        "metadata": {"source": "web", **page.metadata}
    })

# Snowflake table

sf_data = sf.ingest_table("PRODUCTS", limit=100)
documents.extend(sf.export_as_documents(sf_data, text_fields=["NAME", "DESCRIPTION"]))

# XML files

for xml_doc in xml.ingest_directory("configs/xml"):
    documents.append({
        "id": xml_doc.source_path,
        "text": xml_doc.root.get("text", ""),
        "metadata": {"source": "xml", **xml_doc.metadata}
    })

```

All documents share a common schema compatible with Semantica's vector-store and knowledge-graph pipelines.

## Key Implementation Files

| File | Purpose |
|------|---------|
| [`semantica/ingest/config.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/config.py) | Global `IngestConfig` manager |
| [`semantica/ingest/web_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/web_ingestor.py) | Web crawling, rate limiting, robots.txt compliance |
| [`semantica/ingest/snowflake_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/snowflake_ingestor.py) | Snowflake connection, table/query ingestion |
| [`semantica/ingest/xml_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/xml_ingestor.py) | XML parsing, validation, directory ingestion |
| [`semantica/utils/progress_tracker.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/utils/progress_tracker.py) | Progress tracking singleton |
| [`semantica/utils/logging.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/utils/logging.py) | Consistent logging utilities |
| [`semantica/ingest/ssrf.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/ssrf.py) | SSRF-guarded HTTP requests |

## Summary

- **Semantica's ingest module** provides a unified framework for multi-source data ingestion through pluggable source-specific classes.
- **`IngestConfig`** centralizes settings across all ingestors with environment variable, file, and programmatic configuration support.
- **`WebIngestor`**, **`SnowflakeIngestor`**, and **`XMLIngestor`** each implement source-optimized ingestion with shared utilities for safety and observability.
- **Progress tracking** is built into every ingestor via the `get_progress_tracker` singleton.
- **Output normalization** through methods like `export_as_documents` ensures heterogeneous sources produce compatible data for downstream pipelines.

## Frequently Asked Questions

### What file formats does Semantica's XMLIngestor support?

`XMLIngestor` supports standard XML files, compressed XML, and raw XML strings. It uses `lxml` for parsing with configurable options and optional XSD/DTD validation via `_validate_tree`. Directory ingestion with `recursive=True` processes nested folder structures.

### How does WebIngestor handle rate limiting and robots.txt?

`WebIngestor` uses the `RateLimiter` class (lines 80-90) to enforce delays between requests and `RobotsChecker` (lines 20-31) to fetch and parse robots.txt files. Set `respect_robots=True` and `delay` in seconds during instantiation to ensure polite crawling.

### Can I extend the ingest module with custom data sources?

Yes. The plug-in architecture expects new sources to implement a small focused API (methods like `ingest_url`, `ingest_table`, `ingest_file`) and integrate with `IngestConfig` for settings and `get_progress_tracker` for observability. Follow the patterns in [`web_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/web_ingestor.py), [`snowflake_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/snowflake_ingestor.py), or [`xml_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/xml_ingestor.py).

### How do I convert Snowflake query results to documents for vector storage?

Use `SnowflakeIngestor.export_as_documents(data, text_fields=[...])` (lines 26-38). Specify which columns contain text content; the method returns a list of dictionaries with `id`, `text`, and `metadata` keys ready for vector-store indexing.