# How to Ingest Data from Various Sources into Semantica: A Complete Guide

> Easily ingest data from files, web pages, APIs, databases, and streams into Semantica. Our guide details the unified framework for seamless data integration. Learn more now!

- Repository: [Semantica /semantica](https://github.com/semantica-agi/semantica)
- Tags: how-to-guide
- Published: 2026-09-04

---

**Semantica provides a unified, extensible ingestion framework that maps source type strings to concrete ingestor classes through a central method registry, allowing you to pull data from files, web pages, APIs, databases, and real-time streams using a single consistent API.**

Semantica’s ingestion subsystem lives in the **`semantica.ingest`** package and serves as the entry point for all data acquisition pipelines. Whether you are processing local PDFs, crawling websites, or querying PostgreSQL databases, the framework abstracts source complexity behind high-level wrappers like `ingest()`, `ingest_file()`, and `ingest_web()`.

## Core Architecture: The Ingestion Pipeline

At the heart of Semantica’s data ingestion is a five-step workflow orchestrated by the **`MethodRegistry`** in [`semantica/ingest/methods.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/methods.py). When you invoke `ingest(source, source_type=...)`, the system executes:

1. **Method lookup** – `get_ingest_method(source_type)` retrieves the appropriate ingestor class (e.g., `WebIngestor`, `FileIngestor`).
2. **Configuration merge** – The method’s defaults combine with per-call overrides from the global **`ingest_config`** object defined in [`semantica/ingest/config.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/config.py).
3. **Instancing** – The ingestor class initializes with merged settings, including `rate_limit_delay`, `respect_robots`, and `batch_size`.
4. **Execution** – The ingestor’s `ingest_*` method runs, returning **`Document`**-like objects (or `WebContent` for URLs).
5. **Provenance attachment** – Optional mix-ins from [`semantica/ingest/ingest_provenance.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/ingest_provenance.py) automatically tag metadata to track data lineage.

## Supported Data Sources and Implementation Files

Semantica ships with specialized ingestors for diverse source types. Each implements a common interface while handling source-specific concerns like authentication, pagination, and format parsing.

| Source Type | Primary Ingestor Class | Key Implementation File |
|------------|------------------------|--------------------------|
| Local & Cloud Files | `FileIngestor` / `CloudStorageIngestor` | [`semantica/ingest/file_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/file_ingestor.py) |
| Web Pages & Sites | `WebIngestor` | [`semantica/ingest/web_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/web_ingestor.py) |
| Public REST APIs | `PublicAPIIngestor` | [`semantica/ingest/public_api_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/public_api_ingestor.py) |
| RSS/Atom Feeds | `FeedIngestor` | [`semantica/ingest/feed_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/feed_ingestor.py) |
| Real-time Streams | `StreamIngestor` | [`semantica/ingest/stream_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/stream_ingestor.py) |
| Git Repositories | `RepoIngestor` | [`semantica/ingest/repo_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/repo_ingestor.py) |
| Email Mailboxes | `EmailIngestor` | [`semantica/ingest/email_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/email_ingestor.py) |
| Relational Databases | `DBIngestor` | [`semantica/ingest/db_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/db_ingestor.py) |

All ingestors share centralized **progress tracking** via `get_progress_tracker` and unified **error handling** through `ProcessingError` and `ValidationError` in [`semantica/utils/exceptions.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/utils/exceptions.py).

## Ingesting Files from Local and Cloud Storage

To ingest data from filesystems or cloud buckets, use the `ingest_file` wrapper. It handles recursive directory scanning, file type filtering, and batch processing.

```python
from semantica.ingest import ingest_file

# Ingest all PDF files under a directory, using the built-in batch processor

documents = ingest_file(
    "./data/reports",
    method="directory",          # scans recursively

    file_type="pdf",            # filter to PDFs

    batch_size=50,              # process 50 files at a time

)

```

The `FileIngestor` class in [`semantica/ingest/file_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/file_ingestor.py) manages file discovery and type detection, while `CloudStorageIngestor` handles S3, GCS, and Azure Blob variants.

## Crawling Web Pages and Sitemaps

For web ingestion, `ingest_web` dispatches to `WebIngestor`, which integrates `RateLimiter`, `RobotsChecker`, and `ContentExtractor` to ensure respectful crawling.

**Single page ingestion:**

```python
from semantica.ingest import ingest_web

page = ingest_web(
    "https://news.example.com/article/12345",
    rate_limit_delay=0.5,      # half-second between requests

    respect_robots=True,
)
print(page.title, page.text[:200])

```

**Bulk sitemap crawling:**

```python
from semantica.ingest import ingest_web

pages = ingest_web(
    "https://example.com/sitemap.xml",
    method="sitemap",          # triggers SitemapCrawler internally

    max_urls=200,              # stop after 200 URLs

    filters={"domains": ["example.com"]},
)

```

The `WebIngestor` in [`semantica/ingest/web_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/web_ingestor.py) automatically guards against SSRF attacks using `request_with_ssrf_guard`.

## Consuming APIs, Feeds, and Streams

### Public REST APIs

Use `ingest_public_api` to pull JSON data from open endpoints. The `PublicAPIIngestor` extends `RESTIngestor` to handle pagination and authentication.

```python
from semantica.ingest import ingest_public_api

records = ingest_public_api(
    "https://api.example.com/v1/items",
    method="json",             # automatically parses JSON response

    auth=None,                 # no auth needed for public APIs

    batch_size=100,
)

```

### Real-time Streams

For Kafka, RabbitMQ, Kinesis, or Pulsar, the `StreamIngestor` in [`semantica/ingest/stream_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/stream_ingestor.py) provides continuous ingestion capabilities.

## Extracting from Databases and Code Repositories

### Relational Databases

The `DBIngestor` uses SQLAlchemy to execute queries and return batched results as Documents.

```python
from semantica.ingest import ingest_database

rows = ingest_database(
    connection_string="postgresql+psycopg2://user:pwd@host/db",
    query="SELECT * FROM events WHERE event_date > CURRENT_DATE - INTERVAL '30 days'",
    batch_size=500,
)

```

### Git Repositories

The `RepoIngestor` in [`semantica/ingest/repo_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/repo_ingestor.py) clones repositories and extracts source files, commits, and metadata (requires optional `GitPython`).

```python
from semantica.ingest import ingest_repository

repo_docs = ingest_repository(
    "https://github.com/owner/repo.git",
    branch="main",
    include_file_patterns=["*.py", "*.md"],
)

```

## Safety, Configuration, and Provenance

Semantica’s ingestion framework enforces safe defaults through the **`IngestConfig`** class in [`semantica/ingest/config.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/config.py). Key safety features include:

- **Rate limiting** – Configurable delays between requests to prevent overwhelming source servers.
- **Robots.txt compliance** – The `respect_robots` parameter checks `RobotsChecker` before fetching URLs.
- **SSRF protection** – All HTTP ingestors route through `request_with_ssrf_guard` in [`semantica/utils/exceptions.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/utils/exceptions.py).
- **Batch processing** – The `batch_size` parameter controls memory usage during large extractions.

Source metadata is automatically attached via the provenance mix-in system in [`semantica/ingest/ingest_provenance.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/ingest_provenance.py), ensuring downstream pipelines can trace data lineage back to the original `source_type` and URI.

## Summary

- **Unified API** – The `ingest()` function and specialized wrappers (`ingest_file`, `ingest_web`, `ingest_database`) provide a single entry point for all data sources.
- **Extensible Registry** – `MethodRegistry` in [`semantica/ingest/methods.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/methods.py) maps source type strings to concrete ingestor classes, enabling plugin architecture.
- **Built-in Safety** – Rate limiting, robots.txt compliance, and SSRF guards are enforced by default in `WebIngestor` and related network-based ingestors.
- **Provenance Tracking** – Automatic metadata attachment via [`ingest_provenance.py`](https://github.com/semantica-agi/semantica/blob/main/ingest_provenance.py) preserves data lineage for compliance and debugging.
- **Batch Processing** – All ingestors support `batch_size` parameters and integrate with `get_progress_tracker` for memory-efficient handling of large datasets.

## Frequently Asked Questions

### What source types does Semantica support?

Semantica supports local and cloud files, web pages and sitemaps, public REST APIs, RSS/Atom feeds, real-time streams (Kafka, RabbitMQ, Kinesis, Pulsar), Git repositories, email mailboxes (IMAP/POP3), and relational databases via SQLAlchemy. Enterprise sources like Snowflake, Databricks, and SAP OData have dedicated ingestors in the `semantica/ingest/` directory.

### How does Semantica handle rate limiting and web scraping etiquette?

The `WebIngestor` class in [`semantica/ingest/web_ingestor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/web_ingestor.py) integrates `RateLimiter` and `RobotsChecker` to enforce politeness. You can configure `rate_limit_delay` (in seconds) and set `respect_robots=True` to automatically respect robots.txt directives. These defaults prevent IP blacklisting and ensure compliance with site terms of service.

### Can I ingest data from private databases or authenticated APIs?

Yes. While the examples show public endpoints, the `DBIngestor` accepts full SQLAlchemy connection strings with credentials, and `PublicAPIIngestor` (extending `RESTIngestor`) supports authentication headers via the `auth` parameter. For enterprise sources, specific ingestors like `SnowflakeIngestor` handle proprietary authentication schemes.

### How does Semantica track the origin of ingested data?

The provenance system in [`semantica/ingest/ingest_provenance.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ingest/ingest_provenance.py) uses mix-ins to automatically attach source metadata to returned `Document` objects. This includes the original URI, ingestion timestamp, source type, and ingestor class name, enabling full lineage tracing through downstream extraction and embedding pipelines.