How to Ingest Data from Various Sources into Semantica: A Complete Guide

Semantica provides a unified, extensible ingestion framework that maps source type strings to concrete ingestor classes through a central method registry, allowing you to pull data from files, web pages, APIs, databases, and real-time streams using a single consistent API.

Semantica’s ingestion subsystem lives in the semantica.ingest package and serves as the entry point for all data acquisition pipelines. Whether you are processing local PDFs, crawling websites, or querying PostgreSQL databases, the framework abstracts source complexity behind high-level wrappers like ingest(), ingest_file(), and ingest_web().

Core Architecture: The Ingestion Pipeline

At the heart of Semantica’s data ingestion is a five-step workflow orchestrated by the MethodRegistry in semantica/ingest/methods.py. When you invoke ingest(source, source_type=...), the system executes:

1. Method lookup – get_ingest_method(source_type) retrieves the appropriate ingestor class (e.g., WebIngestor, FileIngestor).
2. Configuration merge – The method’s defaults combine with per-call overrides from the global ingest_config object defined in semantica/ingest/config.py.
3. Instancing – The ingestor class initializes with merged settings, including rate_limit_delay, respect_robots, and batch_size.
4. Execution – The ingestor’s ingest_* method runs, returning Document-like objects (or WebContent for URLs).
5. Provenance attachment – Optional mix-ins from semantica/ingest/ingest_provenance.py automatically tag metadata to track data lineage.

Supported Data Sources and Implementation Files

Semantica ships with specialized ingestors for diverse source types. Each implements a common interface while handling source-specific concerns like authentication, pagination, and format parsing.

Source Type	Primary Ingestor Class	Key Implementation File
Local & Cloud Files	FileIngestor / CloudStorageIngestor	semantica/ingest/file_ingestor.py
Web Pages & Sites	WebIngestor	semantica/ingest/web_ingestor.py
Public REST APIs	PublicAPIIngestor	semantica/ingest/public_api_ingestor.py
RSS/Atom Feeds	FeedIngestor	semantica/ingest/feed_ingestor.py
Real-time Streams	StreamIngestor	semantica/ingest/stream_ingestor.py
Git Repositories	RepoIngestor	semantica/ingest/repo_ingestor.py
Email Mailboxes	EmailIngestor	semantica/ingest/email_ingestor.py
Relational Databases	DBIngestor	semantica/ingest/db_ingestor.py

All ingestors share centralized progress tracking via get_progress_tracker and unified error handling through ProcessingError and ValidationError in semantica/utils/exceptions.py.

Ingesting Files from Local and Cloud Storage

To ingest data from filesystems or cloud buckets, use the ingest_file wrapper. It handles recursive directory scanning, file type filtering, and batch processing.

from semantica.ingest import ingest_file

# Ingest all PDF files under a directory, using the built-in batch processor

documents = ingest_file(
    "./data/reports",
    method="directory",          # scans recursively

    file_type="pdf",            # filter to PDFs

    batch_size=50,              # process 50 files at a time

)

The FileIngestor class in semantica/ingest/file_ingestor.py manages file discovery and type detection, while CloudStorageIngestor handles S3, GCS, and Azure Blob variants.

Crawling Web Pages and Sitemaps

For web ingestion, ingest_web dispatches to WebIngestor, which integrates RateLimiter, RobotsChecker, and ContentExtractor to ensure respectful crawling.

Single page ingestion:

from semantica.ingest import ingest_web

page = ingest_web(
    "https://news.example.com/article/12345",
    rate_limit_delay=0.5,      # half-second between requests

    respect_robots=True,
)
print(page.title, page.text[:200])

Bulk sitemap crawling:

from semantica.ingest import ingest_web

pages = ingest_web(
    "https://example.com/sitemap.xml",
    method="sitemap",          # triggers SitemapCrawler internally

    max_urls=200,              # stop after 200 URLs

    filters={"domains": ["example.com"]},
)

The WebIngestor in semantica/ingest/web_ingestor.py automatically guards against SSRF attacks using request_with_ssrf_guard.

Consuming APIs, Feeds, and Streams

Public REST APIs

Use ingest_public_api to pull JSON data from open endpoints. The PublicAPIIngestor extends RESTIngestor to handle pagination and authentication.

from semantica.ingest import ingest_public_api

records = ingest_public_api(
    "https://api.example.com/v1/items",
    method="json",             # automatically parses JSON response

    auth=None,                 # no auth needed for public APIs

    batch_size=100,
)

Real-time Streams

For Kafka, RabbitMQ, Kinesis, or Pulsar, the StreamIngestor in semantica/ingest/stream_ingestor.py provides continuous ingestion capabilities.

Extracting from Databases and Code Repositories

Relational Databases

The DBIngestor uses SQLAlchemy to execute queries and return batched results as Documents.

from semantica.ingest import ingest_database

rows = ingest_database(
    connection_string="postgresql+psycopg2://user:pwd@host/db",
    query="SELECT * FROM events WHERE event_date > CURRENT_DATE - INTERVAL '30 days'",
    batch_size=500,
)

Git Repositories

The RepoIngestor in semantica/ingest/repo_ingestor.py clones repositories and extracts source files, commits, and metadata (requires optional GitPython).

from semantica.ingest import ingest_repository

repo_docs = ingest_repository(
    "https://github.com/owner/repo.git",
    branch="main",
    include_file_patterns=["*.py", "*.md"],
)

Safety, Configuration, and Provenance

Semantica’s ingestion framework enforces safe defaults through the IngestConfig class in semantica/ingest/config.py. Key safety features include:

• Rate limiting – Configurable delays between requests to prevent overwhelming source servers.
• Robots.txt compliance – The respect_robots parameter checks RobotsChecker before fetching URLs.
• SSRF protection – All HTTP ingestors route through request_with_ssrf_guard in semantica/utils/exceptions.py.
• Batch processing – The batch_size parameter controls memory usage during large extractions.

Source metadata is automatically attached via the provenance mix-in system in semantica/ingest/ingest_provenance.py, ensuring downstream pipelines can trace data lineage back to the original source_type and URI.

Summary

• Unified API – The ingest() function and specialized wrappers (ingest_file, ingest_web, ingest_database) provide a single entry point for all data sources.
• Extensible Registry – MethodRegistry in semantica/ingest/methods.py maps source type strings to concrete ingestor classes, enabling plugin architecture.
• Built-in Safety – Rate limiting, robots.txt compliance, and SSRF guards are enforced by default in WebIngestor and related network-based ingestors.
• Provenance Tracking – Automatic metadata attachment via ingest_provenance.py preserves data lineage for compliance and debugging.
• Batch Processing – All ingestors support batch_size parameters and integrate with get_progress_tracker for memory-efficient handling of large datasets.

Frequently Asked Questions

What source types does Semantica support?

Semantica supports local and cloud files, web pages and sitemaps, public REST APIs, RSS/Atom feeds, real-time streams (Kafka, RabbitMQ, Kinesis, Pulsar), Git repositories, email mailboxes (IMAP/POP3), and relational databases via SQLAlchemy. Enterprise sources like Snowflake, Databricks, and SAP OData have dedicated ingestors in the semantica/ingest/ directory.

How does Semantica handle rate limiting and web scraping etiquette?

The WebIngestor class in semantica/ingest/web_ingestor.py integrates RateLimiter and RobotsChecker to enforce politeness. You can configure rate_limit_delay (in seconds) and set respect_robots=True to automatically respect robots.txt directives. These defaults prevent IP blacklisting and ensure compliance with site terms of service.

Can I ingest data from private databases or authenticated APIs?

Yes. While the examples show public endpoints, the DBIngestor accepts full SQLAlchemy connection strings with credentials, and PublicAPIIngestor (extending RESTIngestor) supports authentication headers via the auth parameter. For enterprise sources, specific ingestors like SnowflakeIngestor handle proprietary authentication schemes.

How does Semantica track the origin of ingested data?

The provenance system in semantica/ingest/ingest_provenance.py uses mix-ins to automatically attach source metadata to returned Document objects. This includes the original URI, ingestion timestamp, source type, and ingestor class name, enabling full lineage tracing through downstream extraction and embedding pipelines.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}