What File Types Can Semantica's FileIngestor Process?

Semantica's FileIngestor automatically detects and ingests 37 distinct file formats across documents, images, audio, and video by validating extensions against consolidated constants defined in the framework's constants.py module.

Semantica (semantica-agi/semantica) is an open-source multimodal AI framework designed for comprehensive data processing. The repository's FileIngestor class, implemented in semantica/ingest/file_ingestor.py, serves as the primary entry point for local file ingestion, automatically categorizing content and extracting metadata for downstream analysis pipelines.

Supported File Type Categories

The ingestor organizes supported formats into four distinct categories. These lists are defined as constants in semantica/utils/constants.py (lines 49-71) and merged during FileIngestor initialization.

Documents

Document formats include both office suites and data serialization types:

• pdf, docx, doc, txt, html, xml
• json, csv, xlsx, pptx
• Columnar formats: parquet, pq, arrow, feather, ipc

Images

Image formats cover web, print, and vector graphics:

• jpg, jpeg, png, gif, bmp
• tiff, webp, svg

Audio

Audio formats span compressed and lossless codecs:

• mp3, wav, flac, aac
• ogg, wma, m4a

Video

Video formats include modern web and legacy container types:

• mp4, avi, mov, wmv
• flv, webm, mkv

How FileIngestor Validates File Types

The validation architecture relies on three core components working in sequence.

Format List Construction

During instantiation (lines 34-36 of file_ingestor.py), the ingestor builds self.supported_formats by concatenating four arrays imported from semantica/utils/constants.py:

• SUPPORTED_DOCUMENT_FORMATS
• SUPPORTED_IMAGE_FORMATS
• SUPPORTED_AUDIO_FORMATS
• SUPPORTED_VIDEO_FORMATS

Multi-Layer Type Detection

The FileTypeDetector class (referenced around line 81 of the same file) identifies file types through three mechanisms:

1. Extension analysis of the filename suffix
2. MIME type inspection via system metadata
3. Magic number detection for binary file headers

Validation Workflow

When processing files, the ingestor executes a strict validation protocol:

• Detects the candidate type using FileTypeDetector
• Checks membership against self.supported_formats
• Emits a warning for unsupported extensions
• Creates a FileObject regardless of support status, setting metadata['is_supported'] accordingly

This design allows downstream components to implement fallback handling for unsupported formats while maintaining strict validation gates for standard pipelines.

Practical Code Examples

Ingest an Entire Directory

Scan recursively and verify support status for every discovered file:

from semantica.ingest import FileIngestor

ingestor = FileIngestor()
files = ingestor.ingest_directory("./data", recursive=True)

for f in files:
    print(f"{f.name} → type: {f.file_type} (supported: {f.metadata['is_supported']})")

Process a Single File with Content Extraction

Ingest a PDF and access its extracted text and metadata:

ingestor = FileIngestor()
pdf = ingestor.ingest_file("report.pdf", read_content=True)

print(f"Path: {pdf.path}")
print(f"Size: {pdf.size:,} bytes")
print(f"Detected type: {pdf.file_type}")
print(f"MIME: {pdf.mime_type}")
print(f"First 100 chars of text: {pdf.text[:100]}")

Filter by Specific Extensions

Use scan_directory to target only desired formats while setting minimum file size thresholds:

files_meta = ingestor.scan_directory(
    "./media",
    extensions=["pdf", "png", "mp3"],
    min_size=1024,
    recursive=False
)

print(f"Found {len(files_meta)} supported files:")
for meta in files_meta:
    print(f"  {meta['name']} ({meta['extension']}, {meta['size']} bytes)")

Summary

• FileIngestor supports 37 distinct extensions across four media categories defined in semantica/utils/constants.py (lines 49-71).
• The supported_formats list is constructed dynamically in file_ingestor.py (lines 34-36) by merging document, image, audio, and video constants.
• FileTypeDetector utilizes extension, MIME type, and magic number analysis to ensure accurate format identification even with mislabeled extensions.
• Unsupported files trigger warnings but still generate FileObject instances, enabling flexible pipeline error handling.

Frequently Asked Questions

Does FileIngestor support custom file extensions?

While the base implementation validates against predefined lists in constants.py, the ingestor creates FileObject instances for unrecognized extensions with is_supported set to False. This allows downstream components to implement custom parsers for proprietary formats without modifying core ingestion logic.

How does FileIngestor detect file types without relying solely on extensions?

According to the semantica-agi/semantica source code, FileIngestor delegates detection to the FileTypeDetector class (implemented in file_ingestor.py around line 81). This utility analyzes MIME type headers and file magic numbers in addition to extension strings, ensuring accurate identification even when users rename files with incorrect suffixes.

Can I filter files by specific formats during directory scanning?

Yes. The scan_directory method accepts an extensions parameter accepting a list of strings (e.g., extensions=["pdf", "png", "mp3"]). When provided, the ingestor skips files not matching the specified whitelist, optimizing performance for targeted ingestion workflows.

What happens when FileIngestor encounters an unsupported file?

The ingestor emits a runtime warning via the logging framework but still instantiates a FileObject with complete metadata including the detected MIME type and file size. The metadata['is_supported'] flag is set to False, allowing your application logic to decide whether to skip, convert, or process the file through alternative means.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}