# How Semantica Handles Text Parsing and Normalization: A Deep Dive into the Core Pipeline

> Discover how Semantica performs text parsing and normalization through its core pipeline. Learn about MIME-type aware extraction and graph data conversion for a canonical schema.

- Repository: [Semantica /semantica](https://github.com/semantica-agi/semantica)
- Tags: deep-dive
- Published: 2026-09-04

---

**Semantica processes documents through a two-stage pipeline where [`semantica/mcp/tools/extraction.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/mcp/tools/extraction.py) handles MIME-type-aware text extraction via the `parse` CLI command, while [`semantica/utils/helpers.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/utils/helpers.py) provides `normalize_graph_payload` to convert heterogeneous graph data into a canonical schema of entities, relationships, and triplets.**

The open-source semantica-agi/semantica repository provides a robust framework for knowledge graph construction that relies on precise text parsing and normalization to transform raw inputs into structured data. Understanding how Semantica handles text parsing and normalization is essential for developers extending the platform or troubleshooting data ingestion workflows. This article examines the specific implementation details found in the extraction tools and utility helpers that form the backbone of Semantica's data processing architecture.

## Text Parsing Architecture in Semantica

Semantica's text parsing system converts heterogeneous document formats into a uniform plain-text representation suitable for downstream embedding and graph construction. The system is designed around file-type detection, pluggable parsers, and a consistent JSON output contract.

### CLI Entry Point and Command Registration

The parsing functionality is exposed through the `parse` command defined in [`semantica/mcp/__main__.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/mcp/__main__.py) and wired into the Click framework. This command serves as the primary interface for users extracting text from PDFs, DOCX files, and plain-text documents. When invoked, the CLI delegates actual processing to the extraction layer, ensuring a clean separation between interface logic and parsing implementation.

### File-Type Detection and Parser Dispatch

The core extraction logic resides in [`semantica/mcp/tools/extraction.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/mcp/tools/extraction.py), which inspects the file's extension or MIME type to select the appropriate parser. The system supports multiple backends:

- **PDF documents**: Processed via `pdfminer` to extract structured text content
- **Word documents**: Handled through `python-docx` for DOCX format support
- **Plain text**: Processed using built-in I/O operations for maximum efficiency

This dispatch mechanism ensures that each file format receives specialized handling while presenting a unified interface to calling code.

### JSON Output Structure

Regardless of input format, the extraction layer returns a standardized JSON envelope that downstream tools consume. The parser yields a plain-text string wrapped in a dictionary structure:

```bash
semantica parse path/to/document.pdf

# Output: {"text":"... extracted plain-text content ..."}

```

This consistent output format allows subsequent pipeline stages—such as the `split` and `embed` commands—to operate without format-specific branching logic.

## Graph Normalization Pipeline

Once text is extracted and processed, Semantica frequently ingests graph data from various exporters that may use divergent field names. The normalization layer reconciles these differences into a single predictable schema.

### The normalize_graph_payload Function

Located in [`semantica/utils/helpers.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/utils/helpers.py), the `normalize_graph_payload` function accepts heterogeneous graph payloads and reduces them to a canonical vocabulary. The function signature targets dictionaries containing mixed nomenclature:

```python
from semantica.utils.helpers import normalize_graph_payload

payload = {
    "nodes": [{"id": "n1", "label": "Person"}],
    "edges": [{"source": "n1", "target": "n2", "type": "knows"}],
}
normalized = normalize_graph_payload(payload)

# Result: {"entities": [...], "relationships": [...], "triplets": []}

```

The function handles three key mappings:
- **`entities`** vs. **`nodes`**
- **`relationships`** vs. **`edges`**
- **`triplets`** for raw subject-predicate-object triples

### Validation and Error Handling

The normalization function implements strict validation logic (lines 01038-01057 in the source) to prevent silent data loss. It raises `ValidationError` with descriptive messages when encountering:

- Non-dictionary payloads
- Recognized keys mapping to non-collection values
- Divergent data present under both spellings of the same collection (e.g., both `nodes` and `entities` with different contents)
- Payloads containing data but lacking recognized keys

```python
from semantica.utils.helpers import normalize_graph_payload, ValidationError

try:
    normalize_graph_payload({"nodes": "not a list"})
except ValidationError as exc:
    print(exc)  # Clear message about expected collection type

```

### Canonical Schema Enforcement

By enforcing a single schema where all graph data uses `entities`, `relationships`, and `triplets`, the normalization layer eliminates defensive coding in downstream components. Every consumer—from visualizers to export utilities—can assume consistent field names regardless of the data's origin.

## Integration Across the Codebase

The normalization utility sees widespread adoption across Semantica's visualization and export modules. In [`semantica/visualization/kg_visualizer.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/visualization/kg_visualizer.py), the `normalize_graph_payload` function is called before rendering knowledge graphs to ensure consistent node and edge processing. Similarly, [`semantica/visualization/utils/export_formats.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/visualization/utils/export_formats.py) relies on the normalized payload when generating CSV, JSON, and Parquet exports, preventing format-specific transformation logic from proliferating throughout the codebase.

## Summary

- **Text parsing** in Semantica operates through MIME-type detection in [`semantica/mcp/tools/extraction.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/mcp/tools/extraction.py), supporting PDF, DOCX, and TXT formats via specialized parsers while outputting a uniform `{"text": "..."}` JSON structure.
- **Graph normalization** utilizes `normalize_graph_payload` in [`semantica/utils/helpers.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/utils/helpers.py) to reconcile heterogeneous field names (entities/nodes, relationships/edges) into a canonical schema with strict validation.
- The CLI entry point in [`semantica/mcp/__main__.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/mcp/__main__.py) provides the `parse` command interface, while downstream tools like the KG visualizer and export formatters consume normalized data.
- Validation errors are explicit and descriptive, preventing silent data corruption when processing malformed graph payloads.

## Frequently Asked Questions

### How do I parse a PDF file using Semantica's CLI?

Execute the `parse` command with the file path as an argument: `semantica parse path/to/document.pdf`. The command, registered in [`semantica/mcp/__main__.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/mcp/__main__.py), delegates to [`semantica/mcp/tools/extraction.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/mcp/tools/extraction.py), which selects the PDF parser based on MIME type and returns a JSON object containing the extracted text.

### What schema does normalize_graph_payload enforce for graph data?

The function enforces a canonical schema using three keys: `entities` (formerly `nodes`), `relationships` (formerly `edges`), and `triplets` (for subject-predicate-object representations). According to the implementation in [`semantica/utils/helpers.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/utils/helpers.py), the function resolves duplicate spellings and raises errors if both variants contain divergent data.

### How does Semantica handle invalid graph payloads?

The `normalize_graph_payload` function raises `ValidationError` with specific messages when encountering non-dictionary inputs, non-collection values for recognized keys, or payloads missing all recognized keys. This strict validation, implemented around lines 01038-01057 in [`semantica/utils/helpers.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/utils/helpers.py), prevents downstream components from processing malformed data.

### Can I use the extraction tools programmatically instead of via CLI?

Yes, import `extract_text` directly from `semantica/mcp/tools/extraction` to use the parsing functionality within Python code. This bypasses the Click CLI layer while maintaining the same file-type detection and JSON output format used by the command-line interface.