How Semantica Handles Text Parsing and Normalization: A Deep Dive into the Core Pipeline

Semantica processes documents through a two-stage pipeline where semantica/mcp/tools/extraction.py handles MIME-type-aware text extraction via the parse CLI command, while semantica/utils/helpers.py provides normalize_graph_payload to convert heterogeneous graph data into a canonical schema of entities, relationships, and triplets.

The open-source semantica-agi/semantica repository provides a robust framework for knowledge graph construction that relies on precise text parsing and normalization to transform raw inputs into structured data. Understanding how Semantica handles text parsing and normalization is essential for developers extending the platform or troubleshooting data ingestion workflows. This article examines the specific implementation details found in the extraction tools and utility helpers that form the backbone of Semantica's data processing architecture.

Text Parsing Architecture in Semantica

Semantica's text parsing system converts heterogeneous document formats into a uniform plain-text representation suitable for downstream embedding and graph construction. The system is designed around file-type detection, pluggable parsers, and a consistent JSON output contract.

CLI Entry Point and Command Registration

The parsing functionality is exposed through the parse command defined in semantica/mcp/__main__.py and wired into the Click framework. This command serves as the primary interface for users extracting text from PDFs, DOCX files, and plain-text documents. When invoked, the CLI delegates actual processing to the extraction layer, ensuring a clean separation between interface logic and parsing implementation.

File-Type Detection and Parser Dispatch

The core extraction logic resides in semantica/mcp/tools/extraction.py, which inspects the file's extension or MIME type to select the appropriate parser. The system supports multiple backends:

• PDF documents: Processed via pdfminer to extract structured text content
• Word documents: Handled through python-docx for DOCX format support
• Plain text: Processed using built-in I/O operations for maximum efficiency

This dispatch mechanism ensures that each file format receives specialized handling while presenting a unified interface to calling code.

JSON Output Structure

Regardless of input format, the extraction layer returns a standardized JSON envelope that downstream tools consume. The parser yields a plain-text string wrapped in a dictionary structure:

semantica parse path/to/document.pdf

# Output: {"text":"... extracted plain-text content ..."}

This consistent output format allows subsequent pipeline stages—such as the split and embed commands—to operate without format-specific branching logic.

Graph Normalization Pipeline

Once text is extracted and processed, Semantica frequently ingests graph data from various exporters that may use divergent field names. The normalization layer reconciles these differences into a single predictable schema.

The normalize_graph_payload Function

Located in semantica/utils/helpers.py, the normalize_graph_payload function accepts heterogeneous graph payloads and reduces them to a canonical vocabulary. The function signature targets dictionaries containing mixed nomenclature:

from semantica.utils.helpers import normalize_graph_payload

payload = {
    "nodes": [{"id": "n1", "label": "Person"}],
    "edges": [{"source": "n1", "target": "n2", "type": "knows"}],
}
normalized = normalize_graph_payload(payload)

# Result: {"entities": [...], "relationships": [...], "triplets": []}

The function handles three key mappings:

• entities vs. nodes
• relationships vs. edges
• triplets for raw subject-predicate-object triples

Validation and Error Handling

The normalization function implements strict validation logic (lines 01038-01057 in the source) to prevent silent data loss. It raises ValidationError with descriptive messages when encountering:

• Non-dictionary payloads
• Recognized keys mapping to non-collection values
• Divergent data present under both spellings of the same collection (e.g., both nodes and entities with different contents)
• Payloads containing data but lacking recognized keys

from semantica.utils.helpers import normalize_graph_payload, ValidationError

try:
    normalize_graph_payload({"nodes": "not a list"})
except ValidationError as exc:
    print(exc)  # Clear message about expected collection type

Canonical Schema Enforcement

By enforcing a single schema where all graph data uses entities, relationships, and triplets, the normalization layer eliminates defensive coding in downstream components. Every consumer—from visualizers to export utilities—can assume consistent field names regardless of the data's origin.

Integration Across the Codebase

The normalization utility sees widespread adoption across Semantica's visualization and export modules. In semantica/visualization/kg_visualizer.py, the normalize_graph_payload function is called before rendering knowledge graphs to ensure consistent node and edge processing. Similarly, semantica/visualization/utils/export_formats.py relies on the normalized payload when generating CSV, JSON, and Parquet exports, preventing format-specific transformation logic from proliferating throughout the codebase.

Summary

• Text parsing in Semantica operates through MIME-type detection in semantica/mcp/tools/extraction.py, supporting PDF, DOCX, and TXT formats via specialized parsers while outputting a uniform {"text": "..."} JSON structure.
• Graph normalization utilizes normalize_graph_payload in semantica/utils/helpers.py to reconcile heterogeneous field names (entities/nodes, relationships/edges) into a canonical schema with strict validation.
• The CLI entry point in semantica/mcp/__main__.py provides the parse command interface, while downstream tools like the KG visualizer and export formatters consume normalized data.
• Validation errors are explicit and descriptive, preventing silent data corruption when processing malformed graph payloads.

Frequently Asked Questions

How do I parse a PDF file using Semantica's CLI?

Execute the parse command with the file path as an argument: semantica parse path/to/document.pdf. The command, registered in semantica/mcp/__main__.py, delegates to semantica/mcp/tools/extraction.py, which selects the PDF parser based on MIME type and returns a JSON object containing the extracted text.

What schema does normalize_graph_payload enforce for graph data?

The function enforces a canonical schema using three keys: entities (formerly nodes), relationships (formerly edges), and triplets (for subject-predicate-object representations). According to the implementation in semantica/utils/helpers.py, the function resolves duplicate spellings and raises errors if both variants contain divergent data.

How does Semantica handle invalid graph payloads?

The normalize_graph_payload function raises ValidationError with specific messages when encountering non-dictionary inputs, non-collection values for recognized keys, or payloads missing all recognized keys. This strict validation, implemented around lines 01038-01057 in semantica/utils/helpers.py, prevents downstream components from processing malformed data.

Can I use the extraction tools programmatically instead of via CLI?

Yes, import extract_text directly from semantica/mcp/tools/extraction to use the parsing functionality within Python code. This bypasses the Click CLI layer while maintaining the same file-type detection and JSON output format used by the command-line interface.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}