How Semantica Transforms Unstructured Data into Knowledge Graphs: A Complete Technical Guide

Semantica converts raw text into structured knowledge graphs through a four-stage pipeline combining semantic extraction, coreference resolution, event detection, and graph construction.

The semantica-agi/semantica open-source project provides a modular, production-ready system for transforming free-form documents into queryable graph structures. This article examines the exact mechanisms—source file by source file—that enable this conversion.

The Core Pipeline Architecture

Semantica's transformation process operates as a layered pipeline where each stage refines unstructured input into increasingly structured representations. The pipeline is orchestrated through the Micro-Control-Plane (MCP) and exposed via tool functions in semantica_mcp/mcp/tools/extraction.py.

The four stages execute sequentially:

1. Named Entity Recognition (NER) identifies atomic entities
2. Coreference Resolution collapses multiple mentions
3. Relation & Event Extraction discovers connections and temporal structures
4. Graph Construction materializes nodes, edges, and provenance

Stage 1: Semantic Extraction

The extraction.py module contains three primary entry points that define the extraction surface.

Entity Extraction

The handle_extract_entities function (lines 23-44) executes Named-Entity Recognition to produce labeled spans:

from semantica_mcp.mcp.server import SemanticaMCP

mcp = SemanticaMCP()
entities = mcp.run_tool("extract_entities", {
    "text": "Apple Inc. announced quarterly earnings in Cupertino."
})

# Returns: [{text: "Apple Inc.", type: "ORG", offset: 0}, ...]

Each entity carries provenance metadata: extraction method, confidence score, and source text offsets.

Relation and Triplet Extraction

The handle_extract_relations function (lines 50-78) performs two-phase extraction:

• First, re-runs NER to ensure entity coverage
• Then feeds entities to RelationExtractor (source → target pairs) and TripletExtractor (subject-predicate-object structures)

Relations capture typed connections between entities, while triplets provide semantic (RDF-style) statements suitable for ontology alignment.

Full Extraction Stack

The handle_extract_all function (lines 87-136) executes the complete pipeline:

text = """
Alice bought a new laptop from Acme Corp. She loves its performance, 
but the battery life is disappointing. Bob, her colleague, recommended a different model.
"""

payload = mcp.run_tool("extract_all", {"text": text})

This invokes NER → CoreferenceResolver → RelationExtractor → EventDetector (optional) → TripletExtractor, returning a unified payload with all extract types.

Stage 2: Coreference Resolution and Event Detection

Two optional but critical components normalize extracted content before graph construction.

Coreference Resolution

The CoreferenceResolver (implemented in semantica/semantic_extract/coreference.py) collapses pronoun chains and repeated mentions. In the example above, "She" and "its" are resolved to "Alice" and "laptop" respectively, ensuring a single node represents each real-world entity.

Event Detection

The EventDetector identifies temporal structures—actions with triggers, participants, and time expressions. Events become first-class nodes in the graph with specialized linking to their arguments.

Stage 3: Graph Construction

The semantica/context/context_graph.py module materializes extracted structures into a property graph model.

Node Creation

Every unique entity becomes a node storing:

• Canonical label (post-coreference)
• Semantic type (PERSON, ORGANIZATION, CONCEPT, etc.)
• Provenance metadata (extraction source, confidence, original spans)

Edge Construction

Relations generate typed edges:

• Binary relations: source → target with relation type label
• Triplet edges: subject → object labeled by predicate
• Event linkages: event nodes connected to participant entities

Building from Extraction Payload

# payload from handle_extract_all

graph = mcp.context_graph_from_payload(payload)

print("Nodes:", list(graph.nodes(data=True)))
print("Edges:", list(graph.edges(data=True)))

The resulting ContextGraph supports in-memory operations, traversal queries, and export to persistent stores.

Stage 4: Persistence and Storage

Semantica abstracts graph storage through adapters in semantica/vector_store/*, enabling multiple backend strategies:

Backend	Use Case
In-memory (ContextGraph)	Rapid prototyping, small-scale analysis
Neo4j	Production graph queries, Cypher support
Amazon Neptune	AWS-native, RDF/SPARQL compatibility
Vector stores with embeddings	Semantic similarity search on graph nodes

Persisting to Neo4j

from semantica.context.graph_backends import Neo4jBackend

neo = Neo4jBackend(
    uri="bolt://localhost:7687", 
    auth=("neo4j", "password")
)
neo.save_graph(graph)

Provenance Tracking and Auditability

Every graph element carries provenance metadata managed through semantica/visualization/visualization_provenance.py. This records:

• Which extraction activity produced the node/edge
• Confidence scores and alternative interpretations
• Source text references

This audit trail enables:

• Visualization of extraction uncertainty
• "Right to be forgotten" operations (remove all data derived from specific text spans)
• Debugging and pipeline improvement (identify low-confidence extraction patterns)

Custom Extension Points

The modular architecture supports component replacement without pipeline restructuring:

# Adding domain-specific edges post-extraction

graph.add_node("Acme Corp", type="organization", source="domain_kb")
graph.add_edge(
    "Alice", 
    "Acme Corp", 
    relation="employee_of",
    provenance={"method": "hr_database_lookup"}
)

Swappable components include:

• NER engine (spaCy, transformers, custom models)
• Relation extractor (rule-based, supervised, or LLM-powered)
• Graph backend (any adapter implementing the storage interface)

Summary

• Semantica's extraction pipeline in semantica_mcp/mcp/tools/extraction.py provides three tool functions (handle_extract_entities, handle_extract_relations, handle_extract_all) with progressive depth of analysis.
• Coreference resolution and event detection normalize and enrich extracts before graph construction.
• The ContextGraph class in semantica/context/context_graph.py materializes nodes and edges with full provenance tracking.
• Storage adapters in semantica/vector_store/* enable persistence to memory, Neo4j, Neptune, or embedded vector stores.
• Provenance records in semantica/visualization/visualization_provenance.py ensure the knowledge graph remains auditable and compliant.

Frequently Asked Questions

What file formats can Semantica ingest?

Semantica processes raw text strings as primary input. The MCP tool interface accepts JSON payloads with text fields; preprocessing for PDF, HTML, or DOCX must occur upstream. The extraction engine operates on cleaned UTF-8 text regardless of original format.

How does Semantica handle ambiguous entity mentions?

The CoreferenceResolver (called within handle_extract_all) applies mention clustering to link pronouns and variants to canonical entities. Post-resolution, the graph contains one node per real-world referent. Confidence scores from the resolver are preserved in edge metadata for downstream filtering.

Can I use a custom NER model with Semantica?

Yes. The SemanticaMCP class accepts configurable extractors at initialization. Replace the default NER by passing a custom extractor implementing the EntityExtractor interface to mcp.register_extractor("entities", custom_ner). The pipeline structure in handle_extract_all remains unchanged.

What is the performance characteristic for large documents?

Processing scales with extraction depth. Entity-only extraction (lines 23-44) runs in linear time. Full pipeline with coreference and event detection (lines 87-136) requires quadratic complexity in the worst case due to pairwise relation scoring. For production workloads, chunk documents at paragraph or section boundaries and merge graphs at entity identity boundaries.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}