How Semantica Converts Unstructured Data to Knowledge Graphs: A Technical Deep Dive

Semantica transforms unstructured text into structured knowledge graphs through a three-stage pipeline: extraction of entities and relations using LLMs and pattern matching, semantic deduplication via embeddings, and persistence to graph databases like Neo4j or FalkorDB.

Semantica is an open-source framework designed to bridge the gap between raw, unstructured text and structured knowledge representations. By treating documents, PDFs, and web pages as sources of latent semantic triples, the framework automates the conversion of unstructured data to knowledge graphs through a declarative, configuration-driven architecture. The entire workflow is orchestrated through the pipeline execution engine in semantica/pipeline/execution_engine.py, which coordinates extraction, normalization, and storage operations.

The Three-Stage Architecture for Unstructured Data Processing

Semantica processes unstructured data through a linear pipeline that progressively structures raw text into queryable knowledge graphs. The architecture separates concerns across extraction, enrichment, and persistence layers.

Stage 1: Extraction Layer

The semantic_extract/methods.py module serves as the central hub for entity, relation, and triplet extraction. When the pipeline receives input flagged as DataType.UNSTRUCTURED (defined in semantica/utils/types.py), it automatically triggers specialized extractors:

• Entity Extraction: Functions like extract_entities_llm, extract_entities_spacy, and extract_entities_pattern identify named entities using LLM providers (OpenAI, Groq, HuggingFace), spaCy ML models, or regex heuristics.
• Relation Extraction: Methods such as extract_relations_pattern and extract_relations_llm discover predicates between entities through dependency parsing, co-occurrence analysis, or generative LLM prompts.
• Triplet Extraction: The extract_triplets_* family combines entities and relations into subject-predicate-object structures suitable for graph storage.

Each extractor returns typed Entity or Relation objects defined in semantic_extract/types.py, ensuring consistent data structures downstream.

Stage 2: Normalization and Semantic Deduplication

Raw extraction often produces redundant or near-duplicate nodes. Semantica addresses this through a deduplication layer centered on semantic similarity:

The TextEmbedder class in embeddings/text_embedder.py generates high-dimensional vectors using models like BAAI/BGE-small for semantic comparison. The find_best_match_index function in semantic_extract/methods.py performs fast approximate matching against existing entities, while deduplication/similarity_calculator.py implements exact string matching, synonym resolution, and substring heuristics.

After deduplication, the calculate_weighted_confidence function re-scores entities and relations based on source reliability and similarity metrics, producing a canonical set of nodes and edges ready for persistence.

Stage 3: Graph Store Integration

The normalized data flows into the graph store abstraction layer defined in graph_store/graph_store.py. The GraphStoreRegistry in graph_store/registry.py instantiates concrete implementations such as graph_store/neo4j_store.py, graph_store/falkordb_store.py, or Amazon Neptune backends through a unified API.

The pipeline finalizer calls add_node() and add_edge() methods to materialize the knowledge graph, followed by commit() to persist transactions. For downstream consumption, visualization/kg_visualizer.py generates interactive HTML visualizations, while triplet_store/query_engine.py enables SPARQL querying against the constructed graph.

Entity and Relation Extraction Methods

Semantica supports multiple extraction strategies that can be mixed and matched via configuration. The framework implements graceful degradation—if an LLM provider is unavailable, the pipeline falls back to spaCy or pattern-based methods.

LLM-Driven Extraction provides the highest accuracy for complex domains. The extract_entities_llm function accepts a provider parameter (e.g., "openai", "groq") and optional model specification (e.g., "gpt-4o-mini"), returning typed entities with confidence scores.

Pattern-Based Extraction uses regex and dependency parsing for deterministic, high-speed processing. The extract_relations_pattern method analyzes sentence structure to identify subject-verb-object relationships without external API calls.

Hybrid Approaches combine both strategies, using fast pattern matching for obvious cases and LLM inference for ambiguous or complex constructions.

Configuring the Pipeline for Unstructured Inputs

Processing unstructured data requires setting the DataType.UNSTRUCTURED flag in semantic_extract/config.py. The pipeline engine automatically routes these inputs through the full extraction flow, bypassing structured data parsers.

from semantica.semantic_extract.config import ExtractionConfig, DataType
from semantica.pipeline.execution_engine import PipelineEngine

config = ExtractionConfig(
    data_type=DataType.UNSTRUCTURED,
    entity_extractor="llm",  # or "spacy", "pattern"

    relation_extractor="pattern",
    graph_store="neo4j"
)

engine = PipelineEngine(config)
result = engine.process(raw_text)

The execution_engine.py coordinates method selection, error handling, and transaction management across the extraction and storage phases.

Code Example: End-to-End Unstructured Processing

Below is a complete workflow demonstrating the conversion of raw text into a persisted knowledge graph:

# Load unstructured text

raw_text = """
Apple Inc. was founded by Steve Jobs in 1976.
The company is headquartered in Cupertino, California.
"""

# Stage 1: Extract entities using LLM

from semantica.semantic_extract.methods import extract_entities_llm
entities = extract_entities_llm(
    raw_text,
    provider="openai",
    model="gpt-4o-mini",
    entity_types=["PERSON", "ORG", "GPE", "DATE"]
)

# Extract relations using pattern matching

from semantica.semantic_extract.methods import extract_relations_pattern
relations = extract_relations_pattern(raw_text, entities)

# Build semantic triplets

from semantica.semantic_extract.methods import extract_triplets_pattern
triplets = extract_triplets_pattern(raw_text, entities, relations)

# Stage 3: Persist to Neo4j

from semantica.graph_store.registry import GraphStoreRegistry
store = GraphStoreRegistry.get_store("neo4j")
store.begin_transaction()

for entity in entities:
    store.add_node(
        entity.id, 
        label=entity.type, 
        properties={"text": entity.text, "confidence": entity.confidence}
    )

for relation in relations:
    store.add_edge(
        relation.subject.id, 
        relation.object.id, 
        label=relation.predicate,
        properties={"confidence": relation.confidence}
    )

store.commit()

# Visualize results

from semantica.visualization.kg_visualizer import KGVisualizer
viz = KGVisualizer(store)
viz.render()

Summary

• Semantica converts unstructured data to knowledge graphs through a three-stage pipeline implemented in semantica/pipeline/execution_engine.py: extraction, deduplication, and storage.
• The semantic_extract/methods.py module provides pluggable extractors for entities, relations, and triples, supporting LLM, ML, and pattern-based strategies.
• Semantic deduplication relies on embeddings/text_embedder.py and deduplication/similarity_calculator.py to resolve duplicates via vector similarity and string matching.
• Graph persistence uses the abstract graph_store/graph_store.py interface with concrete implementations for Neo4j, FalkorDB, and other backends.
• Configuration via DataType.UNSTRUCTURED triggers automatic pipeline routing for raw text inputs.

Frequently Asked Questions

What extraction methods does Semantica support for unstructured data?

Semantica supports four primary extraction paradigms: LLM-driven extraction (OpenAI, Groq, HuggingFace), spaCy ML models, rule-based pattern matching, and hybrid combinations. The semantic_extract/methods.py file implements specific functions like extract_entities_llm and extract_relations_pattern that can be configured per pipeline instance. The system automatically falls back to lighter methods when heavy dependencies or API keys are unavailable.

How does Semantica handle duplicate entities during knowledge graph construction?

The framework employs a multi-layered deduplication strategy. First, find_best_match_index in semantic_extract/methods.py performs fast candidate retrieval. Then, deduplication/similarity_calculator.py applies exact matching, synonym checks, and embedding similarity using the TextEmbedder class from embeddings/text_embedder.py. Finally, calculate_weighted_confidence merges duplicate records and recalculates aggregate confidence scores before graph insertion.

Which graph databases are compatible with Semantica?

Semantica supports multiple backends through its registry pattern in graph_store/registry.py. Concrete implementations include Neo4j (graph_store/neo4j_store.py), FalkorDB, Amazon Neptune, and other providers adhering to the abstract graph_store/graph_store.py interface. Each implementation provides unified methods: add_node(), add_edge(), and commit() for transaction management.

Can Semantica process PDFs and web pages, or only plain text?

While the core pipeline operates on DataType.UNSTRUCTURED text as defined in utils/types.py, Semantica handles PDFs, web pages, and documents by first converting them to plain text through upstream preprocessing. The extraction layer in semantic_extract/methods.py then processes this text through the standard entity and relation extraction workflow, making the pipeline agnostic to the original document format once text extraction has occurred.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}