# How Semantica Converts Unstructured Data to Knowledge Graphs: A Technical Deep Dive

> Discover how Semantica converts unstructured data to knowledge graphs using LLMs, embeddings, and graph databases. Learn about entity extraction, semantic deduplication, and persistence.

- Repository: [Semantica /semantica](https://github.com/semantica-agi/semantica)
- Tags: deep-dive
- Published: 2026-08-22

---

**Semantica transforms unstructured text into structured knowledge graphs through a three-stage pipeline: extraction of entities and relations using LLMs and pattern matching, semantic deduplication via embeddings, and persistence to graph databases like Neo4j or FalkorDB.**

Semantica is an open-source framework designed to bridge the gap between raw, unstructured text and structured knowledge representations. By treating documents, PDFs, and web pages as sources of latent semantic triples, the framework automates the conversion of unstructured data to knowledge graphs through a declarative, configuration-driven architecture. The entire workflow is orchestrated through the pipeline execution engine in [`semantica/pipeline/execution_engine.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/pipeline/execution_engine.py), which coordinates extraction, normalization, and storage operations.

## The Three-Stage Architecture for Unstructured Data Processing

Semantica processes unstructured data through a linear pipeline that progressively structures raw text into queryable knowledge graphs. The architecture separates concerns across extraction, enrichment, and persistence layers.

### Stage 1: Extraction Layer

The [`semantic_extract/methods.py`](https://github.com/semantica-agi/semantica/blob/main/semantic_extract/methods.py) module serves as the central hub for entity, relation, and triplet extraction. When the pipeline receives input flagged as `DataType.UNSTRUCTURED` (defined in [`semantica/utils/types.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/utils/types.py)), it automatically triggers specialized extractors:

- **Entity Extraction**: Functions like `extract_entities_llm`, `extract_entities_spacy`, and `extract_entities_pattern` identify named entities using LLM providers (OpenAI, Groq, HuggingFace), spaCy ML models, or regex heuristics.
- **Relation Extraction**: Methods such as `extract_relations_pattern` and `extract_relations_llm` discover predicates between entities through dependency parsing, co-occurrence analysis, or generative LLM prompts.
- **Triplet Extraction**: The `extract_triplets_*` family combines entities and relations into subject-predicate-object structures suitable for graph storage.

Each extractor returns typed `Entity` or `Relation` objects defined in [`semantic_extract/types.py`](https://github.com/semantica-agi/semantica/blob/main/semantic_extract/types.py), ensuring consistent data structures downstream.

### Stage 2: Normalization and Semantic Deduplication

Raw extraction often produces redundant or near-duplicate nodes. Semantica addresses this through a deduplication layer centered on semantic similarity:

The `TextEmbedder` class in [`embeddings/text_embedder.py`](https://github.com/semantica-agi/semantica/blob/main/embeddings/text_embedder.py) generates high-dimensional vectors using models like BAAI/BGE-small for semantic comparison. The `find_best_match_index` function in [`semantic_extract/methods.py`](https://github.com/semantica-agi/semantica/blob/main/semantic_extract/methods.py) performs fast approximate matching against existing entities, while [`deduplication/similarity_calculator.py`](https://github.com/semantica-agi/semantica/blob/main/deduplication/similarity_calculator.py) implements exact string matching, synonym resolution, and substring heuristics.

After deduplication, the `calculate_weighted_confidence` function re-scores entities and relations based on source reliability and similarity metrics, producing a canonical set of nodes and edges ready for persistence.

### Stage 3: Graph Store Integration

The normalized data flows into the graph store abstraction layer defined in [`graph_store/graph_store.py`](https://github.com/semantica-agi/semantica/blob/main/graph_store/graph_store.py). The `GraphStoreRegistry` in [`graph_store/registry.py`](https://github.com/semantica-agi/semantica/blob/main/graph_store/registry.py) instantiates concrete implementations such as [`graph_store/neo4j_store.py`](https://github.com/semantica-agi/semantica/blob/main/graph_store/neo4j_store.py), [`graph_store/falkordb_store.py`](https://github.com/semantica-agi/semantica/blob/main/graph_store/falkordb_store.py), or Amazon Neptune backends through a unified API.

The pipeline finalizer calls `add_node()` and `add_edge()` methods to materialize the knowledge graph, followed by `commit()` to persist transactions. For downstream consumption, [`visualization/kg_visualizer.py`](https://github.com/semantica-agi/semantica/blob/main/visualization/kg_visualizer.py) generates interactive HTML visualizations, while [`triplet_store/query_engine.py`](https://github.com/semantica-agi/semantica/blob/main/triplet_store/query_engine.py) enables SPARQL querying against the constructed graph.

## Entity and Relation Extraction Methods

Semantica supports multiple extraction strategies that can be mixed and matched via configuration. The framework implements graceful degradation—if an LLM provider is unavailable, the pipeline falls back to spaCy or pattern-based methods.

**LLM-Driven Extraction** provides the highest accuracy for complex domains. The `extract_entities_llm` function accepts a `provider` parameter (e.g., `"openai"`, `"groq"`) and optional `model` specification (e.g., `"gpt-4o-mini"`), returning typed entities with confidence scores.

**Pattern-Based Extraction** uses regex and dependency parsing for deterministic, high-speed processing. The `extract_relations_pattern` method analyzes sentence structure to identify subject-verb-object relationships without external API calls.

**Hybrid Approaches** combine both strategies, using fast pattern matching for obvious cases and LLM inference for ambiguous or complex constructions.

## Configuring the Pipeline for Unstructured Inputs

Processing unstructured data requires setting the `DataType.UNSTRUCTURED` flag in [`semantic_extract/config.py`](https://github.com/semantica-agi/semantica/blob/main/semantic_extract/config.py). The pipeline engine automatically routes these inputs through the full extraction flow, bypassing structured data parsers.

```python
from semantica.semantic_extract.config import ExtractionConfig, DataType
from semantica.pipeline.execution_engine import PipelineEngine

config = ExtractionConfig(
    data_type=DataType.UNSTRUCTURED,
    entity_extractor="llm",  # or "spacy", "pattern"

    relation_extractor="pattern",
    graph_store="neo4j"
)

engine = PipelineEngine(config)
result = engine.process(raw_text)

```

The [`execution_engine.py`](https://github.com/semantica-agi/semantica/blob/main/execution_engine.py) coordinates method selection, error handling, and transaction management across the extraction and storage phases.

## Code Example: End-to-End Unstructured Processing

Below is a complete workflow demonstrating the conversion of raw text into a persisted knowledge graph:

```python

# Load unstructured text

raw_text = """
Apple Inc. was founded by Steve Jobs in 1976.
The company is headquartered in Cupertino, California.
"""

# Stage 1: Extract entities using LLM

from semantica.semantic_extract.methods import extract_entities_llm
entities = extract_entities_llm(
    raw_text,
    provider="openai",
    model="gpt-4o-mini",
    entity_types=["PERSON", "ORG", "GPE", "DATE"]
)

# Extract relations using pattern matching

from semantica.semantic_extract.methods import extract_relations_pattern
relations = extract_relations_pattern(raw_text, entities)

# Build semantic triplets

from semantica.semantic_extract.methods import extract_triplets_pattern
triplets = extract_triplets_pattern(raw_text, entities, relations)

# Stage 3: Persist to Neo4j

from semantica.graph_store.registry import GraphStoreRegistry
store = GraphStoreRegistry.get_store("neo4j")
store.begin_transaction()

for entity in entities:
    store.add_node(
        entity.id, 
        label=entity.type, 
        properties={"text": entity.text, "confidence": entity.confidence}
    )

for relation in relations:
    store.add_edge(
        relation.subject.id, 
        relation.object.id, 
        label=relation.predicate,
        properties={"confidence": relation.confidence}
    )

store.commit()

# Visualize results

from semantica.visualization.kg_visualizer import KGVisualizer
viz = KGVisualizer(store)
viz.render()

```

## Summary

- Semantica converts unstructured data to knowledge graphs through a three-stage pipeline implemented in [`semantica/pipeline/execution_engine.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/pipeline/execution_engine.py): extraction, deduplication, and storage.
- The [`semantic_extract/methods.py`](https://github.com/semantica-agi/semantica/blob/main/semantic_extract/methods.py) module provides pluggable extractors for entities, relations, and triples, supporting LLM, ML, and pattern-based strategies.
- Semantic deduplication relies on [`embeddings/text_embedder.py`](https://github.com/semantica-agi/semantica/blob/main/embeddings/text_embedder.py) and [`deduplication/similarity_calculator.py`](https://github.com/semantica-agi/semantica/blob/main/deduplication/similarity_calculator.py) to resolve duplicates via vector similarity and string matching.
- Graph persistence uses the abstract [`graph_store/graph_store.py`](https://github.com/semantica-agi/semantica/blob/main/graph_store/graph_store.py) interface with concrete implementations for Neo4j, FalkorDB, and other backends.
- Configuration via `DataType.UNSTRUCTURED` triggers automatic pipeline routing for raw text inputs.

## Frequently Asked Questions

### What extraction methods does Semantica support for unstructured data?

Semantica supports four primary extraction paradigms: **LLM-driven extraction** (OpenAI, Groq, HuggingFace), **spaCy ML models**, **rule-based pattern matching**, and **hybrid combinations**. The [`semantic_extract/methods.py`](https://github.com/semantica-agi/semantica/blob/main/semantic_extract/methods.py) file implements specific functions like `extract_entities_llm` and `extract_relations_pattern` that can be configured per pipeline instance. The system automatically falls back to lighter methods when heavy dependencies or API keys are unavailable.

### How does Semantica handle duplicate entities during knowledge graph construction?

The framework employs a multi-layered deduplication strategy. First, `find_best_match_index` in [`semantic_extract/methods.py`](https://github.com/semantica-agi/semantica/blob/main/semantic_extract/methods.py) performs fast candidate retrieval. Then, [`deduplication/similarity_calculator.py`](https://github.com/semantica-agi/semantica/blob/main/deduplication/similarity_calculator.py) applies exact matching, synonym checks, and embedding similarity using the `TextEmbedder` class from [`embeddings/text_embedder.py`](https://github.com/semantica-agi/semantica/blob/main/embeddings/text_embedder.py). Finally, `calculate_weighted_confidence` merges duplicate records and recalculates aggregate confidence scores before graph insertion.

### Which graph databases are compatible with Semantica?

Semantica supports multiple backends through its registry pattern in [`graph_store/registry.py`](https://github.com/semantica-agi/semantica/blob/main/graph_store/registry.py). Concrete implementations include **Neo4j** ([`graph_store/neo4j_store.py`](https://github.com/semantica-agi/semantica/blob/main/graph_store/neo4j_store.py)), **FalkorDB**, **Amazon Neptune**, and other providers adhering to the abstract [`graph_store/graph_store.py`](https://github.com/semantica-agi/semantica/blob/main/graph_store/graph_store.py) interface. Each implementation provides unified methods: `add_node()`, `add_edge()`, and `commit()` for transaction management.

### Can Semantica process PDFs and web pages, or only plain text?

While the core pipeline operates on `DataType.UNSTRUCTURED` text as defined in [`utils/types.py`](https://github.com/semantica-agi/semantica/blob/main/utils/types.py), Semantica handles PDFs, web pages, and documents by first converting them to plain text through upstream preprocessing. The extraction layer in [`semantic_extract/methods.py`](https://github.com/semantica-agi/semantica/blob/main/semantic_extract/methods.py) then processes this text through the standard entity and relation extraction workflow, making the pipeline agnostic to the original document format once text extraction has occurred.