# Core Components of Semantica's Data Flow Pipeline: From Ingestion to Hybrid Retrieval

> Discover Semantica's data flow pipeline, from ingestion to hybrid retrieval. Learn how eight stages process decisions for AI-powered management and analytics.

- Repository: [Semantica /semantica](https://github.com/semantica-agi/semantica)
- Tags: deep-dive
- Published: 2026-08-22

---

**Semantica's data flow pipeline processes raw decisions through eight distinct stages—ingestion, semantic extraction, dual embedding generation, storage, and hybrid retrieval—to enable AI-powered decision management and analytics.**

Semantica (semantica-agi/semantica) is an open-source framework designed for decision-oriented data processing and semantic analytics. Understanding the core components of Semantica's data flow pipeline is essential for implementing effective knowledge graph construction, vector search, and decision intelligence workflows.

## 1. Ingestion and Context Creation

The pipeline begins with the **DecisionContext** class in [`semantica/context/decision_context.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/context/decision_context.py). This component accepts raw text or structured records and encapsulates them with essential metadata.

Key responsibilities include:
- Storing the original payload alongside timestamps and provenance information
- Generating unique identifiers for tracking decisions through the pipeline
- Providing the standardized entry point for all downstream processing

## 2. Semantic Extraction

Once ingested, raw decision text undergoes transformation via LLM-powered **extractors** located in `semantica/semantic_extract/`. These modules convert unstructured text into structured knowledge representations.

The extraction layer supports multiple output formats:
- SQL and SPARQL queries for database integration
- RDF triples and semantic relations for knowledge graph construction
- Structured entity-relationship maps that preserve decision logic

## 3. Dual Embedding Generation

Semantica generates two complementary vector representations in [`semantica/vector_store/decision_embedding_pipeline.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/vector_store/decision_embedding_pipeline.py):

**Semantic Embeddings** are dense vectors generated via LLM providers (OpenAI, Groq) that capture the conceptual meaning of the decision text.

**Structural Embeddings** are graph-based vectors derived from node2vec-style algorithms applied to the knowledge graph produced during extraction. These capture topological relationships and community structures within the decision network.

## 4. Pipeline Orchestration

The **DecisionEmbeddingPipeline** class orchestrates the fusion of dual embeddings. Implemented in [`semantica/vector_store/decision_embedding_pipeline.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/vector_store/decision_embedding_pipeline.py), this component manages:

- Configurable weighting via `semantic_weight` and `structural_weight` parameters
- The `process_decision()` method that executes extraction, embedding generation, and enrichment
- Coordination between vector and graph storage backends

## 5. Storage Architecture

The pipeline persists data through two specialized storage layers defined in [`semantica/vector_store/vector_store.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/vector_store/vector_store.py):

**VectorStore** maintains the combined embeddings for high-performance similarity search. The implementation supports SQLite-backed persistence for local deployments.

**GraphStore** persists the complete knowledge graph using backends like Neo4j, Amazon Neptune, or RDF triple stores. This layer provides graph algorithms (shortest-path, community detection, centrality analysis) essential for generating structural embeddings.

## 6. Hybrid Retrieval

Retrieval operations leverage the **HybridSimilarityCalculator** in [`semantica/vector_store/hybrid_similarity_calculator.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/vector_store/hybrid_similarity_calculator.py). This component merges:
- Cosine similarity scores from semantic embeddings
- Graph-based similarity scores derived from topological relationships

The `find_similar_decisions()` method enables retrieval of historically relevant decisions by combining both similarity signals into a unified ranking.

## 7. Downstream Processing and Visualization

Processed decisions flow into the **MCP (Meta-Control-Plane)** orchestration layer located in [`semantica/mcp/server.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/mcp/server.py), which coordinates multi-step workflows and provides the RPC/HTTP interface for external agents.

The **Visualization** subsystem in `semantica/visualization/` renders:
- Provenance graphs showing decision lineage
- Temporal evolution of decision networks
- Semantic network layouts with interactive exploration

Export utilities support data exchange formats including Parquet and Arrow for integration with external analytics tools.

## Implementation Example

The following example demonstrates driving the complete pipeline from context creation through similarity retrieval:

```python
from semantica.vector_store import VectorStore, DecisionEmbeddingPipeline
from semantica.context import DecisionContext

# 1. Create a context from raw decision text

raw_decision = """
Our product team should adopt feature X to improve conversion rates.
"""
ctx = DecisionContext(payload=raw_decision, source="email")

# 2. Initialise the vector store (SQLite-backed) and optionally a graph store

vs = VectorStore(persist_path="data/vectors.db")
gs = None  # e.g. an AmazonNeptuneGraphStore() if you need KG algorithms

# 3. Build the embedding pipeline

pipeline = DecisionEmbeddingPipeline(
    vector_store=vs,
    graph_store=gs,
    semantic_weight=0.7,
    structural_weight=0.3,
)

# 4. Process the decision (extract → embed → store)

enriched = pipeline.process_decision(ctx)

# 5. Find similar past decisions

similar = pipeline.find_similar_decisions(
    decision=enriched,
    top_k=5,
)

print("Top-5 similar decisions:")
for s in similar:
    print(f"- {s['payload'][:80]}…")

```

## Visualization Example

To render the provenance graph of a processed decision:

```python
from semantica.visualization import KGVisualizer

visualizer = KGVisualizer(graph_store=gs)
visualizer.render(decision_id=enriched.id, output_path="graph.html")

```

## Summary

- **DecisionContext** in [`semantica/context/decision_context.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/context/decision_context.py) serves as the standardized entry point for raw decision data
- **Semantic extraction** transforms text into structured knowledge graphs using LLM-powered extractors in `semantica/semantic_extract/`
- **Dual embeddings** (semantic and structural) are generated and blended in [`semantica/vector_store/decision_embedding_pipeline.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/vector_store/decision_embedding_pipeline.py) with configurable weights
- **VectorStore** and **GraphStore** in [`semantica/vector_store/vector_store.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/vector_store/vector_store.py) provide persistent storage for embeddings and knowledge graphs
- **HybridSimilarityCalculator** enables retrieval by combining cosine similarity with graph-based topological analysis
- The **MCP layer** and **Visualization** tools complete the pipeline by enabling workflow orchestration and interactive graph rendering

## Frequently Asked Questions

### What is the difference between semantic and structural embeddings in Semantica?

Semantic embeddings are dense vectors generated by LLMs (such as OpenAI or Groq models) that capture the conceptual meaning of decision text. Structural embeddings are derived from graph algorithms (node2vec-style) applied to the knowledge graph topology, capturing relationship patterns and community structures. The pipeline combines both using configurable weights to enhance retrieval accuracy.

### How does the DecisionEmbeddingPipeline handle storage backend configuration?

The `DecisionEmbeddingPipeline` class accepts `vector_store` and `graph_store` parameters during initialization. The VectorStore typically uses SQLite for local deployments as implemented in [`semantica/vector_store/vector_store.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/vector_store/vector_store.py), while the GraphStore can connect to Neo4j, Amazon Neptune, or other graph databases. If no graph store is provided, the pipeline operates in semantic-only mode.

### Where does the knowledge graph extraction logic reside?

Knowledge graph extraction logic is implemented in `semantica/semantic_extract/` and related extractor modules, which contain LLM-powered extractors capable of generating SQL, SPARQL, RDF triples, and entity relations. These extractors transform raw decision text into structured graph representations used for both storage and structural embedding generation.

### How does hybrid similarity calculation improve decision retrieval?

Hybrid similarity calculation, implemented in [`semantica/vector_store/hybrid_similarity_calculator.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/vector_store/hybrid_similarity_calculator.py), improves retrieval by combining cosine similarity from semantic embeddings with topological similarity scores from the knowledge graph. This dual approach captures both conceptual relevance (what the decision means) and relational context (how the decision connects to others), providing more comprehensive results than vector-only search.