Core Components of Semantica's Data Flow Pipeline: From Ingestion to Hybrid Retrieval

Semantica's data flow pipeline processes raw decisions through eight distinct stages—ingestion, semantic extraction, dual embedding generation, storage, and hybrid retrieval—to enable AI-powered decision management and analytics.

Semantica (semantica-agi/semantica) is an open-source framework designed for decision-oriented data processing and semantic analytics. Understanding the core components of Semantica's data flow pipeline is essential for implementing effective knowledge graph construction, vector search, and decision intelligence workflows.

1. Ingestion and Context Creation

The pipeline begins with the DecisionContext class in semantica/context/decision_context.py. This component accepts raw text or structured records and encapsulates them with essential metadata.

Key responsibilities include:

• Storing the original payload alongside timestamps and provenance information
• Generating unique identifiers for tracking decisions through the pipeline
• Providing the standardized entry point for all downstream processing

2. Semantic Extraction

Once ingested, raw decision text undergoes transformation via LLM-powered extractors located in semantica/semantic_extract/. These modules convert unstructured text into structured knowledge representations.

The extraction layer supports multiple output formats:

• SQL and SPARQL queries for database integration
• RDF triples and semantic relations for knowledge graph construction
• Structured entity-relationship maps that preserve decision logic

3. Dual Embedding Generation

Semantica generates two complementary vector representations in semantica/vector_store/decision_embedding_pipeline.py:

Semantic Embeddings are dense vectors generated via LLM providers (OpenAI, Groq) that capture the conceptual meaning of the decision text.

Structural Embeddings are graph-based vectors derived from node2vec-style algorithms applied to the knowledge graph produced during extraction. These capture topological relationships and community structures within the decision network.

4. Pipeline Orchestration

The DecisionEmbeddingPipeline class orchestrates the fusion of dual embeddings. Implemented in semantica/vector_store/decision_embedding_pipeline.py, this component manages:

• Configurable weighting via semantic_weight and structural_weight parameters
• The process_decision() method that executes extraction, embedding generation, and enrichment
• Coordination between vector and graph storage backends

5. Storage Architecture

The pipeline persists data through two specialized storage layers defined in semantica/vector_store/vector_store.py:

VectorStore maintains the combined embeddings for high-performance similarity search. The implementation supports SQLite-backed persistence for local deployments.

GraphStore persists the complete knowledge graph using backends like Neo4j, Amazon Neptune, or RDF triple stores. This layer provides graph algorithms (shortest-path, community detection, centrality analysis) essential for generating structural embeddings.

6. Hybrid Retrieval

Retrieval operations leverage the HybridSimilarityCalculator in semantica/vector_store/hybrid_similarity_calculator.py. This component merges:

• Cosine similarity scores from semantic embeddings
• Graph-based similarity scores derived from topological relationships

The find_similar_decisions() method enables retrieval of historically relevant decisions by combining both similarity signals into a unified ranking.

7. Downstream Processing and Visualization

Processed decisions flow into the MCP (Meta-Control-Plane) orchestration layer located in semantica/mcp/server.py, which coordinates multi-step workflows and provides the RPC/HTTP interface for external agents.

The Visualization subsystem in semantica/visualization/ renders:

• Provenance graphs showing decision lineage
• Temporal evolution of decision networks
• Semantic network layouts with interactive exploration

Export utilities support data exchange formats including Parquet and Arrow for integration with external analytics tools.

Implementation Example

The following example demonstrates driving the complete pipeline from context creation through similarity retrieval:

from semantica.vector_store import VectorStore, DecisionEmbeddingPipeline
from semantica.context import DecisionContext

# 1. Create a context from raw decision text

raw_decision = """
Our product team should adopt feature X to improve conversion rates.
"""
ctx = DecisionContext(payload=raw_decision, source="email")

# 2. Initialise the vector store (SQLite-backed) and optionally a graph store

vs = VectorStore(persist_path="data/vectors.db")
gs = None  # e.g. an AmazonNeptuneGraphStore() if you need KG algorithms

# 3. Build the embedding pipeline

pipeline = DecisionEmbeddingPipeline(
    vector_store=vs,
    graph_store=gs,
    semantic_weight=0.7,
    structural_weight=0.3,
)

# 4. Process the decision (extract → embed → store)

enriched = pipeline.process_decision(ctx)

# 5. Find similar past decisions

similar = pipeline.find_similar_decisions(
    decision=enriched,
    top_k=5,
)

print("Top-5 similar decisions:")
for s in similar:
    print(f"- {s['payload'][:80]}…")

Visualization Example

To render the provenance graph of a processed decision:

from semantica.visualization import KGVisualizer

visualizer = KGVisualizer(graph_store=gs)
visualizer.render(decision_id=enriched.id, output_path="graph.html")

Summary

• DecisionContext in semantica/context/decision_context.py serves as the standardized entry point for raw decision data
• Semantic extraction transforms text into structured knowledge graphs using LLM-powered extractors in semantica/semantic_extract/
• Dual embeddings (semantic and structural) are generated and blended in semantica/vector_store/decision_embedding_pipeline.py with configurable weights
• VectorStore and GraphStore in semantica/vector_store/vector_store.py provide persistent storage for embeddings and knowledge graphs
• HybridSimilarityCalculator enables retrieval by combining cosine similarity with graph-based topological analysis
• The MCP layer and Visualization tools complete the pipeline by enabling workflow orchestration and interactive graph rendering

Frequently Asked Questions

What is the difference between semantic and structural embeddings in Semantica?

Semantic embeddings are dense vectors generated by LLMs (such as OpenAI or Groq models) that capture the conceptual meaning of decision text. Structural embeddings are derived from graph algorithms (node2vec-style) applied to the knowledge graph topology, capturing relationship patterns and community structures. The pipeline combines both using configurable weights to enhance retrieval accuracy.

How does the DecisionEmbeddingPipeline handle storage backend configuration?

The DecisionEmbeddingPipeline class accepts vector_store and graph_store parameters during initialization. The VectorStore typically uses SQLite for local deployments as implemented in semantica/vector_store/vector_store.py, while the GraphStore can connect to Neo4j, Amazon Neptune, or other graph databases. If no graph store is provided, the pipeline operates in semantic-only mode.

Where does the knowledge graph extraction logic reside?

Knowledge graph extraction logic is implemented in semantica/semantic_extract/ and related extractor modules, which contain LLM-powered extractors capable of generating SQL, SPARQL, RDF triples, and entity relations. These extractors transform raw decision text into structured graph representations used for both storage and structural embedding generation.

How does hybrid similarity calculation improve decision retrieval?

Hybrid similarity calculation, implemented in semantica/vector_store/hybrid_similarity_calculator.py, improves retrieval by combining cosine similarity from semantic embeddings with topological similarity scores from the knowledge graph. This dual approach captures both conceptual relevance (what the decision means) and relational context (how the decision connects to others), providing more comprehensive results than vector-only search.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}