How to Build Custom Extraction Pipelines with the semantic_extract Module in Semantica

The semantic_extract package provides a modular, provider-based architecture for constructing text-to-knowledge-graph pipelines, where each stage—entity recognition, relation extraction, and triplet construction—can be swapped or extended via factory methods in semantica/semantic_extract/methods.py.

The semantic_extract module is the core extraction engine of the semantica-agi/semantica repository. It transforms unstructured text into structured knowledge graphs through a composable pipeline of specialized extractors. This guide demonstrates how to leverage its factory-based design to build custom extraction pipelines tailored to your domain.

Architecture of the semantic_extract Module

The module follows a layered architecture with clear extension points at each stage:

Layer	Responsibility	Key Class	Extension Method
Entity Extraction	Detect named entities (people, organizations, locations, custom types)	NERExtractor in semantica/semantic_extract/ner_extractor.py	Provide a custom SpaCy model, Hugging Face pipeline, or rule-based extractor
Relation Extraction	Identify semantic relationships between entities	RelationExtractor in semantica/semantic_extract/relation_extractor.py	Plug in an LLM-based scorer, pattern matcher, or graph heuristic
Triplet Construction	Assemble (subject, predicate, object) triples	TripletExtractor in semantica/semantic_extract/triplet_extractor.py	Override stitching logic, add provenance, or change output format
Orchestration	Wire components together with lazy-evaluated factories	Method factories in semantica/semantic_extract/methods.py	Register custom providers via create_provider hooks

All layers are connected through method factories that read from semantica/semantic_extract/config.py and return concrete implementations matching the requested provider (e.g., OpenAIProvider, OllamaProvider, HuggingFaceProvider). This design enables runtime provider switching without code changes.

Step-by-Step: Building a Custom Pipeline

Follow this pattern to extend or replace any pipeline stage:

1. Update configuration in semantica/semantic_extract/config.py or pass parameters at runtime
2. Implement a subclass of the target extractor, preserving the public API (e.g., extract(text) → List[Entity])
3. Register the subclass in semantica/semantic_extract/methods.py via factory functions: get_entity_method(), get_relation_method(), or get_triplet_method()
4. Instantiate and run through SemanticNetworkExtractor or call individual extractors directly

Practical Code Examples

Use a Hugging Face NER Model Instead of Default SpaCy

Replace the default entity extractor with a Hugging Face transformer model by updating the config and retrieving the factory-produced instance:

from semantica.semantic_extract.methods import get_entity_method
from semantica.semantic_extract.config import Config

# Update configuration (or edit semantica/semantic_extract/config.py directly)

Config.update({"entity_provider": "huggingface", "hf_model": "dslim/bert-base-NER"})

# Factory returns HuggingFaceExtractor based on config

ner_extractor = get_entity_method()

text = "Ada Lovelace wrote the first algorithm for the Analytical Engine."
entities = ner_extractor.extract(text)
print(entities)

# → [Entity(text='Ada Lovelace', type='PERSON'), ...]

The get_entity_method() factory inspects Config and instantiates the appropriate provider class. Relevant source: semantica/semantic_extract/methods.py, semantica/semantic_extract/config.py.

Plug in an LLM-Based Relation Extractor

Create a custom provider subclass and pass it to the relation factory:

from semantica.semantic_extract.methods import get_relation_method
from semantica.semantic_extract.providers import OpenAIProvider

# Define custom provider with specific model

class MyOpenAIProvider(OpenAIProvider):
    model_name = "gpt-4o-mini"

# Factory uses custom provider class

relation_extractor = get_relation_method(provider=MyOpenAIProvider)

text = "Barack Obama was born in Honolulu and served as President of the United States."
relations = relation_extractor.extract(text)
print(relations)

# → [Relation(subject='Barack Obama', predicate='born_in', object='Honolulu'), ...]

The OpenAIProvider base class handles API authentication and response parsing; your subclass only needs to specify the model. Relevant source: semantica/semantic_extract/relation_extractor.py, semantica/semantic_extract/providers.py.

Assemble Full Pipeline and Export to SQLite

Combine all extractors and persist results to a local database:

from semantica.semantic_extract.semantic_network_extractor import SemanticNetworkExtractor
from semantica.mcp.tools.export import Exporter

# Build complete pipeline with default or configured components

pipeline = SemanticNetworkExtractor()

doc = """
Alice, a software engineer at Acme Corp, attended the AI conference in Berlin.
"""
graph = pipeline.extract(doc)

# Export to SQLite triple store

exporter = Exporter(store_type="sqlite", path="kg.db")
exporter.save(graph)  # Writes triples to kg.db

SemanticNetworkExtractor orchestrates the full extraction sequence: entities → relations → triplets → deduplication. Relevant source: semantica/semantic_extract/semantic_network_extractor.py, semantica/mcp/tools/export.py.

Key Source Files

• semantica/semantic_extract/ner_extractor.py — NERExtractor interface and implementations
• semantica/semantic_extract/relation_extractor.py — RelationExtractor for relationship detection
• semantica/semantic_extract/triplet_extractor.py — TripletExtractor for triple assembly
• semantica/semantic_extract/methods.py — Factory functions get_entity_method(), get_relation_method(), get_triplet_method()
• semantica/semantic_extract/config.py — Runtime configuration and provider resolution
• semantica/deduplication/ — Duplicate detection and merging utilities
• semantica/mcp/tools/export.py — Export adapters for SQLite, Neo4j, Pinecone, and other backends

Summary

• The semantic_extract module uses factory methods in methods.py to wire together swappable extraction components
• Configuration-driven instantiation allows runtime provider switching without code modification
• Three main extension points: entity extractors (NERExtractor), relation extractors (RelationExtractor), and triplet builders (TripletExtractor)
• Custom providers inherit from base classes like OpenAIProvider and are passed to factory functions
• Full pipelines are orchestrated by SemanticNetworkExtractor with export support via Exporter

Frequently Asked Questions

How do I switch between OpenAI and Ollama providers without changing code?

Update the provider key in semantica/semantic_extract/config.py or call Config.update({"provider": "ollama"}) at runtime. The factory methods in methods.py read this value and instantiate the matching provider class.

Can I combine multiple NER models in one pipeline?

Yes. Create a custom NERExtractor subclass that aggregates results from multiple underlying models, then register it in methods.py by overriding get_entity_method() to return your composite extractor when a specific config flag is set.

What format does TripletExtractor output?

By default, TripletExtractor in semantica/semantic_extract/triplet_extractor.py produces (subject, predicate, object) tuples with attached provenance metadata. You can override the extract() method to emit RDF triples, JSON-LD, or domain-specific formats.

Where does deduplication happen in the pipeline?

The SemanticNetworkExtractor class coordinates deduplication after triplet construction, using utilities from semantica/deduplication/. You can inject custom merging logic by subclassing DuplicateDetector and referencing it in the pipeline configuration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}