# How Semantica Performs Entity and Relation Extraction

> Discover how Semantica achieves entity and relation extraction using modular orchestrators, ML, LLMs, and weighted confidence scoring for accurate results.

- Repository: [Semantica /semantica](https://github.com/semantica-agi/semantica)
- Tags: how-to-guide
- Published: 2026-08-22

---

**Semantica performs entity and relation extraction through modular orchestrator classes that dispatch to multiple backend methods—ranging from deterministic regex patterns to transformer-based ML and LLM inference—while applying weighted confidence scoring and intelligent fallback chains.**

Semantica is an open-source semantic extraction framework designed for flexible natural language processing pipelines. The repository `semantica-agi/semantica` implements a layered architecture that decouples extraction logic from method-specific implementations, enabling seamless fallback between spaCy models, HuggingFace transformers, and large language providers. This article examines the concrete implementation details found in the source code, including the orchestration layer, method dispatch system, and confidence scoring algorithms.

## Modular Extraction Architecture

The extraction system is organized into four distinct layers that separate concerns between configuration, method registration, and execution logic.

| Layer | Responsibility | Core Module |
|-------|----------------|-------------|
| **Orchestrator classes** | Manage configuration, fallback chains, progress tracking, and ensemble voting. | `NERExtractor` • `RelationExtractor` |
| **Method registry** | Central lookup for extraction functions by name. | `method_registry` in [`registry.py`](https://github.com/semantica-agi/semantica/blob/main/registry.py) |
| **Extraction functions** | Implement concrete logic for each method. | [`methods.py`](https://github.com/semantica-agi/semantica/blob/main/methods.py) |
| **Shared utilities** | Caching, similarity matching, and confidence weighting. | [`cache.py`](https://github.com/semantica-agi/semantica/blob/main/cache.py) • [`utils/helpers.py`](https://github.com/semantica-agi/semantica/blob/main/utils/helpers.py) |

The two primary orchestrators—**`NERExtractor`** ([`semantica/semantic_extract/ner_extractor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/semantic_extract/ner_extractor.py)) and **`RelationExtractor`** ([`semantica/semantic_extract/relation_extractor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/semantic_extract/relation_extractor.py))—expose uniform `extract()` interfaces while internally coordinating multiple backend strategies. Both classes maintain a `self.method` list that defines the fallback priority, allowing users to specify primary and secondary extraction techniques.

## Entity Extraction Pipeline

### Entry Point and Method Dispatch

Entity extraction begins at `NERExtractor.extract_entities()` (aliased as `extract()`). The orchestrator first filters unavailable methods through `_filter_unusable_methods()`, which removes ML-dependent strategies if spaCy initialization fails. For each remaining method name in the configuration chain, the system calls `get_entity_method()` from [`methods.py`](https://github.com/semantica-agi/semantica/blob/main/methods.py) to retrieve the concrete implementation.

```python

# From semantica/semantic_extract/ner_extractor.py

for method_name in methods:
    method_func = get_entity_method(method_name)      # registry lookup

    entities = method_func(text, **method_options)    # concrete extraction

    if filtered := [e for e in entities if e.confidence >= min_confidence]:
        return filtered

```

### Extraction Methods

The framework supports six distinct extraction strategies, each implemented as a standalone function in [`methods.py`](https://github.com/semantica-agi/semantica/blob/main/methods.py):

- **`extract_entities_pattern`** – Deterministic regular-expression pipelines for high-precision matching
- **`extract_entities_regex`** – Generic regex-based entity detection
- **`extract_entities_rules`** – Rule-based grammar parsing
- **`extract_entities_ml`** – **spaCy ML** integration that lazily loads models via `load_spacy_model()` with process-level caching in `_spacy_model_cache`
- **`extract_entities_huggingface`** – **HuggingFace Transformers** inference using `HuggingFaceModelLoader` with manual IOB tag aggregation fallback
- **`extract_entities_llm`** – Provider-agnostic LLM extraction using OpenAI, Groq, or Gemini via `create_provider()`, enforcing JSON schema output through `EntitiesResponse`

### Confidence Weighting and Filtering

When users supply an `entity_types` whitelist, the system recalculates confidence scores using `calculate_weighted_confidence()`. This function blends the method's native confidence with semantic similarity to allowed types via `find_best_match_index()`, which implements a fast hybrid matcher combining exact lookup, synonym heuristics, embedding similarity, and fuzzy string matching.

```python

# Confidence re-scoring logic

for e in entities:
    e.confidence = calculate_weighted_confidence(
        item_type=e.label,
        original_confidence=e.confidence,
        valid_types=entity_types,
        item_text=e.text,
    )

```

If **ensemble voting** is enabled, results from multiple methods are combined through `_vote_entities()` rather than returning the first successful result.

## Relation Extraction Pipeline

### Pattern-Based Relation Matching

The `RelationExtractor` class ([`semantica/semantic_extract/relation_extractor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/semantic_extract/relation_extractor.py)) implements sophisticated pattern matching in `_extract_with_patterns()`. The method constructs a dynamic entity regex from the provided entity list (sorted by length descending to prevent partial matches), then applies predefined relation patterns such as *founded_by*, *located_in*, *works_for*, and *acquired_by*.

```python

# Dynamic regex construction from known entities

entity_texts = [re.escape(e.text) for e in sorted(entities, key=lambda e: len(e.text), reverse=True)]
ent_pat = f"(?:{'|'.join(entity_texts)})"

# Pattern application with entity resolution

for relation_type, patterns in self.relation_patterns.items():
    for match in re.finditer(pattern, text, re.IGNORECASE):
        subject = match_entity(match.group("subject").strip(), entities)
        object_ = match_entity(match.group("object").strip(), entities)

```

### Co-occurrence and Dependency Parsing

Beyond explicit patterns, the extractor generates **co-occurrence** relations when entities appear within 100 characters of each other, assigning lower confidence scores to these implicit connections. When the `dependency` method is selected, the system utilizes spaCy's syntactic parser to extract grammatical triples and convert them into `Relation` objects.

### Ensemble Voting and Fallbacks

The relation extractor mirrors the NER fallback chain: it attempts methods in sequence (typically `["llm", "dependency", "pattern"]`) and falls back to `_extract_with_patterns()` if no method yields results. Confidence filtering and type-whitelist re-scoring follow the same `calculate_weighted_confidence()` logic used in entity extraction.

## Shared Similarity and Caching Infrastructure

### Hybrid Similarity Engine

The `find_best_match_index()` function in [`utils/helpers.py`](https://github.com/semantica-agi/semantica/blob/main/utils/helpers.py) provides a hierarchical matching strategy:

1. Exact string matching
2. Synonym lookup (`_ENTITY_SYNONYMS`)
3. Substring scoring
4. **Embedding similarity** via cached `TextEmbedder` instances
5. spaCy vector similarity
6. Fuzzy ratio fallback

This ensures low-latency results for common cases while preserving semantic accuracy for ambiguous matches.

### Caching and Progress Tracking

Every extraction call generates a tracking ID via `get_progress_tracker()`, enabling real-time progress reporting for batch operations. The `ExtractionCache` class ([`cache.py`](https://github.com/semantica-agi/semantica/blob/main/cache.py)) stores results keyed by method name, text hash, and configuration parameters, preventing redundant API calls for LLM-based methods.

## Implementation Examples

### Basic spaCy Entity Extraction

```python
from semantica.semantic_extract import NERExtractor

ner = NERExtractor()  # method="ml" by default

text = "Apple Inc. was founded by Steve Jobs in 1976."
entities = ner.extract_entities(text)

# Returns: [Entity(text='Apple Inc.', label='ORG', ...), ...]

```

### LLM-Backed Extraction with Type Whitelist

```python
ner = NERExtractor(
    method="llm",
    provider="openai",
    llm_model="gpt-4o-mini",
    entity_types=["PERSON", "ORG", "DATE"],
)

entities = ner.extract_entities(
    "Elon Musk founded SpaceX in 2002.",
    min_confidence=0.5,
)

```

### Relation Extraction with Fallback Chain

```python
from semantica.semantic_extract import RelationExtractor

rel_extractor = RelationExtractor(
    method=["llm", "dependency", "pattern"],
    provider="groq",
    llm_model="gemma-2b-it",
    confidence_threshold=0.6,
)

relations = rel_extractor.extract_relations(
    text,
    entities,
    relation_types=["founded_by", "located_in"],
)

```

### Complete Pipeline with RDF Serialization

```python
from semantica.semantic_extract import TripletExtractor

triplet_extractor = TripletExtractor(method="llm", provider="openai")
triplets = triplet_extractor.extract_triplets(
    text,
    entities=entities,
    relations=relations,
    triplet_types=["foundedBy", "locatedIn"],
)

turtle = triplet_extractor.serialize_triplets(triplets, format="turtle")

```

## Summary

- **Modular orchestrators** (`NERExtractor` and `RelationExtractor`) coordinate multiple extraction backends through a unified interface
- **Method registry** pattern enables seamless switching between regex, spaCy, HuggingFace, and LLM approaches without changing calling code
- **Intelligent fallback chains** attempt methods in priority order, automatically filtering unavailable ML components
- **Weighted confidence scoring** combines method-native scores with semantic similarity to user-specified type whitelists
- **Process-level caching** for spaCy models, embeddings, and LLM results minimizes redundant computation and API costs
- **Dynamic pattern matching** for relations uses entity-aware regex construction and longest-first matching to ensure accurate entity resolution

## Frequently Asked Questions

### What file contains the main entity extraction logic in Semantica?

The primary orchestration logic resides in [`semantica/semantic_extract/ner_extractor.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/semantic_extract/ner_extractor.py), which dispatches to concrete implementations in [`semantica/semantic_extract/methods.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/semantic_extract/methods.py). The orchestrator handles method selection, confidence weighting, and fallback sequencing, while [`methods.py`](https://github.com/semantica-agi/semantica/blob/main/methods.py) contains the specific implementations for pattern matching, spaCy integration, and LLM prompting.

### How does Semantica handle extraction when the primary ML method fails?

The `NERExtractor` class automatically filters unusable methods via `_filter_unusable_methods()`, which removes spaCy-dependent strategies if model initialization fails. During execution, the system iterates through the configured method list and returns results from the first successful technique, or combines multiple results when `ensemble_voting` is enabled.

### Can Semantica extract relations without pre-identified entities?

Yes. While the `RelationExtractor` accepts pre-identified entities for pattern matching, it can also extract relations independently. The pattern-based method constructs dynamic entity regexes from provided entities, but co-occurrence and dependency parsing methods can identify relational structures directly from text, though with potentially lower precision.

### How does the confidence scoring system work for entity types?

When users specify an `entity_types` whitelist, `calculate_weighted_confidence()` blends the extraction method's native confidence score with the semantic similarity between the extracted label and allowed types. This similarity calculation uses `find_best_match_index()`, which applies a fast hybrid matching strategy including exact matches, embeddings, and fuzzy string comparison, defaulting to equal weighting (0.5/0.5) between method confidence and type similarity.