# How to Use the `evals` Module for KG Quality Evaluation and Regression Tracking in Semantica

> Learn KG quality evaluation and regression tracking with Semantica. Discover OntologyEvaluator for coverage scoring gap detection and manual tracking.

- Repository: [Semantica /semantica](https://github.com/semantica-agi/semantica)
- Tags: how-to-guide
- Published: 2026-08-16

---

**The `semantica.evals` module is a planned future-release component with no runnable code today; for KG quality evaluation, use the fully-functional `OntologyEvaluator` in `semantica.ontology`, which provides coverage scoring, gap detection, and manual regression tracking capabilities.**

The `semantica` library (maintained by semantica-agi) positions itself as an end-to-end knowledge-graph construction pipeline. While its dedicated evaluation framework is still under development, practitioners need reliable tools to validate KG quality and catch regressions. This guide explains the current state of `semantica.evals`, what you can use immediately, and how to prepare for the full-featured release.

## Current Status: The `evals` Placeholder

As of the latest source code, [`semantica/evals/__init__.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/evals/__init__.py) contains only:

```python
__all__ = []

```

No classes, functions, or APIs are implemented. The placeholder is documented at [`docs/reference/evals.md`](https://github.com/semantica-agi/semantica/blob/main/docs/reference/evals.md), which outlines the **three-planned evaluation concerns**:

| Planned Component | Purpose | Target Metrics |
|---|---|---|
| `KGEvaluator` | Knowledge-graph quality assessment | Completeness, consistency, schema compliance, coverage, orphan-node detection |
| `ExtractionEvaluator` | Extraction accuracy validation | NER precision/recall/F1, relation-extraction metrics against gold data |
| `PipelineBenchmark` + `RegressionTracker` | Performance monitoring and regression detection | Throughput (docs/sec), per-step latency, peak memory, error rate, metric drift across commits |

When released, these evaluators will feed metrics into a `RegressionTracker` with persistent storage (JSON/SQLite) and generate `EvalReport` objects for CI/CD integration.

## What to Use Today: `OntologyEvaluator`

Until `semantica.evals` ships, the **recommended workaround** is `semantica.ontology.OntologyEvaluator`. This concrete implementation in [`semantica/ontology/ontology_evaluator.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ontology/ontology_evaluator.py) evaluates the **schema layer of your KG**—the ontology itself—against competency questions.

### Core Components

- **`OntologyEvaluator`** – orchestrates evaluation, progress tracking, and report generation
- **`EvaluationResult`** – dataclass aggregating scores, gaps, and suggestions
- **`CompetencyQuestionsManager`** – validates ontology coverage against user-supplied questions ([`semantica/ontology/competency_questions.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ontology/competency_questions.py))

### Evaluation Flow

```

User ontology + competency questions
    ↓
OntologyEvaluator.__init__()
    ↓
evaluate_ontology()
    ↓
CompetencyQuestionsManager validates → Coverage & completeness computed
    ↓
Gaps identified → EvaluationResult created → generate_report() (optional)

```

The evaluator uses [`semantica/utils/progress_tracker.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/utils/progress_tracker.py) for execution tracking via `start_tracking()`, `update_tracking()`, and `stop_tracking()` hooks.

## Practical Code Examples

### Basic Ontology Evaluation

Evaluate an ontology and extract quality metrics:

```python
from semantica.ontology import OntologyEvaluator

# Sample ontology (normally loaded from JSON/YAML)

ontology = {
    "classes": [
        {"name": "Person", "uri": "ex:Person", "label": "Person"},
        {"name": "Organization", "uri": "ex:Org", "label": "Organization"},
    ],
    "properties": [
        {"name": "worksFor", "type": "object", "uri": "ex:worksFor"},
    ],
}

# Competency questions drive coverage scoring

questions = [
    "Who works for which organization?",
    "What are the employees of each organization?",
]

evaluator = OntologyEvaluator()
result = evaluator.evaluate_ontology(ontology, competency_questions=questions)

print("Coverage score:", result.coverage_score)
print("Completeness:", result.completeness_score)
print("Gaps:", result.gaps)
print("Suggestions:", result.suggestions)

# Generate structured report

report = evaluator.generate_report(ontology)
print(report)  # Dict with "evaluation" and "relation_completeness" sections

```

Source: [`semantica/ontology/ontology_evaluator.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ontology/ontology_evaluator.py)

### Manual Regression Tracking

Until `RegressionTracker` arrives, implement baseline comparison yourself:

```python
import json
from pathlib import Path
from semantica.ontology import OntologyEvaluator

# Load previous evaluation from CI artifacts

previous_path = Path("reports/ontology_eval_2024-08-15.json")
previous = json.loads(previous_path.read_text()) if previous_path.exists() else {}

evaluator = OntologyEvaluator()
current = evaluator.evaluate_ontology(ontology, competency_questions=questions)

def compare(prev, cur, tol=0.01):
    """Detect metric drift beyond tolerance."""
    for metric in ["coverage_score", "completeness_score"]:
        drift = abs(prev.get(metric, 0) - getattr(cur, metric))
        if drift > tol:
            print(f"⚠️ Regression detected on {metric}: drift = {drift:.3f}")

compare(previous, current)

```

Store `current` (via `dataclasses.asdict()` or JSON serialization) for future comparisons.

### Future API Preview

Once `semantica.evals` releases, the above manual logic simplifies to:

```python
from semantica.evals import KGEvaluator, RegressionTracker, EvalReport

# KG evaluation

kg_eval = KGEvaluator(schema=my_schema)
scores = kg_eval.evaluate(my_knowledge_graph)

# Automated regression tracking

tracker = RegressionTracker()
tracker.record("kg_quality", scores)
tracker.compare_against("kg_quality", baseline="main")  # Auto-detects drift

report = EvalReport(scores).as_markdown()

```

## Key Source Files and Their Roles

| File | Purpose |
|---|---|
| [`docs/reference/evals.md`](https://github.com/semantica-agi/semantica/blob/main/docs/reference/evals.md) | Official roadmap documentation for the planned evaluation framework |
| [`semantica/ontology/ontology_evaluator.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ontology/ontology_evaluator.py) | Current implementation of `OntologyEvaluator` and `EvaluationResult` |
| [`semantica/ontology/competency_questions.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ontology/competency_questions.py) | `CompetencyQuestionsManager` for question-driven validation |
| [`semantica/utils/progress_tracker.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/utils/progress_tracker.py) | Progress tracking infrastructure used across evaluation workflows |

## Summary

- **`semantica.evals` is a placeholder**—no runnable code exists today
- **`OntologyEvaluator` is the current solution** for KG quality evaluation via ontology validation
- **`EvaluationResult` provides** coverage scores, completeness metrics, identified gaps, and actionable suggestions
- **Manual regression tracking is possible** by serializing results and comparing against baselines
- **Future releases will add** `KGEvaluator`, `ExtractionEvaluator`, `PipelineBenchmark`, and `RegressionTracker` for unified, automated evaluation

## Frequently Asked Questions

### Why is `semantica.evals` empty?

The module is intentionally reserved for a unified evaluation framework. The maintainers prioritized the ontology layer first, as schema quality underpins all downstream KG construction. The placeholder documents the API design without breaking future compatibility.

### Can `OntologyEvaluator` evaluate actual graph instances?

No—it evaluates the **schema/ontology**, not populated knowledge graphs. For graph-instance evaluation (orphan nodes, consistency checks), wait for `KGEvaluator` or implement custom logic against your graph store.

### How stable is `OntologyEvaluator` for production use?

The implementation in [`semantica/ontology/ontology_evaluator.py`](https://github.com/semantica-agi/semantica/blob/main/semantica/ontology/ontology_evaluator.py) is fully functional with established method signatures: `evaluate_ontology()`, `generate_report()`, and the `EvaluationResult` dataclass. The progress-tracking integration via [`progress_tracker.py`](https://github.com/semantica-agi/semantica/blob/main/progress_tracker.py) indicates production-ready instrumentation.