How to Use the `evals` Module for KG Quality Evaluation and Regression Tracking in Semantica

The semantica.evals module is a planned future-release component with no runnable code today; for KG quality evaluation, use the fully-functional OntologyEvaluator in semantica.ontology, which provides coverage scoring, gap detection, and manual regression tracking capabilities.

The semantica library (maintained by semantica-agi) positions itself as an end-to-end knowledge-graph construction pipeline. While its dedicated evaluation framework is still under development, practitioners need reliable tools to validate KG quality and catch regressions. This guide explains the current state of semantica.evals, what you can use immediately, and how to prepare for the full-featured release.

Current Status: The evals Placeholder

As of the latest source code, semantica/evals/__init__.py contains only:

__all__ = []

No classes, functions, or APIs are implemented. The placeholder is documented at docs/reference/evals.md, which outlines the three-planned evaluation concerns:

Planned Component	Purpose	Target Metrics
KGEvaluator	Knowledge-graph quality assessment	Completeness, consistency, schema compliance, coverage, orphan-node detection
ExtractionEvaluator	Extraction accuracy validation	NER precision/recall/F1, relation-extraction metrics against gold data
PipelineBenchmark + RegressionTracker	Performance monitoring and regression detection	Throughput (docs/sec), per-step latency, peak memory, error rate, metric drift across commits

When released, these evaluators will feed metrics into a RegressionTracker with persistent storage (JSON/SQLite) and generate EvalReport objects for CI/CD integration.

What to Use Today: OntologyEvaluator

Until semantica.evals ships, the recommended workaround is semantica.ontology.OntologyEvaluator. This concrete implementation in semantica/ontology/ontology_evaluator.py evaluates the schema layer of your KG—the ontology itself—against competency questions.

Core Components

• OntologyEvaluator – orchestrates evaluation, progress tracking, and report generation
• EvaluationResult – dataclass aggregating scores, gaps, and suggestions
• CompetencyQuestionsManager – validates ontology coverage against user-supplied questions (semantica/ontology/competency_questions.py)

Evaluation Flow

User ontology + competency questions
    ↓
OntologyEvaluator.__init__()
    ↓
evaluate_ontology()
    ↓
CompetencyQuestionsManager validates → Coverage & completeness computed
    ↓
Gaps identified → EvaluationResult created → generate_report() (optional)

The evaluator uses semantica/utils/progress_tracker.py for execution tracking via start_tracking(), update_tracking(), and stop_tracking() hooks.

Practical Code Examples

Basic Ontology Evaluation

Evaluate an ontology and extract quality metrics:

from semantica.ontology import OntologyEvaluator

# Sample ontology (normally loaded from JSON/YAML)

ontology = {
    "classes": [
        {"name": "Person", "uri": "ex:Person", "label": "Person"},
        {"name": "Organization", "uri": "ex:Org", "label": "Organization"},
    ],
    "properties": [
        {"name": "worksFor", "type": "object", "uri": "ex:worksFor"},
    ],
}

# Competency questions drive coverage scoring

questions = [
    "Who works for which organization?",
    "What are the employees of each organization?",
]

evaluator = OntologyEvaluator()
result = evaluator.evaluate_ontology(ontology, competency_questions=questions)

print("Coverage score:", result.coverage_score)
print("Completeness:", result.completeness_score)
print("Gaps:", result.gaps)
print("Suggestions:", result.suggestions)

# Generate structured report

report = evaluator.generate_report(ontology)
print(report)  # Dict with "evaluation" and "relation_completeness" sections

Source: semantica/ontology/ontology_evaluator.py

Manual Regression Tracking

Until RegressionTracker arrives, implement baseline comparison yourself:

import json
from pathlib import Path
from semantica.ontology import OntologyEvaluator

# Load previous evaluation from CI artifacts

previous_path = Path("reports/ontology_eval_2024-08-15.json")
previous = json.loads(previous_path.read_text()) if previous_path.exists() else {}

evaluator = OntologyEvaluator()
current = evaluator.evaluate_ontology(ontology, competency_questions=questions)

def compare(prev, cur, tol=0.01):
    """Detect metric drift beyond tolerance."""
    for metric in ["coverage_score", "completeness_score"]:
        drift = abs(prev.get(metric, 0) - getattr(cur, metric))
        if drift > tol:
            print(f"⚠️ Regression detected on {metric}: drift = {drift:.3f}")

compare(previous, current)

Store current (via dataclasses.asdict() or JSON serialization) for future comparisons.

Future API Preview

Once semantica.evals releases, the above manual logic simplifies to:

from semantica.evals import KGEvaluator, RegressionTracker, EvalReport

# KG evaluation

kg_eval = KGEvaluator(schema=my_schema)
scores = kg_eval.evaluate(my_knowledge_graph)

# Automated regression tracking

tracker = RegressionTracker()
tracker.record("kg_quality", scores)
tracker.compare_against("kg_quality", baseline="main")  # Auto-detects drift

report = EvalReport(scores).as_markdown()

Key Source Files and Their Roles

File	Purpose
docs/reference/evals.md	Official roadmap documentation for the planned evaluation framework
semantica/ontology/ontology_evaluator.py	Current implementation of OntologyEvaluator and EvaluationResult
semantica/ontology/competency_questions.py	CompetencyQuestionsManager for question-driven validation
semantica/utils/progress_tracker.py	Progress tracking infrastructure used across evaluation workflows

Summary

• semantica.evals is a placeholder—no runnable code exists today
• OntologyEvaluator is the current solution for KG quality evaluation via ontology validation
• EvaluationResult provides coverage scores, completeness metrics, identified gaps, and actionable suggestions
• Manual regression tracking is possible by serializing results and comparing against baselines
• Future releases will add KGEvaluator, ExtractionEvaluator, PipelineBenchmark, and RegressionTracker for unified, automated evaluation

Frequently Asked Questions

Why is semantica.evals empty?

The module is intentionally reserved for a unified evaluation framework. The maintainers prioritized the ontology layer first, as schema quality underpins all downstream KG construction. The placeholder documents the API design without breaking future compatibility.

Can OntologyEvaluator evaluate actual graph instances?

No—it evaluates the schema/ontology, not populated knowledge graphs. For graph-instance evaluation (orphan nodes, consistency checks), wait for KGEvaluator or implement custom logic against your graph store.

How stable is OntologyEvaluator for production use?

The implementation in semantica/ontology/ontology_evaluator.py is fully functional with established method signatures: evaluate_ontology(), generate_report(), and the EvaluationResult dataclass. The progress-tracking integration via progress_tracker.py indicates production-ready instrumentation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}