How to Write and Run Evals Using OmniRoute's Generic Eval Framework: A Complete Guide

OmniRoute's generic eval framework lets you define JSON-based test suites, persist them in SQLite, and execute automated evaluations against LLM outputs using strategies like exact match, substring contains, or regex validation.

The evaluation engine in diegosouzapw/OmniRoute provides a standardized way to measure LLM response quality against golden-set benchmarks. Whether you are testing routing logic or validating model outputs, the framework separates suite definition, execution, and persistence into discrete layers. This guide covers how to write and run evals using OmniRoute's generic eval framework from both the CLI and programmatic interfaces.

Understanding the Evaluation Architecture

The framework consists of three distinct layers that handle different stages of the evaluation lifecycle.

Suite Definition Layer stores pure data describing test cases. Built-in suites ship with the repository in src/lib/evals/evalRunner/builtinSuites.ts, while custom suites persist in SQLite via src/lib/db/evals.ts. Each suite contains input messages, expected output strategies, and optional metadata tags.

Runner Core Layer handles execution logic in src/lib/evals/evalRunner.ts. This module registers suites, evaluates individual cases using configurable strategies, aggregates results into suite runs, and generates human-readable scorecards.

Persistence and API Layer manages data storage and access. The SQLite database (schemas defined in migrations 030_create_eval_runs.sql and 031_create_eval_suites.sql) stores runs and suites, while REST endpoints in src/app/api/evals/route.ts and the CLI wrapper in bin/cli/commands/eval.mjs provide interfaces for triggering and inspecting evaluations.

Writing a Custom Evaluation Suite

A suite follows the EvalSuite interface requiring an id, name, optional description, and an array of cases. Each EvalCase must specify:

• id – unique identifier for the case
• name – human-readable description
• model – target model identifier (optional for routing)
• input – object containing a messages array (OpenAI-style chat format)
• expected – object with a strategy ("exact", "contains", "regex", or "custom") and value
• tags – optional string array for filtering (e.g., "safety")

Create a JSON file following this structure:

{
  "id": "my-suite",
  "name": "My Quick Test Suite",
  "description": "A tiny demo suite for rapid checks.",
  "cases": [
    {
      "id": "case-01",
      "name": "Simple greeting",
      "model": "gpt-4o",
      "input": { "messages": [{ "role": "user", "content": "Hello" }] },
      "expected": { "strategy": "contains", "value": "hello" },
      "tags": ["quick"]
    },
    {
      "id": "case-02",
      "name": "JSON-only output",
      "model": "gpt-4o",
      "input": {
        "messages": [
          { "role": "system", "content": "Respond ONLY with valid JSON." },
          { "role": "user", "content": "Give me {\"status\":\"ok\"}" }
        ]
      },
      "expected": { "strategy": "regex", "value": "^\\s*\\{" }
    }
  ]
}

The strategy field determines how evaluateCase() in src/lib/evals/evalRunner.ts validates responses. Use "contains" for case-insensitive substring checks, "exact" for strict equality, or "regex" for pattern matching (limited to 512 characters). The "custom" strategy accepts a function but only works when constructing suites programmatically at runtime, as JSON cannot serialize functions.

Persisting and Registering Suites

Before running evaluations, you must register the suite in OmniRoute's SQLite database.

Via the CLI

Use the suites create command to import your JSON file:

omniroute eval suites create --file my-suite.json

The CLI invokes runEvalSuitesCreate, which POSTs the payload to /api/evals/suites. The backend calls saveCustomEvalSuite() in src/lib/db/evals.ts, creating or updating records in the eval_suites and eval_cases tables.

Via Direct HTTP

Send the JSON directly to the API endpoint:

curl -X POST http://localhost:20128/api/evals/suites \
  -H "Content-Type: application/json" \
  -d @my-suite.json

Both methods return a confirmation showing the Suite ID, name, sample count, and timestamp.

Running Evaluations

Once persisted, you can execute suites against live models or mock outputs.

CLI Execution with Live Watching

Run a suite and monitor progress in real-time:

omniroute eval run my-suite --watch

This command constructs a request body with suiteId, model (defaults to "auto"), concurrency (default 4), and optional tag or combo filters. It POSTs to /api/evals, which:

1. Retrieves the suite using getSuite() (checking built-in or custom sources)
2. Streams inputs to the selected provider(s)
3. Calls evaluateCase() for each output against the expected strategy
4. Persists results via saveEvalRun() in src/lib/db/evals.ts
5. Returns a run record with status transitioning from "queued" to "completed"

The --watch flag polls /api/evals/:runId until completion and renders a scorecard using renderScorecard in the CLI.

Programmatic Execution

Import the runner directly to integrate evaluations into Node.js scripts or CI pipelines:

import { runSuite, getSuite, createScorecard } from "@/lib/evals/evalRunner";
import { saveEvalRun } from "@/lib/db/evals";

// Load the suite definition
const suiteId = "my-suite";
const suite = await getSuite(suiteId);

// Prepare outputs (in production, fetch from your LLM provider)
const outputs: Record<string, string> = {
  "case-01": "Hello there!",
  "case-02": "{\"status\":\"ok\"}"
};

// Execute evaluation
const result = runSuite(suiteId, outputs);

// Persist for audit trails
await saveEvalRun({
  suiteId,
  suiteName: result.suiteName,
  target: { type: "suite-default", id: null, label: "CLI demo" },
  summary: result.summary,
  results: result.results,
  outputs
});

// Generate report
console.log(`✅ ${result.summary.passed}/${result.summary.total} passed`);
console.table(result.results.map(r => ({
  case: r.caseName,
  passed: r.passed,
  snippet: r.details?.actualSnippet
})));

This approach bypasses the HTTP layer while reusing the same core logic: evaluateCase() for individual validation, runSuite() for aggregation, and createScorecard() for reporting.

Analyzing Results and Scorecards

After execution, use the CLI to inspect historical and current runs:

• omniroute eval list – Display recent runs with filtering by suite, status, or time range
• omniroute eval get <runId> – Retrieve the full JSON payload including per-case details
• omniroute eval results <runId> – View tabular case scores with pass/fail indicators
• omniroute eval scorecard <runId> – Pretty-print overall pass rates and breakdowns
• omniroute eval cancel <runId> – Abort active evaluations

All commands interface with the backend routes defined in src/app/api/evals/route.ts and src/app/api/evals/suites/route.ts, querying the eval_runs table for historical data.

Summary

• Define suites as JSON objects following the EvalSuite interface, specifying cases with exact, contains, or regex validation strategies.
• Persist suites using omniroute eval suites create or POST to /api/evals/suites, which writes to eval_suites and eval_cases tables via saveCustomEvalSuite().
• Execute evaluations via CLI (omniroute eval run) or programmatically using runSuite() from src/lib/evals/evalRunner.ts.
• Store results automatically in SQLite through saveEvalRun(), enabling historical tracking and audit trails.
• Inspect outcomes using CLI list, get, and scorecard commands that query the REST API endpoints.

Frequently Asked Questions

How do I filter evaluation cases by tags?

Use the --tag flag when running a suite: omniroute eval run my-suite --tag quick. The runner evaluates only cases containing the specified tag in their tags array, allowing you to run subsets of large suites without modifying the suite definition.

Can I use custom validation logic beyond regex?

Yes, but only programmatically. While JSON-based suites in builtinSuites.ts or custom SQLite entries support exact, contains, and regex strategies, the custom strategy accepts a JavaScript function when building suites at runtime. Construct the suite object in code rather than JSON, passing a function to the expected field, then pass it to runSuite().

What database tables store evaluation data?

OmniRoute uses three core tables defined in migrations 030_create_eval_runs.sql and 031_create_eval_suites.sql: eval_suites stores suite metadata, eval_cases stores individual test cases with their expected values, and eval_runs stores execution results including pass/fail status and actual outputs. Access these via src/lib/db/evals.ts functions like saveEvalRun() and getSuite().

How does the CLI --watch mode work?

The --watch flag triggers a polling loop that repeatedly calls GET /api/evals/:runId after initiating a run. Once the run status changes from "queued" to "completed", the CLI fetches the final results and invokes renderScorecard to display a formatted summary with pass rates and per-case breakdowns in the terminal.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}