# How to Write and Run Evals Using OmniRoute's Generic Eval Framework: A Complete Guide > Learn to write and run evals with OmniRoute's generic eval framework. Define JSON test suites, store in SQLite, and automate LLM output validation with exact match, substring, or regex. - Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute) - Tags: how-to-guide - Published: 2026-07-19 --- **OmniRoute's generic eval framework lets you define JSON-based test suites, persist them in SQLite, and execute automated evaluations against LLM outputs using strategies like exact match, substring contains, or regex validation.** The evaluation engine in [diegosouzapw/OmniRoute](https://github.com/diegosouzapw/OmniRoute) provides a standardized way to measure LLM response quality against golden-set benchmarks. Whether you are testing routing logic or validating model outputs, the framework separates suite definition, execution, and persistence into discrete layers. This guide covers how to write and run evals using OmniRoute's generic eval framework from both the CLI and programmatic interfaces. ## Understanding the Evaluation Architecture The framework consists of three distinct layers that handle different stages of the evaluation lifecycle. **Suite Definition Layer** stores pure data describing test cases. Built-in suites ship with the repository in [`src/lib/evals/evalRunner/builtinSuites.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/evals/evalRunner/builtinSuites.ts), while custom suites persist in SQLite via [`src/lib/db/evals.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/db/evals.ts). Each suite contains input messages, expected output strategies, and optional metadata tags. **Runner Core Layer** handles execution logic in [`src/lib/evals/evalRunner.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/evals/evalRunner.ts). This module registers suites, evaluates individual cases using configurable strategies, aggregates results into suite runs, and generates human-readable scorecards. **Persistence and API Layer** manages data storage and access. The SQLite database (schemas defined in migrations [`030_create_eval_runs.sql`](https://github.com/diegosouzapw/OmniRoute/blob/main/030_create_eval_runs.sql) and [`031_create_eval_suites.sql`](https://github.com/diegosouzapw/OmniRoute/blob/main/031_create_eval_suites.sql)) stores runs and suites, while REST endpoints in [`src/app/api/evals/route.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/app/api/evals/route.ts) and the CLI wrapper in `bin/cli/commands/eval.mjs` provide interfaces for triggering and inspecting evaluations. ## Writing a Custom Evaluation Suite A suite follows the `EvalSuite` interface requiring an `id`, `name`, optional `description`, and an array of `cases`. Each `EvalCase` must specify: - `id` – unique identifier for the case - `name` – human-readable description - `model` – target model identifier (optional for routing) - `input` – object containing a `messages` array (OpenAI-style chat format) - `expected` – object with a `strategy` (`"exact"`, `"contains"`, `"regex"`, or `"custom"`) and `value` - `tags` – optional string array for filtering (e.g., `"safety"`) Create a JSON file following this structure: ```json { "id": "my-suite", "name": "My Quick Test Suite", "description": "A tiny demo suite for rapid checks.", "cases": [ { "id": "case-01", "name": "Simple greeting", "model": "gpt-4o", "input": { "messages": [{ "role": "user", "content": "Hello" }] }, "expected": { "strategy": "contains", "value": "hello" }, "tags": ["quick"] }, { "id": "case-02", "name": "JSON-only output", "model": "gpt-4o", "input": { "messages": [ { "role": "system", "content": "Respond ONLY with valid JSON." }, { "role": "user", "content": "Give me {\"status\":\"ok\"}" } ] }, "expected": { "strategy": "regex", "value": "^\\s*\\{" } } ] } ``` The `strategy` field determines how `evaluateCase()` in [`src/lib/evals/evalRunner.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/evals/evalRunner.ts) validates responses. Use `"contains"` for case-insensitive substring checks, `"exact"` for strict equality, or `"regex"` for pattern matching (limited to 512 characters). The `"custom"` strategy accepts a function but only works when constructing suites programmatically at runtime, as JSON cannot serialize functions. ## Persisting and Registering Suites Before running evaluations, you must register the suite in OmniRoute's SQLite database. ### Via the CLI Use the `suites create` command to import your JSON file: ```bash omniroute eval suites create --file my-suite.json ``` The CLI invokes `runEvalSuitesCreate`, which POSTs the payload to `/api/evals/suites`. The backend calls `saveCustomEvalSuite()` in [`src/lib/db/evals.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/db/evals.ts), creating or updating records in the `eval_suites` and `eval_cases` tables. ### Via Direct HTTP Send the JSON directly to the API endpoint: ```bash curl -X POST http://localhost:20128/api/evals/suites \ -H "Content-Type: application/json" \ -d @my-suite.json ``` Both methods return a confirmation showing the Suite ID, name, sample count, and timestamp. ## Running Evaluations Once persisted, you can execute suites against live models or mock outputs. ### CLI Execution with Live Watching Run a suite and monitor progress in real-time: ```bash omniroute eval run my-suite --watch ``` This command constructs a request body with `suiteId`, `model` (defaults to `"auto"`), `concurrency` (default 4), and optional `tag` or `combo` filters. It POSTs to `/api/evals`, which: 1. Retrieves the suite using `getSuite()` (checking built-in or custom sources) 2. Streams inputs to the selected provider(s) 3. Calls `evaluateCase()` for each output against the expected strategy 4. Persists results via `saveEvalRun()` in [`src/lib/db/evals.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/db/evals.ts) 5. Returns a run record with status transitioning from `"queued"` to `"completed"` The `--watch` flag polls `/api/evals/:runId` until completion and renders a scorecard using `renderScorecard` in the CLI. ### Programmatic Execution Import the runner directly to integrate evaluations into Node.js scripts or CI pipelines: ```typescript import { runSuite, getSuite, createScorecard } from "@/lib/evals/evalRunner"; import { saveEvalRun } from "@/lib/db/evals"; // Load the suite definition const suiteId = "my-suite"; const suite = await getSuite(suiteId); // Prepare outputs (in production, fetch from your LLM provider) const outputs: Record = { "case-01": "Hello there!", "case-02": "{\"status\":\"ok\"}" }; // Execute evaluation const result = runSuite(suiteId, outputs); // Persist for audit trails await saveEvalRun({ suiteId, suiteName: result.suiteName, target: { type: "suite-default", id: null, label: "CLI demo" }, summary: result.summary, results: result.results, outputs }); // Generate report console.log(`✅ ${result.summary.passed}/${result.summary.total} passed`); console.table(result.results.map(r => ({ case: r.caseName, passed: r.passed, snippet: r.details?.actualSnippet }))); ``` This approach bypasses the HTTP layer while reusing the same core logic: `evaluateCase()` for individual validation, `runSuite()` for aggregation, and `createScorecard()` for reporting. ## Analyzing Results and Scorecards After execution, use the CLI to inspect historical and current runs: - `omniroute eval list` – Display recent runs with filtering by suite, status, or time range - `omniroute eval get ` – Retrieve the full JSON payload including per-case details - `omniroute eval results ` – View tabular case scores with pass/fail indicators - `omniroute eval scorecard ` – Pretty-print overall pass rates and breakdowns - `omniroute eval cancel ` – Abort active evaluations All commands interface with the backend routes defined in [`src/app/api/evals/route.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/app/api/evals/route.ts) and [`src/app/api/evals/suites/route.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/app/api/evals/suites/route.ts), querying the `eval_runs` table for historical data. ## Summary - **Define suites** as JSON objects following the `EvalSuite` interface, specifying cases with `exact`, `contains`, or `regex` validation strategies. - **Persist suites** using `omniroute eval suites create` or POST to `/api/evals/suites`, which writes to `eval_suites` and `eval_cases` tables via `saveCustomEvalSuite()`. - **Execute evaluations** via CLI (`omniroute eval run`) or programmatically using `runSuite()` from [`src/lib/evals/evalRunner.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/evals/evalRunner.ts). - **Store results** automatically in SQLite through `saveEvalRun()`, enabling historical tracking and audit trails. - **Inspect outcomes** using CLI list, get, and scorecard commands that query the REST API endpoints. ## Frequently Asked Questions ### How do I filter evaluation cases by tags? Use the `--tag` flag when running a suite: `omniroute eval run my-suite --tag quick`. The runner evaluates only cases containing the specified tag in their `tags` array, allowing you to run subsets of large suites without modifying the suite definition. ### Can I use custom validation logic beyond regex? Yes, but only programmatically. While JSON-based suites in [`builtinSuites.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/builtinSuites.ts) or custom SQLite entries support `exact`, `contains`, and `regex` strategies, the `custom` strategy accepts a JavaScript function when building suites at runtime. Construct the suite object in code rather than JSON, passing a function to the `expected` field, then pass it to `runSuite()`. ### What database tables store evaluation data? OmniRoute uses three core tables defined in migrations [`030_create_eval_runs.sql`](https://github.com/diegosouzapw/OmniRoute/blob/main/030_create_eval_runs.sql) and [`031_create_eval_suites.sql`](https://github.com/diegosouzapw/OmniRoute/blob/main/031_create_eval_suites.sql): `eval_suites` stores suite metadata, `eval_cases` stores individual test cases with their expected values, and `eval_runs` stores execution results including pass/fail status and actual outputs. Access these via [`src/lib/db/evals.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/db/evals.ts) functions like `saveEvalRun()` and `getSuite()`. ### How does the CLI `--watch` mode work? The `--watch` flag triggers a polling loop that repeatedly calls `GET /api/evals/:runId` after initiating a run. Once the run status changes from `"queued"` to `"completed"`, the CLI fetches the final results and invokes `renderScorecard` to display a formatted summary with pass rates and per-case breakdowns in the terminal.