How the OmniRoute Evals Framework Tests Combo and Model Performance

The OmniRoute evals framework is a lightweight, plug-in-style evaluation system that benchmarks individual models and combo-routing strategies against predefined golden sets using exact matching, substring containment, regex validation, or custom evaluation functions.

The OmniRoute repository implements a modular evaluation system designed to validate large language model outputs and routing combinations. This framework enables developers to programmatically verify whether specific models or combo strategies meet functional expectations against trusted test suites.

Suite Registration and Built-in Test Cases

The framework organizes evaluations around suites, which are collections of test cases stored as pure data. In src/lib/evals/evalRunner/builtinSuites.ts, OmniRoute defines several built-in suites including the golden set, coding proficiency, reasoning tasks, multilingual prompts, and safety checks.

Each suite contains an array of EvalCase objects that specify the target model, request payload, and expected evaluation strategy. These cases serve as the ground truth for assessing whether a model or combo produces correct outputs.

Runtime Registration and Suite Management

When the module src/lib/evals/evalRunner.ts loads, it registers every built-in suite in a module-wide Map data structure. The framework exposes registerSuite for adding custom test suites at runtime, allowing teams to integrate proprietary evaluation criteria without modifying core library files.

Developers can inspect available suites using the listSuites function, which returns metadata including suite IDs, names, and case counts for all registered built-in and custom collections.

Evaluation Strategies for Model Outputs

The evaluateCase function in src/lib/evals/evalRunner.ts implements four distinct validation strategies:

• exact – Performs direct string equality comparison between the expected and actual outputs.
• contains – Executes a case-insensitive substring search to verify expected content appears within the response.
• regex – Constructs a safe regular expression with pattern length limits and stripped g/y flags to prevent catastrophic backtracking.
• custom – Accepts a user-supplied function that receives the raw LLM output and case definition, returning a boolean pass/fail determination.

Each evaluation returns an EvalResult object containing the pass/fail status, execution duration, and optional diagnostic metadata.

Executing Test Suites Against Model Combos

The runSuite function executes evaluations by accepting a suiteId and a mapping of caseId → actualOutput strings. This design allows the router to dispatch identical requests to multiple models in a combo, collect the generated text, and feed the aggregated outputs into the evaluation engine.

When invoked, runSuite retrieves the specified suite and iterates over its cases, calling evaluateCase for each entry. The function produces a summary statistics object containing total cases processed, passed count, failed count, and calculated pass rate percentages.

Scorecard Aggregation for Performance Metrics

After executing multiple suites—potentially testing different combo configurations—the createScorecard function aggregates individual suite results into a comprehensive performance overview. This aggregation folds suite-level summaries into per-suite pass rates and global statistics, generating data suitable for CI dashboards and automated quality gates.

The scorecard structure includes total case counts across all suites, overall pass rates, and breakdowns by evaluation category, enabling direct comparison between routing strategies.

Integration with the Router Runtime

The public API surface in src/lib/evals/runtime.ts receives evaluation requests containing a suite ID and raw model outputs. This module forwards payloads to runSuite and returns enriched results to callers.

In practice, the router's combo engine dispatches requests to each target model, aggregates responses, and invokes the eval framework through the Next.js endpoint defined in src/app/api/evals/route.ts. This workflow enables a single API call to assess how well a combo performed against golden expectations.

Practical Implementation Examples

List available test suites and inspect their configuration:

import { listSuites } from "@/lib/evals/evalRunner";

console.log(listSuites());
// → [{ id: "golden-set", name: "OmniRoute Golden Set", caseCount: 10, … }, …]

Execute a suite against collected model outputs:

import { runSuite } from "@/lib/evals/evalRunner";

const outputs = {
  "gs-01": "Hello! How can I help?",
  "gs-02": "The answer is 4.",
  // … additional caseId → LLM response mappings
};

const result = runSuite("golden-set", outputs);
console.log(result.summary);
// { total: 10, passed: 9, failed: 1, passRate: 90 }

Generate a comparative scorecard across multiple evaluation runs:

import { createScorecard, runSuite } from "@/lib/evals/evalRunner";

const suiteRuns = [
  runSuite("golden-set", outputsStrategyA),
  runSuite("coding-proficiency", outputsStrategyB),
];

const scorecard = createScorecard(suiteRuns);
console.log(scorecard.overallPassRate); // 93

Summary

• Suite-based organization – Test cases live in builtinSuites.ts as EvalCase objects grouped by domain (golden set, coding, reasoning).
• Flexible validation – Four evaluation strategies (exact, contains, regex, custom) accommodate diverse output requirements.
• Combo testing – The runSuite function maps case IDs to model outputs, enabling simultaneous validation of multiple routing strategies.
• Performance metrics – createScorecard aggregates multi-suite results for CI dashboards and regression detection.
• Runtime integration – The runtime.ts module and Next.js API route expose evaluation capabilities to the router's combo engine.

Frequently Asked Questions

How does the OmniRoute evals framework handle testing for multiple model combinations?

The framework accepts a mapping of caseId → actualOutput in the runSuite function, allowing you to collect responses from several models or combo strategies into a single object. By running the same suite against different output collections and comparing scorecards, you can determine which combination yields higher pass rates against the golden set.

Can I implement custom evaluation logic beyond the built-in strategies?

Yes. The framework supports a custom evaluation type where you provide a function that receives the raw LLM output and the case definition. Register your custom suite using registerSuite in src/lib/evals/evalRunner.ts, and the runtime will invoke your function during case evaluation, returning results alongside standard strategy outputs.

How does the regex strategy prevent performance issues during evaluation?

According to the source code in src/lib/evals/evalRunner.ts, the regex implementation limits pattern length and automatically strips global (g) and sticky (y) flags from user-supplied patterns. These safety measures prevent catastrophic backtracking and ensure deterministic matching behavior during large-scale evaluation runs.

Where can I find example implementations of the eval framework in testing?

Reference implementations appear in tests/unit/batch-b-final.test.ts, which exercises runSuite, createScorecard, and error handling scenarios. Additionally, the Next.js endpoint in src/app/api/evals/route.ts demonstrates how to integrate the evaluation runtime with external API consumers.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}