# How the OmniRoute Evals Framework Tests Combo and Model Performance

> Discover how the OmniRoute evals framework tests combo and model performance with exact matching, regex validation, and custom functions. Benchmark your strategies efficiently.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: how-to-guide
- Published: 2026-08-04

---

**The OmniRoute evals framework is a lightweight, plug-in-style evaluation system that benchmarks individual models and combo-routing strategies against predefined golden sets using exact matching, substring containment, regex validation, or custom evaluation functions.**

The OmniRoute repository implements a modular evaluation system designed to validate large language model outputs and routing combinations. This framework enables developers to programmatically verify whether specific models or combo strategies meet functional expectations against trusted test suites.

## Suite Registration and Built-in Test Cases

The framework organizes evaluations around **suites**, which are collections of test cases stored as pure data. In [`src/lib/evals/evalRunner/builtinSuites.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/evals/evalRunner/builtinSuites.ts), OmniRoute defines several built-in suites including the golden set, coding proficiency, reasoning tasks, multilingual prompts, and safety checks.

Each suite contains an array of **EvalCase** objects that specify the target model, request payload, and expected evaluation strategy. These cases serve as the ground truth for assessing whether a model or combo produces correct outputs.

## Runtime Registration and Suite Management

When the module [`src/lib/evals/evalRunner.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/evals/evalRunner.ts) loads, it registers every built-in suite in a module-wide `Map` data structure. The framework exposes `registerSuite` for adding custom test suites at runtime, allowing teams to integrate proprietary evaluation criteria without modifying core library files.

Developers can inspect available suites using the `listSuites` function, which returns metadata including suite IDs, names, and case counts for all registered built-in and custom collections.

## Evaluation Strategies for Model Outputs

The `evaluateCase` function in [`src/lib/evals/evalRunner.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/evals/evalRunner.ts) implements four distinct validation strategies:

- **exact** – Performs direct string equality comparison between the expected and actual outputs.
- **contains** – Executes a case-insensitive substring search to verify expected content appears within the response.
- **regex** – Constructs a safe regular expression with pattern length limits and stripped `g`/`y` flags to prevent catastrophic backtracking.
- **custom** – Accepts a user-supplied function that receives the raw LLM output and case definition, returning a boolean pass/fail determination.

Each evaluation returns an **EvalResult** object containing the pass/fail status, execution duration, and optional diagnostic metadata.

## Executing Test Suites Against Model Combos

The `runSuite` function executes evaluations by accepting a `suiteId` and a mapping of `caseId → actualOutput` strings. This design allows the router to dispatch identical requests to multiple models in a combo, collect the generated text, and feed the aggregated outputs into the evaluation engine.

When invoked, `runSuite` retrieves the specified suite and iterates over its cases, calling `evaluateCase` for each entry. The function produces a summary statistics object containing total cases processed, passed count, failed count, and calculated pass rate percentages.

## Scorecard Aggregation for Performance Metrics

After executing multiple suites—potentially testing different combo configurations—the `createScorecard` function aggregates individual suite results into a comprehensive performance overview. This aggregation folds suite-level summaries into per-suite pass rates and global statistics, generating data suitable for CI dashboards and automated quality gates.

The scorecard structure includes total case counts across all suites, overall pass rates, and breakdowns by evaluation category, enabling direct comparison between routing strategies.

## Integration with the Router Runtime

The public API surface in [`src/lib/evals/runtime.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/evals/runtime.ts) receives evaluation requests containing a suite ID and raw model outputs. This module forwards payloads to `runSuite` and returns enriched results to callers.

In practice, the router's combo engine dispatches requests to each target model, aggregates responses, and invokes the eval framework through the Next.js endpoint defined in [`src/app/api/evals/route.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/app/api/evals/route.ts). This workflow enables a single API call to assess how well a combo performed against golden expectations.

## Practical Implementation Examples

List available test suites and inspect their configuration:

```typescript
import { listSuites } from "@/lib/evals/evalRunner";

console.log(listSuites());
// → [{ id: "golden-set", name: "OmniRoute Golden Set", caseCount: 10, … }, …]

```

Execute a suite against collected model outputs:

```typescript
import { runSuite } from "@/lib/evals/evalRunner";

const outputs = {
  "gs-01": "Hello! How can I help?",
  "gs-02": "The answer is 4.",
  // … additional caseId → LLM response mappings
};

const result = runSuite("golden-set", outputs);
console.log(result.summary);
// { total: 10, passed: 9, failed: 1, passRate: 90 }

```

Generate a comparative scorecard across multiple evaluation runs:

```typescript
import { createScorecard, runSuite } from "@/lib/evals/evalRunner";

const suiteRuns = [
  runSuite("golden-set", outputsStrategyA),
  runSuite("coding-proficiency", outputsStrategyB),
];

const scorecard = createScorecard(suiteRuns);
console.log(scorecard.overallPassRate); // 93

```

## Summary

- **Suite-based organization** – Test cases live in [`builtinSuites.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/builtinSuites.ts) as **EvalCase** objects grouped by domain (golden set, coding, reasoning).
- **Flexible validation** – Four evaluation strategies (exact, contains, regex, custom) accommodate diverse output requirements.
- **Combo testing** – The `runSuite` function maps case IDs to model outputs, enabling simultaneous validation of multiple routing strategies.
- **Performance metrics** – `createScorecard` aggregates multi-suite results for CI dashboards and regression detection.
- **Runtime integration** – The [`runtime.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/runtime.ts) module and Next.js API route expose evaluation capabilities to the router's combo engine.

## Frequently Asked Questions

### How does the OmniRoute evals framework handle testing for multiple model combinations?

The framework accepts a mapping of `caseId → actualOutput` in the `runSuite` function, allowing you to collect responses from several models or combo strategies into a single object. By running the same suite against different output collections and comparing scorecards, you can determine which combination yields higher pass rates against the golden set.

### Can I implement custom evaluation logic beyond the built-in strategies?

Yes. The framework supports a **custom** evaluation type where you provide a function that receives the raw LLM output and the case definition. Register your custom suite using `registerSuite` in [`src/lib/evals/evalRunner.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/evals/evalRunner.ts), and the runtime will invoke your function during case evaluation, returning results alongside standard strategy outputs.

### How does the regex strategy prevent performance issues during evaluation?

According to the source code in [`src/lib/evals/evalRunner.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/evals/evalRunner.ts), the regex implementation limits pattern length and automatically strips global (`g`) and sticky (`y`) flags from user-supplied patterns. These safety measures prevent catastrophic backtracking and ensure deterministic matching behavior during large-scale evaluation runs.

### Where can I find example implementations of the eval framework in testing?

Reference implementations appear in [`tests/unit/batch-b-final.test.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/tests/unit/batch-b-final.test.ts), which exercises `runSuite`, `createScorecard`, and error handling scenarios. Additionally, the Next.js endpoint in [`src/app/api/evals/route.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/app/api/evals/route.ts) demonstrates how to integrate the evaluation runtime with external API consumers.