# How to Use the OmniRoute Evals Framework to Test Combo Performance

> Learn to test combo performance using the OmniRoute evals framework. Measure LLM output quality and validate responses against expected criteria for robust routing configurations.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: how-to-guide
- Published: 2026-08-01

---

**The OmniRoute evals framework provides a lightweight, extensible system for measuring LLM output quality by running evaluation suites against combo routing configurations and validating responses against expected criteria.**

The OmniRoute repository includes a comprehensive evaluation system for benchmarking LLM routing performance. This evals framework enables developers to programmatically test combo configurations—OmniRoute's routing engine that selects provider/model targets—and verify correctness against custom or built-in criteria. By leveraging the framework's suite-based architecture, you can automate latency testing and output validation without modifying core routing logic.

## Core Concepts of the OmniRoute Evals Framework

The evals framework in [`src/lib/evals/evalRunner.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/evals/evalRunner.ts) operates on three architectural pillars that orchestrate the testing process.

### Eval Suite

An **Eval Suite** represents a collection of test definitions stored in memory. Built-in suites reside in [`src/lib/evals/evalRunner/builtinSuites.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/evals/evalRunner/builtinSuites.ts), while custom suites can be persisted in the database via `src/lib/db/evals`. Each suite contains metadata and an array of test cases targeting specific model configurations.

### Eval Case

An **Eval Case** defines a single test scenario that runs a request against a provider or combo and validates the response. According to the source code in [`src/lib/evals/evalRunner.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/evals/evalRunner.ts), the `evaluateCase()` function (line 25) implements validation strategies including `exact`, `contains`, `regex`, and `custom`. Custom functions receive the raw LLM text for specialized validation logic.

### Runner

The **Runner** orchestrates suite execution through the `runSuite()` function (line 212 of [`evalRunner.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/evalRunner.ts)). This method retrieves the suite via `getSuite()`, invokes `evaluateCase()` for each test, and aggregates results into a summary object matching the API contract used by [`src/app/api/evals/route.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/app/api/evals/route.ts).

## Testing Combo Performance with Custom Suites

A **combo** represents OmniRoute's routing engine that selects one or more provider/model targets for requests, implemented in [`src/open-sse/services/combo.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/open-sse/services/combo.ts). To benchmark combo performance, create an eval suite that invokes the combo via the HTTP API or the internal `handleComboChat()` function, then processes raw LLM responses through the evaluation pipeline.

### Step 1: Create a Custom Suite

Define a suite with cases targeting your combo configuration. Each case specifies the combo identifier as the model and defines expected validation criteria:

```typescript
// src/lib/evals/customComboSuite.ts
export const comboPerfSuite = {
  id: "combo-performance",
  name: "Combo Performance",
  description: "Measures latency & correctness of a specific combo configuration",
  cases: [
    {
      id: "c1",
      name: "Basic greeting via combo",
      model: "combo-id-example",               // the combo identifier
      input: { messages: [{ role: "user", content: "Hello!" }] },
      expected: { strategy: "contains", value: "hello" },
      tags: ["combo", "latency"],
    },
    // … more cases covering different models, prompts, or payload sizes …
  ],
};

```

### Step 2: Register the Suite

Import and register your custom suite alongside built-in definitions:

```typescript
// src/lib/evals/evalRunner.ts (after built-in registration)
import { comboPerfSuite } from "./customComboSuite";
registerSuite(comboPerfSuite);

```

### Step 3: Invoke the Combo and Gather Outputs

Execute the combo for each case using the combo ID as the model identifier:

```typescript
import { handleComboChat } from "@/open-sse/services/combo";

async function invokeCombo(caseDef) {
  const { model, input } = caseDef;
  // `model` contains the combo ID; the request body matches the OpenAI-compatible schema
  const resp = await handleComboChat({
    comboId: model,
    body: input,
    // additional context (auth, rate-limit, etc.) can be stubbed for tests
  });
  return resp?.choices?.[0]?.message?.content ?? "";
}

```

### Step 4: Execute the Evaluation Suite

Map case IDs to observed outputs and trigger the runner:

```typescript
import { runSuite, listSuites } from "@/lib/evals/evalRunner";

async function evaluateComboSuite() {
  const suiteId = "combo-performance";
  const suite = listSuites().find((s) => s.id === suiteId);
  if (!suite) throw new Error(`Suite ${suiteId} not found`);

  const outputs: Record<string, string> = {};
  for (const c of suite.cases) {
    outputs[c.id] = await invokeCombo(c);
  }

  const result = runSuite(suiteId, outputs);
  console.log("Combo performance result:", result);
}

evaluateComboSuite();

```

### Step 5: Interpret the Summary Results

The `result.summary` object provides total case counts, passed/failed statistics, and pass-rate percentages. For latency analysis, pass the optional `caseMetrics` argument to `runSuite()` (see lines 206-215 of [`src/lib/evals/evalRunner.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/evals/evalRunner.ts)) to include timing data alongside correctness validation.

## Key Source Files

Understanding the following files is essential for implementing combo performance testing:

- **[`src/lib/evals/evalRunner.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/evals/evalRunner.ts)** – Core runner implementing `runSuite()`, `evaluateCase()`, and `createScorecard()`, plus suite registration logic.
- **[`src/lib/evals/evalRunner/builtinSuites.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/evals/evalRunner/builtinSuites.ts)** – Built-in suite definitions including golden sets, coding benchmarks, and safety tests.
- **[`src/lib/evals/runtime.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/evals/runtime.ts)** – Thin wrapper exposing runner functions to the broader codebase.
- **[`src/open-sse/services/combo.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/open-sse/services/combo.ts)** – Combo routing engine that evaluation cases invoke.
- **[`src/app/api/evals/route.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/app/api/evals/route.ts)** – HTTP API endpoint for triggering suite runs from external tools or web interfaces.

## Summary

- The OmniRoute evals framework uses a three-tier architecture: **Eval Suite** (test collections), **Eval Case** (individual validations), and **Runner** (orchestration engine).
- **Combo performance testing** requires creating custom suites that reference combo IDs as model targets and validate outputs using `contains`, `exact`, `regex`, or `custom` strategies.
- Implementation centers on [`src/lib/evals/evalRunner.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/evals/evalRunner.ts), specifically the `runSuite()` function (line 212) and `evaluateCase()` function (line 25).
- Latency metrics can be captured via the `caseMetrics` parameter (lines 206-215) while correctness is verified against expected criteria.
- The framework supports both programmatic invocation via `handleComboChat()` and HTTP API access through [`src/app/api/evals/route.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/app/api/evals/route.ts).

## Frequently Asked Questions

### What evaluation strategies does the OmniRoute evals framework support?

The framework supports four validation strategies implemented in `evaluateCase()`: `exact` (string equality), `contains` (substring matching), `regex` (pattern matching), and `custom` (user-defined function receiving raw LLM text). These strategies are defined in the `expected` field of each Eval Case.

### How do I add latency metrics to combo performance tests?

Pass the optional `caseMetrics` argument to `runSuite()` (lines 206-215 of [`src/lib/evals/evalRunner.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/evals/evalRunner.ts)) containing timing data for each case ID. The runner will include these metrics in the final summary alongside pass/fail statistics, enabling comprehensive performance analysis of combo configurations.

### Can I use built-in suites instead of creating custom ones for combo testing?

Yes. Built-in suites defined in [`src/lib/evals/evalRunner/builtinSuites.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/evals/evalRunner/builtinSuites.ts) can be reused for combo testing by modifying the model field to reference your combo ID. However, custom suites are recommended when you need specific input payloads, specialized validation criteria, or tags that categorize combo-specific test scenarios.

### Where is the HTTP endpoint for running evaluation suites remotely?

The HTTP API endpoint is implemented in [`src/app/api/evals/route.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/app/api/evals/route.ts). This route consumes the same `runSuite()` function used internally, allowing external CI/CD pipelines or monitoring tools to trigger combo performance evaluations via HTTP requests without direct code integration.