How to Use the OmniRoute Evals Framework to Test Combo Performance

The OmniRoute evals framework provides a lightweight, extensible system for measuring LLM output quality by running evaluation suites against combo routing configurations and validating responses against expected criteria.

The OmniRoute repository includes a comprehensive evaluation system for benchmarking LLM routing performance. This evals framework enables developers to programmatically test combo configurations—OmniRoute's routing engine that selects provider/model targets—and verify correctness against custom or built-in criteria. By leveraging the framework's suite-based architecture, you can automate latency testing and output validation without modifying core routing logic.

Core Concepts of the OmniRoute Evals Framework

The evals framework in src/lib/evals/evalRunner.ts operates on three architectural pillars that orchestrate the testing process.

Eval Suite

An Eval Suite represents a collection of test definitions stored in memory. Built-in suites reside in src/lib/evals/evalRunner/builtinSuites.ts, while custom suites can be persisted in the database via src/lib/db/evals. Each suite contains metadata and an array of test cases targeting specific model configurations.

Eval Case

An Eval Case defines a single test scenario that runs a request against a provider or combo and validates the response. According to the source code in src/lib/evals/evalRunner.ts, the evaluateCase() function (line 25) implements validation strategies including exact, contains, regex, and custom. Custom functions receive the raw LLM text for specialized validation logic.

Runner

The Runner orchestrates suite execution through the runSuite() function (line 212 of evalRunner.ts). This method retrieves the suite via getSuite(), invokes evaluateCase() for each test, and aggregates results into a summary object matching the API contract used by src/app/api/evals/route.ts.

Testing Combo Performance with Custom Suites

A combo represents OmniRoute's routing engine that selects one or more provider/model targets for requests, implemented in src/open-sse/services/combo.ts. To benchmark combo performance, create an eval suite that invokes the combo via the HTTP API or the internal handleComboChat() function, then processes raw LLM responses through the evaluation pipeline.

Step 1: Create a Custom Suite

Define a suite with cases targeting your combo configuration. Each case specifies the combo identifier as the model and defines expected validation criteria:

// src/lib/evals/customComboSuite.ts
export const comboPerfSuite = {
  id: "combo-performance",
  name: "Combo Performance",
  description: "Measures latency & correctness of a specific combo configuration",
  cases: [
    {
      id: "c1",
      name: "Basic greeting via combo",
      model: "combo-id-example",               // the combo identifier
      input: { messages: [{ role: "user", content: "Hello!" }] },
      expected: { strategy: "contains", value: "hello" },
      tags: ["combo", "latency"],
    },
    // … more cases covering different models, prompts, or payload sizes …
  ],
};

Step 2: Register the Suite

Import and register your custom suite alongside built-in definitions:

// src/lib/evals/evalRunner.ts (after built-in registration)
import { comboPerfSuite } from "./customComboSuite";
registerSuite(comboPerfSuite);

Step 3: Invoke the Combo and Gather Outputs

Execute the combo for each case using the combo ID as the model identifier:

import { handleComboChat } from "@/open-sse/services/combo";

async function invokeCombo(caseDef) {
  const { model, input } = caseDef;
  // `model` contains the combo ID; the request body matches the OpenAI-compatible schema
  const resp = await handleComboChat({
    comboId: model,
    body: input,
    // additional context (auth, rate-limit, etc.) can be stubbed for tests
  });
  return resp?.choices?.[0]?.message?.content ?? "";
}

Step 4: Execute the Evaluation Suite

Map case IDs to observed outputs and trigger the runner:

import { runSuite, listSuites } from "@/lib/evals/evalRunner";

async function evaluateComboSuite() {
  const suiteId = "combo-performance";
  const suite = listSuites().find((s) => s.id === suiteId);
  if (!suite) throw new Error(`Suite ${suiteId} not found`);

  const outputs: Record<string, string> = {};
  for (const c of suite.cases) {
    outputs[c.id] = await invokeCombo(c);
  }

  const result = runSuite(suiteId, outputs);
  console.log("Combo performance result:", result);
}

evaluateComboSuite();

Step 5: Interpret the Summary Results

The result.summary object provides total case counts, passed/failed statistics, and pass-rate percentages. For latency analysis, pass the optional caseMetrics argument to runSuite() (see lines 206-215 of src/lib/evals/evalRunner.ts) to include timing data alongside correctness validation.

Key Source Files

Understanding the following files is essential for implementing combo performance testing:

• src/lib/evals/evalRunner.ts – Core runner implementing runSuite(), evaluateCase(), and createScorecard(), plus suite registration logic.
• src/lib/evals/evalRunner/builtinSuites.ts – Built-in suite definitions including golden sets, coding benchmarks, and safety tests.
• src/lib/evals/runtime.ts – Thin wrapper exposing runner functions to the broader codebase.
• src/open-sse/services/combo.ts – Combo routing engine that evaluation cases invoke.
• src/app/api/evals/route.ts – HTTP API endpoint for triggering suite runs from external tools or web interfaces.

Summary

• The OmniRoute evals framework uses a three-tier architecture: Eval Suite (test collections), Eval Case (individual validations), and Runner (orchestration engine).
• Combo performance testing requires creating custom suites that reference combo IDs as model targets and validate outputs using contains, exact, regex, or custom strategies.
• Implementation centers on src/lib/evals/evalRunner.ts, specifically the runSuite() function (line 212) and evaluateCase() function (line 25).
• Latency metrics can be captured via the caseMetrics parameter (lines 206-215) while correctness is verified against expected criteria.
• The framework supports both programmatic invocation via handleComboChat() and HTTP API access through src/app/api/evals/route.ts.

Frequently Asked Questions

What evaluation strategies does the OmniRoute evals framework support?

The framework supports four validation strategies implemented in evaluateCase(): exact (string equality), contains (substring matching), regex (pattern matching), and custom (user-defined function receiving raw LLM text). These strategies are defined in the expected field of each Eval Case.

How do I add latency metrics to combo performance tests?

Pass the optional caseMetrics argument to runSuite() (lines 206-215 of src/lib/evals/evalRunner.ts) containing timing data for each case ID. The runner will include these metrics in the final summary alongside pass/fail statistics, enabling comprehensive performance analysis of combo configurations.

Can I use built-in suites instead of creating custom ones for combo testing?

Yes. Built-in suites defined in src/lib/evals/evalRunner/builtinSuites.ts can be reused for combo testing by modifying the model field to reference your combo ID. However, custom suites are recommended when you need specific input payloads, specialized validation criteria, or tags that categorize combo-specific test scenarios.

Where is the HTTP endpoint for running evaluation suites remotely?

The HTTP API endpoint is implemented in src/app/api/evals/route.ts. This route consumes the same runSuite() function used internally, allowing external CI/CD pipelines or monitoring tools to trigger combo performance evaluations via HTTP requests without direct code integration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}