How OmniRoute Uses Auto-Combo Scoring Factors to Select the Best Model

OmniRoute evaluates provider-model candidates using a weighted scoring system that combines latency, quota, health, cost, diversity, and task fitness into a single numeric score, then selects the highest-ranked candidate from the pool.

OmniRoute is an intelligent routing layer for LLM applications hosted at diegosouzapw/OmniRoute. Its auto-combo scoring factors provide a multi-dimensional evaluation framework that transforms raw provider metrics into actionable rankings, enabling the system to dynamically select the optimal model for each request based on real-time conditions and business priorities.

The Three-Stage Scoring Pipeline

The scoring logic lives in open-sse/services/autoCombo/scoring.ts and executes in three distinct phases to transform raw provider data into a final selection.

Stage 1: Gathering Factors with calculateFactors

For each eligible ProviderCandidate, the engine constructs a ScoringFactors object through the calculateFactors function located at line 179. This object aggregates seven critical metrics:

• Latency measurements from recent requests
• Quota remaining as a percentage of total capacity
• Health status derived from error rates and cooldown states
• Normalized cost per token or request
• Diversity boost to encourage traffic distribution
• Task fitness scores indicating model specialization alignment
• Mode-pack weights reflecting user-defined priorities

Stage 2: Applying Weights with calculateScore

The system references a ScoringWeights object—default exported as DEFAULT_WEIGHTS—to assign numerical importance to each factor. The calculateScore function at line 98 performs a weighted sum calculation:

const score = (latencyFactor * weights.latency) +
              (quotaFactor * weights.quota) +
              (healthFactor * weights.health) +
              (costFactor * weights.cost) +
              ...;

This produces a single numeric score where higher values indicate more suitable candidates.

Stage 3: Ranking the Pool with scorePool

The scorePool function at line 215 orchestrates the evaluation across all candidates. It executes calculateScore for every entry, filters invalid configurations through validateWeights, and returns an array of ScoredProvider objects sorted in descending order. The combo router in open-sse/services/autoCombo/engine.ts then selects the first entry, applying tie-breaker strategies when scores are equal. The final selection drives the handleComboChat flow in open-sse/services/combo/combo.ts.

The Seven Scoring Factors Explained

Each factor in the ScoringFactors object contributes to the final ranking based on specific business logic:

latencyFactor
Measures the round-trip time to the upstream provider. Lower latency values increase the score, prioritizing faster response times when the "fast" mode-pack is active.

quotaFactor
Represents the percentage of remaining request quota for the provider account. Providers approaching quota limits receive reduced scores, and those past the hard cutoff are removed from consideration entirely before scoring begins (as verified in combo-quota-cutoff tests).

healthFactor
Derived from recent error rates and active cooldown periods. Healthier providers receive multiplicative score bonuses, while degraded providers are penalized proportionally to their failure rates.

costFactor
Reflects the normalized cost per token or per request. When the "cheap" mode-pack is selected, this factor receives elevated weight, favoring cost-effective providers.

diversityBoost
A bonus value that increases when selecting a provider would improve overall traffic distribution across the provider pool. This prevents over-concentration on single high-performing models.

taskFitness
A specialization score from taskFitness.ts indicating how well the model matches the request's task type (coding, reasoning, vision, etc.). Domain-specific models receive higher rankings for relevant workloads.

modePackWeight
Dynamic weight overrides supplied by the selected mode pack (e.g., MODE_PACKS["reliable"]). These override default weights to prioritize specific operational characteristics without changing the underlying scoring logic.

Implementation Examples

Scoring a Single Candidate

Use calculateFactors and calculateScore to evaluate individual providers:

import {
  calculateFactors,
  calculateScore,
  DEFAULT_WEIGHTS,
} from '@omniroute/open-sse/services/autoCombo/scoring';

const candidate = {
  provider: 'openai',
  model: 'gpt-4o',
  latencyMs: 78,
  quotaRemaining: 0.73,
  healthScore: 0.95,
  costPer1kTokens: 0.015,
  taskType: 'coding',
};

const factors = calculateFactors(candidate, {
  latencyFactor: 1,
  quotaFactor: 1,
  healthFactor: 1,
  costFactor: 1,
  diversityBoost: 0,
  taskFitness: 1,
});

const score = calculateScore(factors, DEFAULT_WEIGHTS);
console.log('Score for candidate:', score);

Scoring an Entire Pool

The scorePool function handles batch evaluation and sorting:

import {
  scorePool,
  DEFAULT_WEIGHTS,
} from '@omniroute/open-sse/services/autoCombo/scoring';
import { selectProvider } from '@omniroute/open-sse/services/autoCombo/engine';

const pool: ProviderCandidate[] = await fetchCandidatesFromDb();

// Returns array sorted by score (highest first)
const ranked = scorePool(pool, DEFAULT_WEIGHTS);
const best = ranked[0];
console.log('Best provider:', best.provider, best.model);

// Or use the engine for full selection logic
const chosen = await selectProvider({
  candidates: pool,
  requestContext: { /* request data */ },
  weights: DEFAULT_WEIGHTS,
});

Using Custom Mode Packs

Override default weights with predefined mode packs for specific optimization targets:

import { MODE_PACKS } from '@omniroute/open-sse/services/autoCombo/modePacks';
import { scorePool } from '@omniroute/open-sse/services/autoCombo/scoring';

const cheapWeights = MODE_PACKS['cheap'];
const cheapRanked = scorePool(pool, cheapWeights);
console.log('Cost-optimized winner:', cheapRanked[0].provider);

Generating Scoring Inspector Reports

Debug routing decisions using the diagnostic API at src/app/api/usage/combo-scoring-inspector/route.ts:

import { buildComboScoringInspectorResponse } from '@/lib/usage/comboScoringInspector';

const comboId = '123e4567-e89b-12d3-a456-426614174000';
const report = await buildComboScoringInspectorResponse({ comboId });

console.log(report.scoring?.combos?.[0]?.targets);

Key Implementation Files

File	Purpose
open-sse/services/autoCombo/scoring.ts	Core factor calculation, weighting, and pool ranking
open-sse/services/autoCombo/engine.ts	High-level selection logic calling scorePool
open-sse/services/combo/combo.ts	Consumer of scoring results via handleComboChat
open-sse/services/autoCombo/modePacks.ts	Pre-defined weight bundles for "fast", "cheap", "reliable"
src/lib/usage/comboScoringInspector.ts	Diagnostic report generation
src/shared/validation/schemas/combo.ts	Zod schema for validating custom ScoringWeights
tests/unit/auto-combo-scoring-clamp.test.ts	Unit tests for weight clamping behavior
tests/unit/complexity-aware-scoring-wiring.test.ts	Integration tests with complexity router

Summary

• Auto-combo scoring factors combine latency, quota, health, cost, diversity, and task fitness into a weighted numeric score for each provider-model candidate.
• The scoring pipeline executes in three stages: factor gathering via calculateFactors (line 179), weight application via calculateScore (line 98), and pool ranking via scorePool (line 215).
• Mode packs allow runtime customization of scoring priorities without modifying core logic.
• Providers with insufficient quota are filtered before scoring, while health penalties and diversity bonuses dynamically adjust rankings.
• The system is fully observable through the scoring inspector API, which reconstructs the decision factors for any historical request.

Frequently Asked Questions

What are the default weights in OmniRoute's auto-combo scoring?

The system exports DEFAULT_WEIGHTS from open-sse/services/autoCombo/scoring.ts, which provides balanced weighting across all seven factors. Operators can view these defaults in the source code or override them using mode packs like MODE_PACKS["fast"] or MODE_PACKS["cheap"] to prioritize specific operational characteristics.

How does OmniRoute handle providers with low quota in the scoring system?

Providers that have exceeded the hard quota cutoff defined in combo-quota-cutoff tests are removed from the candidate pool before scoring begins. For remaining providers, the quotaFactor reduces the score proportionally to quota consumption, ensuring heavily utilized providers are deprioritized in favor of those with ample capacity.

Can I customize the auto-combo scoring factors for specific requests?

Yes, the ScoringWeights schema in src/shared/validation/schemas/combo.ts allows custom weight configurations. Users can inject custom weights through request headers or apply predefined mode packs. The validateWeights function ensures custom configurations meet system constraints before they are applied to the scoring calculation.

How does OmniRoute prevent overloading a single high-performing provider?

The diversityBoost factor adds a bonus to providers that would improve overall traffic distribution. When combined with the weighted scoring system, this encourages the router to spread requests across multiple providers rather than concentrating all traffic on the single highest-scoring candidate, improving system resilience.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}