OmniRoute Scoring Engine: The 13 Candidate Evaluation Criteria Explained

The OmniRoute auto-combo scoring engine evaluates provider candidates using 13 normalized telemetry factors—quota, health, cost, latency, task fit, stability, tier priority, tier affinity, specificity match, context affinity, cache affinity, reset window affinity, and connection density—that are weighted and summed to produce a final score between 0 and 1.

OmniRoute is an open-source routing layer for LLM and AI service providers. Its auto-combo routing engine, implemented in open-sse/services/autoCombo/scoring.ts, automatically selects the optimal provider for each request by scoring candidates against a multi-dimensional evaluation framework. This article breaks down every candidate evaluation criterion, how each is computed from raw telemetry, and how the final ranking is determined.

How the Scoring Pipeline Works

The scoring process follows three distinct stages as implemented in the source code:

1. Telemetry collection — Gather raw metrics for each candidate provider (quota remaining, circuit-breaker state, latency statistics, etc.)
2. Factor normalization — Convert each metric into a normalized factor in the range [0, 1] using the calculateFactors function
3. Weighted aggregation — Apply configurable scoring weights via calculateScore to produce a final ranking value, clamped to [0, 1]

All 13 evaluation criteria are defined by the ScoringFactors interface and computed within calculateFactors (lines 20–37).

The 13 Candidate Evaluation Criteria

Each criterion is derived from provider telemetry and normalized before entering the weighted sum. Here's the complete breakdown:

Resource Availability Factors

Criterion	Derivation	Source Lines
quota	candidate.quotaRemaining / 100 — percentage of remaining quota for the account	20–21
health	Circuit-breaker state: CLOSED = 1, HALF_OPEN = 0.5, otherwise 0	22–26
connectionDensity	Normalized connection pool size: (size - 1) / 10	37

Cost and Performance Factors

Criterion	Derivation	Source Lines
costInv	Inverted cost: 1 - candidate.costPer1MTokens / maxCost	27
latencyInv	Inverted P95 latency: 1 - candidate.p95LatencyMs / maxLatency	28
stability	Inverted latency variance: 1 - candidate.latencyStdDev / maxStdDev	30

Task and Model Alignment Factors

Criterion	Derivation	Source Lines
taskFit	Model-to-task fitness from pluggable getTaskFitness() function	29
tierPriority	Normalized tier score (Ultra/Pro/Standard/Free) plus reset-interval bonus via calculateTierScore	31
tierAffinity	Alignment with manifest's recommended-min-tier via calculateTierAffinity	32
specificityMatch	Match against manifest's specificity score via calculateSpecificityMatch	33

Session and Cache Affinity Factors

Criterion	Derivation	Source Lines
contextAffinity	Re-use preference for current session's provider/account/model; defaults to 0.5	34
cacheAffinity	Preference for cached prompt-key hits; defaults to 0	35
resetWindowAffinity	Preference for providers with nearer quota-reset windows; defaults to 0.5	36

Each factor is individually clamped to [0, 1] by clamp01 (lines 15–18) to prevent malformed telemetry from corrupting the final score.

Default Weight Configuration

The DEFAULT_WEIGHTS object in scoring.ts establishes the relative importance of each criterion:

export const DEFAULT_WEIGHTS: ScoringWeights = {
  quota: 0.15,
  health: 0.20,
  costInv: 0.15,
  latencyInv: 0.12,
  taskFit: 0.08,
  stability: 0.05,
  tierPriority: 0.05,
  tierAffinity: 0.05,
  specificityMatch: 0.05,
  contextAffinity: 0.05,
  cacheAffinity: 0,          // disabled by default
  resetWindowAffinity: 0,    // disabled by default
  connectionDensity: 0.05,
};

Notable design decisions in the default configuration:

• Health (0.20) and quota (0.15) receive highest priority — reliability and capacity dominate routing decisions
• Cache affinity and reset window affinity are zeroed out unless explicitly enabled
• Weights are automatically normalized to sum to 1 by normalizeScoringWeights (lines 59–76)

Calculating the Final Score

The calculateScore function performs a weighted sum of all factors, with defensive null-coalescing for optional weights:

export function calculateScore(factors: ScoringFactors, weights: ScoringWeights): number {
  return clamp01(
    weights.quota * factors.quota +
    weights.health * factors.health +
    weights.costInv * factors.costInv +
    weights.latencyInv * factors.latencyInv +
    weights.taskFit * factors.taskFit +
    weights.stability * factors.stability +
    weights.tierPriority * factors.tierPriority +
    (weights.tierAffinity ?? 0) * factors.tierAffinity +
    (weights.specificityMatch ?? 0) * factors.specificityMatch +
    (weights.contextAffinity ?? 0) * factors.contextAffinity +
    (weights.cacheAffinity ?? 0) * (factors.cacheAffinity ?? 0) +
    (weights.resetWindowAffinity ?? 0) * factors.resetWindowAffinity +
    (weights.connectionDensity ?? 0) * factors.connectionDensity
  );
}

The clamped result (0–1) directly determines candidate ranking. Higher scores indicate more suitable providers for the current request.

Practical Usage Examples

Scoring a Candidate Pool

Use scorePool() to rank providers for a specific task:

// open-sse/services/autoCombo/scoring.ts
const scored = scorePool(
  providerCandidates,    // ProviderCandidate[]
  "chat.completions",    // task type
  customWeights,         // optional ScoringWeights override
  modelFitnessFn,        // optional task-fit function
  manifestRoutingHint    // optional manifest hint
);
// Returns: ScoredProvider[] sorted by descending score

Customizing Weights Per Request

Override default weights through the autoConfig.weights object in your combo request:

{
  "autoConfig": {
    "weights": {
      "quota": 0.30,
      "health": 0.25,
      "costInv": 0.20,
      "latencyInv": 0.15,
      "taskFit": 0.10,
      "stability": 0,
      "tierPriority": 0,
      "tierAffinity": 0,
      "specificityMatch": 0,
      "contextAffinity": 0,
      "cacheAffinity": 0,
      "resetWindowAffinity": 0,
      "connectionDensity": 0
    }
  }
}

The parseAutoConfig routine in open-sse/services/combo/autoConfig.ts processes this configuration, normalizes the weights, and passes them to scorePool.

Extending with Custom Factors

Add new evaluation criteria by extending the core interfaces:

// Step 1: Extend interfaces
interface ScoringFactors { 
  reliability: number;
  // ... existing fields
}

interface ScoringWeights { 
  reliability: number;
  // ... existing fields
}

// Step 2: Add to calculateScore
weights.reliability * factors.reliability

// Step 3: Populate in calculateFactors
reliability: clamp01(candidate.reliabilityMetric ?? 0.5)

Update DEFAULT_WEIGHTS with your new factor's default value. All downstream code, including the test suite in tests/unit/auto-combo-scoring-clamp.test.ts, will recognize the extension.

Key Source Files

File	Purpose
open-sse/services/autoCombo/scoring.ts	Core scoring logic: ScoringFactors, ScoringWeights, calculateFactors, calculateScore, DEFAULT_WEIGHTS
open-sse/services/combo/autoConfig.ts	Parses per-combo weight overrides and routing configuration
open-sse/services/combo/targetResolution.ts	Orchestrates candidate resolution and calls scorePool
src/lib/usage/comboScoringInspector.ts	Debug/inspection utilities for scoring outcomes
tests/unit/auto-combo-scoring-clamp.test.ts	Validates factor clamping and weight normalization

Summary

• 13 normalized factors constitute OmniRoute's complete candidate evaluation criteria: quota, health, costInv, latencyInv, taskFit, stability, tierPriority, tierAffinity, specificityMatch, contextAffinity, cacheAffinity, resetWindowAffinity, and connectionDensity
• Each factor is derived from live telemetry and clamped to [0, 1] before weighting
• Configurable weights default to health-first (0.20) with cost and quota as secondary priorities (0.15 each)
• The scoring pipeline is fully extensible — new factors can be added by updating interfaces and the calculation functions
• All critical logic resides in open-sse/services/autoCombo/scoring.ts with per-request customization supported through autoConfig.weights

Frequently Asked Questions

How does OmniRoute handle providers with zero remaining quota?

The quota factor evaluates to 0 when candidate.quotaRemaining is 0, effectively eliminating that provider from selection unless other factors and weight adjustments compensate. Since quota carries a 0.15 default weight, a zero-quota provider faces severe penalization in the final ranking.

Can I disable specific evaluation criteria entirely?

Yes. Set any weight to 0 in your custom ScoringWeights configuration. The calculateScore function uses null-coalescing (?? 0) for optional weights, so omitted or zero-valued criteria contribute nothing to the final sum. This is how cacheAffinity and resetWindowAffinity are disabled by default.

What happens if telemetry data is missing or malformed?

All factors pass through clamp01 during calculation in calculateFactors (lines 15–18). This guarantees every input is bounded to [0, 1] regardless of source data quality. The defensive design prevents a single corrupted metric from destabilizing the entire routing decision.

How does OmniRoute balance cost versus latency in routing decisions?

By default, costInv (0.15) and latencyInv (0.12) are weighted similarly but not identically. Cost receives slight precedence. You can invert this priority by supplying custom weights — for latency-sensitive workloads, increase latencyInv and reduce costInv proportionally.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}