How OmniRoute Evaluates LLM Providers in Real-Time Using 14-Factor Auto-Combo Scoring

OmniRoute's Auto-Combo engine computes a real-time weighted score for every LLM provider candidate by combining live telemetry—including quota, circuit-breaker health, cost, latency, stability, and contextual affinity—into a single normalized ranking from 0 to 1.

The Auto-Combo scoring system is the core intelligence behind OmniRoute's ability to route requests dynamically. Instead of relying on static rules, it evaluates every available provider-model pair at request time using 14 weighted factors. This article breaks down how the scoring algorithm works, where it lives in the codebase, and how you can customize it for your workload.

Where the Scoring Logic Lives

The scoring implementation is concentrated in two files:

• Core algorithm: [open-sse/services/autoCombo/scoring.ts](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.50/open-sse/services/autoCombo/scoring.ts) — contains calculateFactors, calculateScore, and scorePool
• Documentation: [docs/routing/AUTO-COMBO.md](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.50/docs/routing/AUTO-COMBO.md) — explains the 14-factor weighting system and mode packs

All scoring functions are pure TypeScript utilities with no external dependencies beyond the tier resolver.

Step 1: Gathering Live Telemetry with ProviderCandidate

Before any scoring happens, OmniRoute builds a ProviderCandidate object for every available provider-model pair. This happens at request time, so the data reflects the current state of each connection.

The ProviderCandidate interface (defined in scoring.ts) captures:

Field	Purpose
quotaRemaining	Tokens or requests left in current window
circuitBreakerState	Closed, half-open, or open (for resilience)
costPer1MTokens	Pricing for cost-optimization
p95LatencyMs	Measured tail latency
latencyStdDev	Stability indicator (lower is better)
getTaskFitness	Plug-in function for model-task matching
tier / quotaResetInterval	Account-level priority handling
Affinity hints	Context, cache, session, reset-window, connection density

This telemetry is fetched fresh for every routing decision, making the 14-factor Auto-Combo scoring system truly real-time.

Step 2: Normalizing Pool-Wide Maxima

The calculateFactors function first computes inverse baselines across the candidate pool:

const maxCost = Math.max(...candidates.map(c => c.costPer1MTokens));
const maxLatency = Math.max(...candidates.map(c => c.p95LatencyMs));
const maxStdDev = Math.max(...candidates.map(c => c.latencyStdDev));

These maxima enable relative scoring: a candidate's cost or latency is judged against the worst performer in the pool, not against absolute thresholds. This keeps scores portable across different deployment environments.

Step 3: Deriving the 14 Scoring Factors

Each candidate is transformed into a ScoringFactors object with all values clamped to [0, 1] via clamp01:

Factor	Calculation	Meaning
quota	quotaRemaining / 100	More quota remaining = higher score
health	1.0 (closed), 0.5 (half-open), 0 (open)	Circuit-breaker state
costInv	1 - (cost / maxCost)	Cheaper providers score higher
latencyInv	1 - (latency / maxLatency)	Faster providers score higher
stability	1 - (stdDev / maxStdDev)	More consistent latency = higher score
taskFit	getTaskFitness(model, taskType)	Model-task alignment from plug-in
tierPriority	calculateTierScore(tier, resetInterval)	Account tier preference
tierAffinity	Optional affinity boost	Tier-to-tier routing preference
specificityMatch	Optional match score	Exact model matching
contextAffinity	Optional state score	Sticky context routing
cacheAffinity	Optional hit rate	Prompt cache benefit
sessionAvailability	Optional availability	Session continuity
resetWindowAffinity	Optional temporal score	Quota reset timing
connectionDensity	Optional load balance	Connection distribution

Factor extraction is handled by calculateFactors in the scoring module. All optional affinity factors default to neutral values (typically 0.5) when not provided.

Step 4: Applying Configurable Weights

The 14-factor Auto-Combo scoring system uses DEFAULT_WEIGHTS as a baseline, but callers can supply custom weights via the ScoringWeights interface:

export const DEFAULT_WEIGHTS: ScoringWeights = {
  quota: 0.10,
  health: 0.15,
  costInv: 0.15,
  latencyInv: 0.15,
  taskFit: 0.15,
  stability: 0.10,
  tierPriority: 0.05,
  tierAffinity: 0.05,
  specificityMatch: 0.03,
  contextAffinity: 0.03,
  cacheAffinity: 0.02,
  sessionAvailability: 0.02,
  resetWindowAffinity: 0.02,
  connectionDensity: 0.03,
};

The validateWeights helper ensures weights sum to approximately 1. For UI-driven customizations, normalizeScoringWeights rescales any input to a proper distribution.

Step 5: Computing the Final Score

The calculateScore function performs a weighted sum:

export function calculateScore(
  factors: ScoringFactors,
  weights: ScoringWeights
): number {
  const score = Object.keys(weights).reduce((sum, key) => {
    return sum + (factors[key] ?? 0.5) * (weights[key] ?? 0);
  }, 0);
  return clamp01(score);
}

The result is clamped to [0, 1] and represents the candidate's overall suitability for the current request.

Step 6: Selecting the Best Provider with scorePool

The scorePool function orchestrates the entire pipeline:

export function scorePool(
  candidates: ProviderCandidate[],
  taskType: string,
  weights: ScoringWeights = DEFAULT_WEIGHTS,
  taskFitness: TaskFitnessFn = defaultTaskFitness,
  manifestHint: ManifestHint | null = null
): ScoredProvider[] {
  const maxima = computePoolMaxima(candidates);
  
  return candidates
    .map(candidate => {
      const factors = calculateFactors(candidate, maxima, taskType, taskFitness, manifestHint);
      const score = calculateScore(factors, weights);
      return { ...candidate, score, factors };
    })
    .sort((a, b) => b.score - a.score);
}

The returned ScoredProvider[] is sorted by descending score. The first element is the optimal choice for the request.

Practical Example: Customizing Weights for Latency-Sensitive Workloads

import {
  scorePool,
  DEFAULT_WEIGHTS,
  ScoringWeights,
} from '@omniroute/open-sse/services/autoCombo/scoring';

// Retrieve live candidates from the connection registry
const candidates = await getProviderCandidates(); // → ProviderCandidate[]

// Define task-specific fitness: boost code-generation models
function taskFitness(model: string, task: string): number {
  return task === 'coding' && model.includes('codex') ? 0.9 : 0.5;
}

// Customize weights for latency-critical routing
const fastWeights: ScoringWeights = {
  ...DEFAULT_WEIGHTS,
  latencyInv: 0.25,    // prioritize speed
  costInv: 0.10,       // de-emphasize cost
  stability: 0.15,     // ensure consistency
};

const scored = scorePool(
  candidates,
  'coding',
  fastWeights,
  taskFitness,
  null
);

const best = scored[0];
console.log(`Selected ${best.provider}/${best.model} (score=${best.score.toFixed(3)})`);
// Example output: Selected azure/gpt-4 (score=0.847)

How Real-Time Evaluation Enables Dynamic Routing

Because all 14 factors are computed from live telemetry, OmniRoute can adapt instantly to:

• Quota exhaustion — providers with depleted budgets automatically score lower via quota and tierPriority
• Circuit breaker trips — unhealthy providers score near-zero via health
• Latency spikes — degraded providers lose ground via latencyInv and stability
• Context affinity — sticky routing via contextAffinity and sessionAvailability
• Cache optimization — prompt cache hits boost cacheAffinity

This makes the 14-factor Auto-Combo scoring system suitable for production workloads requiring sub-second failover without manual intervention.

Key Files and Functions Reference

Component	Location	Role
scorePool	open-sse/services/autoCombo/scoring.ts	Main entry point for candidate ranking
calculateFactors	Same file	Transforms telemetry into normalized factors
calculateScore	Same file	Weighted sum with clamping
ProviderCandidate interface	Same file	Telemetry schema for candidates
DEFAULT_WEIGHTS	Same file	Baseline 14-factor weight distribution
validateWeights / normalizeScoringWeights	Same file	Weight validation and rescaling
calculateTierScore	Imported from ../tierResolver	Account tier scoring
classifyTier	../tierResolver	Tier classification utility
User documentation	docs/routing/AUTO-COMBO.md	Conceptual guide and mode packs

Summary

• Real-time telemetry — ProviderCandidate collects quota, health, cost, latency, and affinity data fresh for every request
• 14 normalized factors — calculateFactors produces values in [0,1] including inverses for cost/latency and plug-in task fitness
• Configurable weights — DEFAULT_WEIGHTS covers all 14 factors, customizable via ScoringWeights with validation
• Weighted scoring — calculateScore combines factors into a single suitability score
• Ranked selection — scorePool returns candidates sorted by score, enabling automatic optimal selection

The scoring system is stateless, pure, and fast enough to run on every request without caching, ensuring routing decisions always reflect current conditions.

Frequently Asked Questions

What are the 14 factors in OmniRoute's Auto-Combo scoring?

The 14 factors are: quota, health, costInv, latencyInv, taskFit, stability, tierPriority, tierAffinity, specificityMatch, contextAffinity, cacheAffinity, sessionAvailability, resetWindowAffinity, and connectionDensity. They cover resource availability, performance, cost, reliability, account priority, and contextual routing preferences.

How does OmniRoute handle providers with depleted quotas?

The quota factor directly incorporates quotaRemaining, while tierPriority and resetWindowAffinity consider account tier and quota reset timing. A provider with zero remaining quota scores near-zero on quota, causing it to drop in the rankings unless other weights are extreme.

Can I create custom weight profiles for different use cases?

Yes. Pass a custom ScoringWeights object to scorePool. The validateWeights function ensures your weights sum to ~1, and normalizeScoringWeights can rescale UI inputs. Common profiles include "fast" (high latencyInv), "cheap" (high costInv), and "resilient" (high health and stability).

Why does OmniRoute use inverse scoring for cost and latency?

Cost and latency are minimization objectives (lower is better), while scores must maximize toward 1. The formulas 1 - (value / maxValue) flip these so that cheaper/faster candidates score higher, keeping all factors directionally consistent.

How does the scoring system respond to circuit breaker state changes?

The health factor reads circuitBreakerState directly: closed circuits score 1.0, half-open score 0.5, and open circuits score 0.0. Because telemetry is fetched at request time, a tripped breaker immediately zeros out the provider's health contribution, removing it from contention.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}