How OmniRoute's Auto Combo Strategy Works: A Technical Deep Dive

OmniRoute's "auto" combo strategy is a self-optimizing routing algorithm that evaluates all available LLM providers using a 15-factor heuristic, applies cost and quota filters, and automatically selects the highest-scoring candidate for each request.

OmniRoute is an open-source intelligent routing layer designed to distribute Large Language Model (LLM) requests across multiple providers. The auto combo strategy serves as the default decision engine, continuously optimizing for cost, latency, and reliability without requiring manual provider selection. This strategy analyzes real-time provider health, pricing tiers, and quota availability to route traffic through the most efficient path available.

Architecture Overview

The auto combo strategy operates through a three-stage pipeline implemented primarily within the open-sse/services/autoCombo/ directory and coordinated by the core combo service.

Stage 1: Candidate Generation

The process initiates in open-sse/services/combo.ts, where the resolveComboTargets() function enumerates all eligible provider-model pairs. This function respects authentication credentials, rate-limiting state, and model lockout configurations to compile the initial candidate pool. Only providers passing basic availability checks proceed to the scoring phase.

Stage 2: Multi-Factor Scoring and Filtering

The auto engine orchestrates a sophisticated evaluation pipeline across several specialized modules:

• engine.ts – Coordinates the overall execution flow, invoking scoring and filtering functions sequentially.
• scoring.ts – Computes the AUTO-COMBO score for each candidate using a 15-factor model that weighs cost, latency, remaining quota, and provider health metrics.
• paidModelFilter.ts – Excludes paid models when free-tier quota remains available.
• freeAccessQuota.ts – Boosts candidates that offer zero-cost access via remaining free-tier allowances.
• connectionBilling.ts – Calculates real-time cost factors based on current billing state and quota consumption.
• providerDiversity.ts – Penalizes over-reliance on single providers to ensure resilience against provider-wide incidents.
• complexityRouter.ts – Adjusts scores based on model complexity and expected token latency, favoring lighter models for simple requests.
• chaosEngine.ts – Randomly deprioritizes a small fraction of candidates to surface latent issues and trigger automatic credential healing.
• routerStrategy.ts – Executes the final selection logic, choosing the candidate with the highest remaining score after all filters.

Stage 3: Execution and Telemetry

Once selected, the target is handed to open-sse/executors/baseExecutor.ts, which performs the actual HTTP request and streams the response back to the client. Post-execution, persistence.ts records granular telemetry data—including latency, cost, and success rates—to inform subsequent routing decisions.

The 15-Factor Scoring Model

The scoring.ts module implements the core heuristic that drives the auto strategy. Each candidate receives a composite score based on:

1. Health & Availability – Providers with open circuit-breakers (tracked in src/shared/utils/circuitBreaker.ts) or recent connection cooldowns receive exclusion flags.
2. Cost Optimization – Real-time pricing tiers and quota usage data from connectionBilling.ts directly influence the score.
3. Free-Tier Preference – Candidates with remaining free-tier quota receive significant scoring bonuses via freeAccessQuota.ts.
4. Provider Diversity – The algorithm penalizes concentration risk, ensuring requests distribute across multiple backends.
5. Complexity & Latency – High-token-count or historically slow models are deprioritized unless the request explicitly demands high capacity.
6. Chaos & Self-Healing – Randomized deprioritization helps detect stale credentials or degraded endpoints before they impact production traffic.

After factor computation, the engine ranks candidates by composite score and passes the top entries to the router strategy.

Implementation Details

Core Engine Orchestration

Located at open-sse/services/autoCombo/engine.ts, the core engine exposes the primary entry point for auto-combo execution. It manages the async pipeline of candidate resolution, parallel scoring, and filter application. The engine also integrates with src/app/api/v1/chat/completions/route.ts, where handleChatCore() invokes the auto strategy when no explicit provider is specified.

Router Strategy Selection

The final selection logic resides in open-sse/services/autoCombo/routerStrategy.ts. This module receives the filtered, scored candidate list and implements the deterministic selection of the highest-scoring provider. If multiple candidates share top-tier scores, the strategy applies secondary tie-breakers based on recent latency performance recorded in the persistence layer.

Usage Examples

Example 1: Default Auto Routing via Node.js SDK

When using the OmniRoute SDK, omitting the combo parameter defaults to the auto strategy:

import { fetchChatCompletions } from '@omniroute/open-sse';

const resp = await fetchChatCompletions({
  model: 'gpt-4o',               // Hint only; auto may override based on availability
  messages: [{ role: 'user', content: 'Explain quantum tunnelling.' }],
});

console.log(resp.choices[0].message.content);

This request flows through handleChatCore() → resolveComboTargets() → engine.ts → scoring.ts before executing against the selected provider.

Example 2: Explicit Auto Selection via REST API

curl -X POST http://localhost:20128/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
        "model": "gpt-4o",
        "messages": [{"role":"user","content":"Explain quantum tunnelling."}],
        "combo": "auto"
      }'

While "combo": "auto" is optional (being the system default), explicit declaration ensures the request flows through the full candidate generation and scoring pipeline.

Example 3: Debugging Candidate Scores

OmniRoute includes a test harness for inspecting intermediate scoring values:

import { runAutoCombo } from 'open-sse/services/autoCombo/engine';

const result = await runAutoCombo({
  modelHint: 'gpt-4o',
  prompt: 'Summarize the repo README.',
});

console.table(result.candidateScores);

Running this snippet reveals the computed 15-factor scores for each provider-model pair, useful for tuning weights or diagnosing routing decisions.

Summary

• Three-stage pipeline: Candidate generation (combo.ts), multi-factor scoring (scoring.ts, filters), and execution (baseExecutor.ts).
• 15-factor heuristic: Evaluates health, cost, quota, diversity, complexity, and chaos resilience to determine optimal routing.
• Self-optimizing: persistence.ts records telemetry from each request, continuously refining future scoring accuracy.
• Default behavior: The auto strategy activates automatically when no specific provider is requested, ensuring zero-configuration optimal routing.

Frequently Asked Questions

How does the auto combo strategy handle provider outages?

The strategy integrates with the circuit-breaker utility at src/shared/utils/circuitBreaker.ts to exclude unhealthy providers during the candidate generation phase. Additionally, chaosEngine.ts proactively tests provider health through randomized traffic shifting, triggering automatic failovers before complete outages occur.

Can developers customize the 15-factor scoring weights?

Currently, the factor weights in scoring.ts are predefined to balance cost, latency, and reliability for general-purpose workloads. While the open-source codebase allows modification of the scoring algorithm itself, runtime configuration of individual factor weights is not exposed through the public API in version 3.8.51.

What distinguishes the auto strategy from manual combo configurations?

Manual combo strategies require explicit provider lists and fixed failover sequences, whereas the auto strategy dynamically generates candidates, applies cost-aware filtering (via paidModelFilter.ts and freeAccessQuota.ts), and self-heals by learning from persistence.ts metrics. This eliminates the need for manual provider prioritization while optimizing for real-time quota and pricing conditions.

Does the auto strategy support streaming responses?

Yes. Once routerStrategy.ts selects the optimal candidate, open-sse/executors/baseExecutor.ts handles the request execution, including Server-Sent Events (SSE) streaming. The auto strategy's decision-making occurs entirely before the connection establishment, ensuring zero latency overhead during the actual response stream.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}