OmniRoute Multi-Provider Combo Routing: How 17+ Strategies Work

OmniRoute's combo routing system resolves a named configuration into an ordered list of LLM targets and applies one of 17 routing strategies—such as auto, cost-optimized, and power-of-two-choices—to determine execution order, falling back to the next provider until a successful response is returned.

OmniRoute is an open-source LLM gateway that unifies access to hundreds of providers through a single OpenAI-compatible API. Its multi-provider combo routing feature allows developers to define groups of models with specific failover and load-balancing behaviors. According to the OmniRoute source code, the system implements 17 distinct routing strategies that control how requests are distributed across configured targets.

How Combo Routing Resolves Targets

The combo routing engine operates in two phases: target resolution and iterative execution. When a client sends a request to /api/v1/chat/completions using a combo name as the model identifier, the system expands that combo into concrete provider targets.

Target Resolution with resolveComboTargets()

In src/lib/db/combo.ts, the resolveComboTargets() function reads the combo definition from the database and constructs a ResolvedComboTarget[] array. Each target object contains:

• Provider ID and model name
• Account credentials and API keys
• Per-target weight and priority overrides
• Context window constraints

This ordered list reflects the selected routing strategy applied to the raw combo configuration as defined in src/shared/constants/routingStrategies.ts.

Execution Loop in handleComboChat()

The resolved targets are passed to handleComboChat() in open-sse/services/combo.ts. This function iterates through the target array, invoking handleSingleModel() for each entry:

1. Attempt the first target in the list
2. If successful, stream the response to the client and terminate
3. If failed, log the error to open-sse/utils/usageTracking.ts and proceed to the next target
4. Return an error only if all targets exhaust their retry limits

This failover mechanism ensures high availability across multiple providers without requiring client-side complexity.

The 17+ Routing Strategies Explained

OmniRoute defines 17 built-in strategies in src/shared/constants/routingStrategies.ts. These algorithms range from simple round-robin distributions to sophisticated, metric-driven selection optimized for modern multi-core servers.

Priority-based strategies:

• priority – Honors an explicit ordering defined in the combo configuration, ideal for strict failover chains (primary → secondary)
• fill-first – Fills a primary target's quota before moving to the next provider, ensuring maximum utilization of preferred accounts
• weighted – Randomly selects targets based on numeric weight fields for simple proportional load balancing

Load-balancing strategies:

• round-robin – Cycles through targets evenly to distribute traffic uniformly
• p2c (Power-of-Two-Choices) – Samples two random targets and selects the one with lower current load to prevent hotspots
• random – Uses uniform random selection for stateless, low-overhead routing
• least-used – Chooses the target with the fewest recent requests to keep all providers active

Quota and timing strategies:

• reset-aware – Avoids providers that have just reset their quotas to prevent post-reset exhaustion spikes
• reset-window – Prefers providers inside their reset windows to align traffic with upcoming quota refresh cycles
• headroom – Selects the provider with the most remaining token quota to guarantee full response generation

Cost and optimization strategies:

• cost-optimized – Sorts targets by per-token cost from src/lib/db/pricing.ts to minimize API spend
• strict-random – Random selection that respects hard limits such as max-tokens for compliance
• auto – Dynamically switches between least-used, cost-optimized, and headroom based on real-time metrics from usageTracking.ts
• lkgp (last-known-good-provider) – Sticks to the last provider that succeeded for this combo to improve cache hit rates
• context-optimized – Prefers providers capable of handling the current input context length to avoid truncation
• context-relay – Routes oversized requests across multiple providers via relay logic for hybrid processing
• fusion – Combines multiple providers in a single request for advanced co-generation scenarios

Configuring and Using Combo Routing

Step 1: Define a Combo with Strategy

Store the combo configuration via the API or admin interface. The strategy field accepts any value from ROUTING_STRATEGY_VALUES:

{
  "name": "smart-failover",
  "strategy": "auto",
  "targets": [
    { "provider": "openai", "model": "gpt-4o-mini", "weight": 5 },
    { "provider": "anthropic", "model": "claude-3-haiku-20240307", "weight": 3 },
    { "provider": "gemini", "model": "gemini-1.5-flash", "weight": 2 }
  ]
}

Step 2: Call the Combo via API

Send requests using the combo name as the model identifier. The entry point in src/app/api/v1/chat/completions/route.ts detects combo usage and dispatches to the router:

curl -X POST https://api.omniroute.online/v1/chat/completions \
  -H "Authorization: Bearer $OMNIRoute_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "smart-failover",
    "messages": [{"role": "user", "content": "Explain quantum computing."}]
  }'

Step 3: Override Strategy for Specific Requests

Override the default strategy temporarily using the strategy field:

curl -X POST https://api.omniroute.online/v1/chat/completions \
  -H "Authorization: Bearer $OMNIRoute_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "smart-failover",
    "strategy": "cost-optimized",
    "messages": [{"role": "user", "content": "Summarize this article."}]
  }'

Key Source Files and Implementation

The routing implementation spans these critical files:

• open-sse/services/combo.ts – Contains handleComboChat() and the iterative failover logic
• src/shared/constants/routingStrategies.ts – Defines the 17 strategy constants and validation logic
• src/lib/db/combo.ts – Implements resolveComboTargets() for database hydration
• open-sse/utils/usageTracking.ts – Provides live metrics for latency-aware strategies
• src/lib/db/pricing.ts – Supplies cost data for cost-optimized routing decisions
• src/app/api/v1/chat/completions/route.ts – Entry point that detects combo usage and forwards requests

On modern multi-core hardware, these strategies leverage concurrent metric updates from usageTracking.ts to maintain sub-millisecond routing overhead while processing thousands of requests per second.

Summary

• OmniRoute converts combo names into ordered target lists via resolveComboTargets() and executes them through handleComboChat() with automatic fallback
• The system provides 17 routing strategies including load-balancing algorithms (round-robin, p2c), cost optimizers (cost-optimized), and intelligent selectors (auto, headroom)
• Real-time metrics from usageTracking.ts and pricing data from pricing.ts drive decisions for latency-aware and cost-aware strategies
• Developers configure combos in the database via src/lib/db/combo.ts and invoke them using the combo name as the model parameter
• Individual requests can override the default strategy using the strategy field for temporary routing adjustments

Frequently Asked Questions

What is the difference between a combo and a provider in OmniRoute?

A provider represents a single LLM API endpoint (e.g., OpenAI GPT-4), while a combo is a named collection of providers managed in src/lib/db/combo.ts with an assigned routing strategy. Combos provide failover chains and load-balancing across multiple endpoints, whereas providers define individual connection parameters.

How does the 'auto' strategy decide which provider to use?

The auto strategy queries current metrics from open-sse/utils/usageTracking.ts including per-provider latency, token throughput, and error rates. It dynamically selects between least-used, cost-optimized, or headroom algorithms on a per-request basis based on which metric indicates the best performance-to-cost ratio at that moment.

Can I override the routing strategy for a single request?

Yes. Include the strategy field in your JSON payload when calling /api/v1/chat/completions. This temporarily overrides the combo's default strategy for that specific request without modifying the stored configuration in src/lib/db/combo.ts.

What happens if all providers in a combo fail?

If handleComboChat() exhausts all targets in the ResolvedComboTarget[] array without receiving a successful response, the function returns an aggregated error to the client. Each failure is logged to usageTracking.ts with provider-specific error codes, allowing operators to diagnose systemic issues versus isolated provider outages.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}