How OmniRoute Routes Traffic Across Multiple LLM Providers: Architecture & Implementation

OmniRoute routes traffic across multiple LLM providers through a combo engine that filters active connections by health and cost, scores candidates using a 15-factor algorithm, and dispatches requests via provider-specific executors.

OmniRoute is an open-source LLM gateway that dynamically distributes API requests across dozens of providers. Understanding how OmniRoute routes traffic across multiple LLM providers requires examining its combo engine architecture, virtual auto-combo factory, and resilience layers implemented in TypeScript.

Request Entry Point and Model Detection

All routing begins at the chat completions endpoint. When OmniRoute receives a POST request to /v1/chat/completions, the chat handler (src/sse/handlers/chat.ts) extracts the model field from the request body to determine the routing path.

The handler categorizes every request into one of three types:

• Static model – Direct mapping to a specific provider model
• Persisted combo – A named configuration stored in the system
• Zero-config auto-combo – Dynamic routing when the model name starts with auto/

If the model field contains the auto/ prefix, the handler short-circuits to the virtual auto-combo factory. Otherwise, the request flows into the standard combo engine with either a static target or a persisted combo definition.

The Combo Engine: Core Routing Logic

All routing ultimately passes through the combo engine located in open-sse/services/combo.ts. The core entry point is the handleComboChat function, which receives three critical parameters: the request body, the combo definition (or a virtual auto-combo), and a handleSingleModel callback that knows how to invoke a provider’s HTTP API.

The engine iterates over the list of targets and applies the selected routing strategy. Each target represents a candidate provider-model pair. The executeTarget function inside combo.ts evaluates runtime guards before dispatching, ensuring failed candidates trigger immediate fallback to the next target in the pool.

Zero-Config Auto-Combo Routing

When handling requests with the auto/ prefix, OmniRoute bypasses static configuration and builds an in-memory combo on-the-fly. The virtual auto-combo factory (open-sse/services/autoCombo/virtualFactory.ts) constructs this pool dynamically:

1. Fetches all active provider connections via getProviderConnections({ isActive: true })
2. Discards connections with expired OAuth tokens or missing API keys
3. Looks up each provider’s model catalog and pricing metadata
4. Creates a VirtualAutoComboCandidate for every (provider, model, connection) tuple
5. Scores each candidate using the 15-factor algorithm in open-sse/services/autoCombo/scoring.ts
6. Returns an AutoComboConfig consumed by the regular combo engine

The 15 scoring factors include quota head-room, circuit-breaker health, cost per token, latency percentiles, task-fit alignment, tier priority, and context affinity. This ensures the routing decision weighs both economic efficiency and performance characteristics.

Routing Strategies

OmniRoute supports 19 distinct routing strategies defined in src/shared/constants/routingStrategies.ts. The combo engine selects the strategy from the combo’s strategy field or, for auto-combos, from default configuration.

Available strategies include:

• Priority – Failover cascade based on ordered preference
• Weighted – Traffic distribution by percentage weights
• Round-robin – Even rotation across healthy providers
• LKGP – Latency-keeper greedy placement for performance
• Fusion – Parallel dispatch to multiple providers with response aggregation
• Rules – Default 15-factor scoring for auto-combos

Override the strategy for a single request using the X-OmniRoute-Mode header:

curl -X POST http://localhost:20128/v1/chat/completions \
  -H "Authorization: Bearer <api-key>" \
  -H "X-OmniRoute-Mode: quality" \
  -H "Content-Type: application/json" \
  -d '{"model":"auto","messages":[{"role":"user","content":"Write a recursive function"}]}'

Resilience Layers

Before dispatching to any candidate, the combo engine evaluates three runtime guards in executeTarget:

Provider-Level Circuit Breaker – If a provider’s breaker state is OPEN, the candidate is skipped. The breaker implementation lives in src/shared/utils/circuitBreaker.ts, tracking error rates and failure thresholds per provider.

Connection Cooldown – Each connection maintains a rateLimitedUntil timestamp. The isProviderInCooldown function in open-sse/services/providerCooldownTracker.ts prevents traffic to throttled connections.

Model Lockout – When a specific model is over quota on a connection, isModelLocked in open-sse/services/accountFallback.ts blocks that model while preserving the connection for other models.

If any guard fails, the engine records the reason (e.g., circuit_open, provider_cooldown) and proceeds to the next target.

Provider Dispatch

Once the engine selects a candidate, it invokes the provider executor (open-sse/executors/*). Each executor constructs provider-specific HTTP requests, handles authentication, manages streaming responses, and normalizes output formats.

For the fusion strategy, the engine dispatches to multiple executors simultaneously and aggregates panel responses before returning to the client. Standard strategies use a single executor call per request.

Zero-config routing example:

curl -X POST http://localhost:20128/v1/chat/completions \
  -H "Authorization: Bearer <api-key>" \
  -H "Content-Type: application/json" \
  -d '{"model":"auto/fast","messages":[{"role":"user","content":"Explain quantum entanglement"}]}'

Summary

• OmniRoute detects request types in src/sse/handlers/chat.ts, distinguishing between static models and auto-combos via the auto/ prefix.
• The combo engine (open-sse/services/combo.ts) processes all routing through handleComboChat, iterating over candidate targets.
• Auto-combos build candidate pools dynamically in open-sse/services/autoCombo/virtualFactory.ts using the 15-factor scoring system.
• Nineteen routing strategies in src/shared/constants/routingStrategies.ts control selection logic, overridable via the X-OmniRoute-Mode header.
• Circuit breakers, cooldown trackers, and model lockouts ensure resilient failover across provider connections.
• Provider-specific executors in open-sse/executors/* handle final HTTP dispatch and response streaming.

Frequently Asked Questions

How does OmniRoute handle provider failures during routing?

OmniRoute isolates failures through provider-level circuit breakers implemented in src/shared/utils/circuitBreaker.ts. When error rates exceed thresholds, the breaker opens and the combo engine skips that provider. Additionally, the isProviderInCooldown check in open-sse/services/providerCooldownTracker.ts prevents sending traffic to rate-limited connections, ensuring automatic failover to healthy alternatives.

What is the difference between a persisted combo and an auto-combo?

A persisted combo is a named configuration stored in the system that defines specific provider targets, weights, and strategies. An auto-combo (triggered by the auto/ prefix) is generated dynamically for each request by open-sse/services/autoCombo/virtualFactory.ts, which scans all active connections and scores them using the 15-factor algorithm without requiring manual configuration.

Can I force OmniRoute to use a specific routing strategy for one request?

Yes. Send the X-OmniRoute-Mode header with values like quality, cost, latency, or lkgp to override the default strategy for that single request. The combo engine checks this header in open-sse/services/combo.ts before applying the persisted or auto-combo configuration.

How does the 15-factor scoring work in auto-combo routing?

The scoring function in open-sse/services/autoCombo/scoring.ts evaluates every candidate provider-model pair across 15 dimensions including cost per token, latency percentiles, quota availability, circuit-breaker health, and task-fit affinity. Each factor receives a weighted score, and the engine selects the highest-ranked candidate that passes the resilience guards (circuit breaker, cooldown, and lockout checks).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}