# How OmniRoute Routes Traffic Across Multiple LLM Providers: Architecture & Implementation

> Discover how OmniRoute routes traffic across multiple LLM providers using a combo engine. Learn its architecture, implementation, and scoring algorithm for optimal performance.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: architecture
- Published: 2026-08-28

---

**OmniRoute routes traffic across multiple LLM providers through a combo engine that filters active connections by health and cost, scores candidates using a 15-factor algorithm, and dispatches requests via provider-specific executors.**

OmniRoute is an open-source LLM gateway that dynamically distributes API requests across dozens of providers. Understanding how OmniRoute routes traffic across multiple LLM providers requires examining its combo engine architecture, virtual auto-combo factory, and resilience layers implemented in TypeScript.

## Request Entry Point and Model Detection

All routing begins at the chat completions endpoint. When OmniRoute receives a `POST` request to `/v1/chat/completions`, the **chat handler** ([`src/sse/handlers/chat.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/handlers/chat.ts)) extracts the `model` field from the request body to determine the routing path.

The handler categorizes every request into one of three types:

- **Static model** – Direct mapping to a specific provider model
- **Persisted combo** – A named configuration stored in the system
- **Zero-config auto-combo** – Dynamic routing when the model name starts with `auto/`

If the model field contains the `auto/` prefix, the handler short-circuits to the virtual auto-combo factory. Otherwise, the request flows into the standard combo engine with either a static target or a persisted combo definition.

## The Combo Engine: Core Routing Logic

All routing ultimately passes through the **combo engine** located in [`open-sse/services/combo.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/combo.ts). The core entry point is the `handleComboChat` function, which receives three critical parameters: the request body, the combo definition (or a virtual auto-combo), and a `handleSingleModel` callback that knows how to invoke a provider’s HTTP API.

The engine iterates over the list of **targets** and applies the selected routing strategy. Each target represents a candidate provider-model pair. The `executeTarget` function inside [`combo.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/combo.ts) evaluates runtime guards before dispatching, ensuring failed candidates trigger immediate fallback to the next target in the pool.

## Zero-Config Auto-Combo Routing

When handling requests with the `auto/` prefix, OmniRoute bypasses static configuration and builds an in-memory combo on-the-fly. The **virtual auto-combo factory** ([`open-sse/services/autoCombo/virtualFactory.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/autoCombo/virtualFactory.ts)) constructs this pool dynamically:

1. Fetches all active provider connections via `getProviderConnections({ isActive: true })`
2. Discards connections with expired OAuth tokens or missing API keys
3. Looks up each provider’s model catalog and pricing metadata
4. Creates a `VirtualAutoComboCandidate` for every `(provider, model, connection)` tuple
5. Scores each candidate using the 15-factor algorithm in [`open-sse/services/autoCombo/scoring.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/autoCombo/scoring.ts)
6. Returns an `AutoComboConfig` consumed by the regular combo engine

The 15 scoring factors include quota head-room, circuit-breaker health, cost per token, latency percentiles, task-fit alignment, tier priority, and context affinity. This ensures the routing decision weighs both economic efficiency and performance characteristics.

## Routing Strategies

OmniRoute supports **19 distinct routing strategies** defined in [`src/shared/constants/routingStrategies.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/constants/routingStrategies.ts). The combo engine selects the strategy from the combo’s `strategy` field or, for auto-combos, from default configuration.

Available strategies include:

- **Priority** – Failover cascade based on ordered preference
- **Weighted** – Traffic distribution by percentage weights
- **Round-robin** – Even rotation across healthy providers
- **LKGP** – Latency-keeper greedy placement for performance
- **Fusion** – Parallel dispatch to multiple providers with response aggregation
- **Rules** – Default 15-factor scoring for auto-combos

Override the strategy for a single request using the `X-OmniRoute-Mode` header:

```bash
curl -X POST http://localhost:20128/v1/chat/completions \
  -H "Authorization: Bearer <api-key>" \
  -H "X-OmniRoute-Mode: quality" \
  -H "Content-Type: application/json" \
  -d '{"model":"auto","messages":[{"role":"user","content":"Write a recursive function"}]}'

```

## Resilience Layers

Before dispatching to any candidate, the combo engine evaluates three runtime guards in `executeTarget`:

**Provider-Level Circuit Breaker** – If a provider’s breaker state is **OPEN**, the candidate is skipped. The breaker implementation lives in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts), tracking error rates and failure thresholds per provider.

**Connection Cooldown** – Each connection maintains a `rateLimitedUntil` timestamp. The `isProviderInCooldown` function in [`open-sse/services/providerCooldownTracker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/providerCooldownTracker.ts) prevents traffic to throttled connections.

**Model Lockout** – When a specific model is over quota on a connection, `isModelLocked` in [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts) blocks that model while preserving the connection for other models.

If any guard fails, the engine records the reason (e.g., `circuit_open`, `provider_cooldown`) and proceeds to the next target.

## Provider Dispatch

Once the engine selects a candidate, it invokes the **provider executor** (`open-sse/executors/*`). Each executor constructs provider-specific HTTP requests, handles authentication, manages streaming responses, and normalizes output formats.

For the `fusion` strategy, the engine dispatches to multiple executors simultaneously and aggregates panel responses before returning to the client. Standard strategies use a single executor call per request.

Zero-config routing example:

```bash
curl -X POST http://localhost:20128/v1/chat/completions \
  -H "Authorization: Bearer <api-key>" \
  -H "Content-Type: application/json" \
  -d '{"model":"auto/fast","messages":[{"role":"user","content":"Explain quantum entanglement"}]}'

```

## Summary

- OmniRoute detects request types in [`src/sse/handlers/chat.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/handlers/chat.ts), distinguishing between static models and auto-combos via the `auto/` prefix.
- The combo engine ([`open-sse/services/combo.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/combo.ts)) processes all routing through `handleComboChat`, iterating over candidate targets.
- Auto-combos build candidate pools dynamically in [`open-sse/services/autoCombo/virtualFactory.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/autoCombo/virtualFactory.ts) using the 15-factor scoring system.
- Nineteen routing strategies in [`src/shared/constants/routingStrategies.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/constants/routingStrategies.ts) control selection logic, overridable via the `X-OmniRoute-Mode` header.
- Circuit breakers, cooldown trackers, and model lockouts ensure resilient failover across provider connections.
- Provider-specific executors in `open-sse/executors/*` handle final HTTP dispatch and response streaming.

## Frequently Asked Questions

### How does OmniRoute handle provider failures during routing?

OmniRoute isolates failures through provider-level circuit breakers implemented in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts). When error rates exceed thresholds, the breaker opens and the combo engine skips that provider. Additionally, the `isProviderInCooldown` check in [`open-sse/services/providerCooldownTracker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/providerCooldownTracker.ts) prevents sending traffic to rate-limited connections, ensuring automatic failover to healthy alternatives.

### What is the difference between a persisted combo and an auto-combo?

A **persisted combo** is a named configuration stored in the system that defines specific provider targets, weights, and strategies. An **auto-combo** (triggered by the `auto/` prefix) is generated dynamically for each request by [`open-sse/services/autoCombo/virtualFactory.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/autoCombo/virtualFactory.ts), which scans all active connections and scores them using the 15-factor algorithm without requiring manual configuration.

### Can I force OmniRoute to use a specific routing strategy for one request?

Yes. Send the `X-OmniRoute-Mode` header with values like `quality`, `cost`, `latency`, or `lkgp` to override the default strategy for that single request. The combo engine checks this header in [`open-sse/services/combo.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/combo.ts) before applying the persisted or auto-combo configuration.

### How does the 15-factor scoring work in auto-combo routing?

The scoring function in [`open-sse/services/autoCombo/scoring.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/autoCombo/scoring.ts) evaluates every candidate provider-model pair across 15 dimensions including cost per token, latency percentiles, quota availability, circuit-breaker health, and task-fit affinity. Each factor receives a weighted score, and the engine selects the highest-ranked candidate that passes the resilience guards (circuit breaker, cooldown, and lockout checks).