# OmniRoute Combo Routing System Architecture: A Deep Dive Into Multi‑Provider LLM Resilience

> Explore OmniRoute's combo routing system architecture. Discover how this multi-stage pipeline achieves LLM resilience through provider distribution, circuit-breakers, and response validation.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: architecture
- Published: 2026-08-07

---

**OmniRoute's combo routing system is a modular, multi-stage pipeline that distributes requests across LLM providers using configurable strategies, applies circuit-breaker and cooldown-based resilience logic, and validates response quality before returning results.**

The combo routing layer sits at the heart of the [OmniRoute](https://github.com/diegosouzapw/OmniRoute) open-source gateway. It transforms a single API request into an orchestrated sequence of provider attempts—automatically handling fallbacks, retries, and quality checks. This article examines the complete architecture, from request entry through final response, based on the actual source code in release v3.8.50.

## The 10-Stage Combo Routing Pipeline

OmniRoute processes every combo request through a strictly ordered pipeline. Each stage is implemented in a dedicated module with clear responsibilities:

| Stage | Module | Core Function |
|-------|--------|-------------|
| 1 | [`open-sse/handlers/chatCore.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/handlers/chatCore.ts) | Entry point validation |
| 2 | [`combo/context.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/combo/context.ts) | Context assembly |
| 3 | [`combo/dispatchPrelude.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/combo/dispatchPrelude.ts) | Pre-dispatch shortcuts |
| 4 | [`combo/comboSetup.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/combo/comboSetup.ts) | Auto-candidate generation |
| 5 | [`combo/comboStructure.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/combo/comboStructure.ts) | Target resolution |
| 6 | [`combo/shadowRouting.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/combo/shadowRouting.ts) | Shadow routing |
| 7 | [`combo.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/combo.ts) | Main execution loop |
| 8 | [`combo/cooldownRetry.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/combo/cooldownRetry.ts) | Fallback orchestration |
| 9 | [`comboMetrics.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/comboMetrics.ts) | Observability |
| 10 | [`open-sse/utils/error.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/utils/error.ts) | Response sanitization |

## Request Entry and Context Creation

All combo requests enter through **`handleComboChat`** in [`open-sse/handlers/chatCore.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/handlers/chatCore.ts). This handler validates the request body, extracts the combo definition, and delegates to the combo engine.

The immediate next step is **context creation** in [`combo/context.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/combo/context.ts). The `createComboContext` function packages everything the pipeline needs into a single `ComboContext` object:

- Original request body
- Combo definition (name, strategy, configuration)
- Resilience settings
- Logger instance
- Provider connection caches

This context object is immutable and passes through every subsequent stage, ensuring consistent state without side effects.

## Pre-Dispatch Shortcuts: Pinned, Fusion, and Chaos Modes

Before the generic target loop runs, **`tryPinnedModelDispatch`** in [`combo/dispatchPrelude.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/combo/dispatchPrelude.ts) checks for three special execution modes:

- **Pinned-model dispatch** — Routes directly to a specific model when session affinity requires it
- **Fusion mode** — Executes multiple models in parallel and applies a judge to select or merge responses
- **Chaos mode** — Intentionally fans out to multiple providers simultaneously for redundancy or A/B testing

These shortcuts bypass the standard candidate scoring when the combo configuration explicitly demands deterministic or parallel behavior.

## Auto-Candidate Generation: The 13-Factor Scoring Model

When a combo uses the **`auto` strategy**, the system dynamically ranks available providers. The `buildAutoCandidates` function in [`combo/comboSetup.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/combo/comboSetup.ts) constructs `AutoProviderCandidate` objects by querying:

- Live provider connection caches
- Real-time quota information
- P95 latency statistics
- Cost-per-token pricing
- Circuit-breaker state
- Hidden-model filters

The actual scoring happens in [`autoStrategy.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/autoStrategy.ts), which implements a **13-factor model**:

```ts
// From autoStrategy.ts — simplified scoring weights
const score = 
  p95LatencyMs * 0.25 +
  errorRate * 0.20 +
  (1 - quotaRemaining / 100) * 0.15 +
  costPer1MTokens * 0.15 +
  resetWindowAffinity * 0.10 +
  successRate * 0.10 +
  circuitBreakerHealth * 0.05;

```

Higher scores indicate better candidates. The top-scoring providers become the ordered target list.

## Target Resolution: From Definitions to Concrete Connections

The `resolveComboTargets` function in [`combo/comboStructure.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/combo/comboStructure.ts) transforms abstract combo definitions into executable `ResolvedComboTarget[]` objects. This stage handles:

- Provider wildcard expansion (e.g., `"provider": "*"` matches all configured providers)
- Model name resolution to actual API endpoints
- Connection ID assignment
- Step ID generation for pipeline tracking
- Execution key derivation for telemetry

The output is a fully materialized execution plan with no remaining symbolic references.

## The Main Target Loop: Resilience in Action

The **`executeTarget`** function in [`combo.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/combo.ts) iterates over resolved targets, applying a comprehensive resilience stack at each attempt:

```ts
// Pseudocode representing the executeTarget checks
async function executeTarget(context, target) {
  // 1. Circuit breaker check
  if (isProviderCircuitOpen(target.provider)) return skip;
  
  // 2. Connection cooldown check
  if (isProviderInCooldown(target.connectionId)) return skip;
  
  // 3. Model lockout check
  if (isModelLocked(target.model)) return skip;
  
  // 4. Quota exhaustion check
  if (isQuotaExhausted(target)) return skip;
  
  // 5. Credential validation
  if (!credentialGate(target)) return skip;
  
  // 6. Concurrency semaphore acquire
  await rateLimitSemaphore.acquire(target);
  
  // 7. Predictive TTFT zero-latency path
  if (predictiveTtftAvailable(target)) {
    return await predictiveExecute(target);
  }
  
  // 8. Standard execution with reasoning-token buffering
  const response = await executeWithReasoningBuffer(target);
  
  // 9. Proactive compression for fallback readiness
  await prepareCompressionFallback(response);
  
  // 10. Quality validation
  if (!validateResponseQuality(response)) {
    releasePin(target);
    return failure; // Triggers next target
  }
  
  return response;
}

```

Each check consults a dedicated subsystem:

| Check | Source File | Purpose |
|-------|-------------|---------|
| Circuit breaker | [`circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/circuitBreaker.ts) | Halt traffic to failing providers |
| Connection cooldown | [`providerCooldownTracker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/providerCooldownTracker.ts) | Respect 429 retry-after headers |
| Model lockout | [`quotaExhaustionCutoff.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/quotaExhaustionCutoff.ts) | Disable specific models per connection |
| Credential gate | [`credentialGate.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/credentialGate.ts) | Validate API key permissions |
| Rate limit semaphore | [`rateLimitSemaphore.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/rateLimitSemaphore.ts) | Enforce global concurrency limits |
| Compression strategy | [`compression/strategySelector.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/compression/strategySelector.ts) | Prepare fallback-compressed requests |

## Fallback, Retry, and Recovery Orchestration

When a target fails, the combo routing system consults **`resolveComboCooldownWaitDecision`** in [`combo/cooldownRetry.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/combo/cooldownRetry.ts). This function implements three recovery paths:

1. **Immediate retry** — Re-attempt the same target (up to `maxRetries`)
2. **Cooldown-aware wait** — Pause execution based on `Retry-After` headers, then retry the full target set
3. **Diagnostic error return** — Exhaustion path with structured recovery hints

The [`pinRecovery.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/pinRecovery.ts) module generates recovery hints for sticky sessions. When a pinned model fails, it suggests alternatives like:

> "Increase `max_tokens` — reasoning model exhausted token budget"

or

> "Provider cooldown expires in 47s; consider `fallbackDelayMs: 5000`"

## Quality Guardrails Beyond HTTP Status Codes

OmniRoute treats **HTTP 200 ≠ success**. The `validateResponseQuality` function in [`combo/validateQuality.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/combo/validateQuality.ts) inspects every successful response for:

- Empty content
- Disallowed token patterns
- Schema violations (when `responseValidation` is enabled)
- Streaming truncation indicators

Failed quality checks trigger the same fallback logic as provider errors, with the additional step of **pin release**—allowing subsequent requests to route away from the degraded model.

## Shadow Routing and Observability

The **`scheduleShadowRouting`** function in [`combo/shadowRouting.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/combo/shadowRouting.ts) optionally dispatches duplicate requests to secondary targets. These shadow executions:

- Do not block the primary response
- Record latency and success metrics
- Enable provider performance comparison without user impact

Primary metrics collection happens in [`comboMetrics.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/comboMetrics.ts), which aggregates per-target data including:

- Attempt order and outcome
- Excluded providers and exclusion reasons
- Fallback counts and recovery hints

All errors flow through **`errorResponseWithComboDiagnostics`** in [`open-sse/utils/error.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/utils/error.ts), ensuring no stack traces leak while providing structured debugging information.

## Configuring a Combo: API and Code Examples

### Dashboard/JSON Definition

```json
{
  "name": "quick-search-combo",
  "strategy": "auto",
  "models": [
    { "provider": "openai", "model": "gpt-4o-mini" },
    { "provider": "anthropic", "model": "claude-sonnet-4.6" },
    { "provider": "google", "model": "gemini-2.5-flash" }
  ],
  "config": {
    "maxRetries": 2,
    "fallbackDelayMs": 0,
    "predictiveTtftMs": 300,
    "responseValidation": true
  }
}

```

### Client Invocation

```bash
curl -X POST http://localhost:20128/v1/chat/completions \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "quick-search-combo",
    "messages": [{"role": "user", "content": "Explain quantum tunneling."}],
    "max_tokens": 500
  }'

```

### Custom Strategy Extension

```ts
// open-sse/services/combo/autoStrategy.ts
export const scoreAutoTargets = (candidates: AutoProviderCandidate[]) => {
  return candidates.map(c => ({
    ...c,
    score: 
      c.p95LatencyMs * 0.35 +
      c.errorRate * 0.25 +
      (1 - c.quotaRemaining / 100) * 0.20 +
      c.costPer1MTokens * 0.15 +
      (c.usageCount ?? 0) * -0.05  // Prefer less-used providers
  }));
};

```

Recompile to activate the new factor across all `auto` strategy combos.

## Summary

- **OmniRoute combo routing** implements a 10-stage pipeline for resilient multi-provider LLM requests
- **Auto-candidate generation** uses 13-factor scoring including latency, cost, quota, and circuit-breaker state
- **Three resilience layers**—circuit breaker, connection cooldown, and model lockout—operate inside the main execution loop
- **Quality validation** rejects HTTP 200 responses that fail content guardrails
- **Shadow routing** enables passive observability without latency impact
- **19 built-in strategies** include priority, weighted, round-robin, cost-optimized, and context-relay modes

## Frequently Asked Questions

### What makes OmniRoute's combo routing different from simple load balancing?

Simple load balancers distribute requests based on health checks and round-robin policies. OmniRoute's combo routing adds **provider-aware scoring** (13 factors), **proactive resilience** (circuit breakers, cooldowns, model lockouts), and **semantic quality validation** that treats malformed 200 responses as failures. The architecture specifically handles LLM-specific failure modes like quota exhaustion and reasoning-token limits.

### How does the auto strategy select which provider to use?

The **auto strategy** in [`autoStrategy.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/autoStrategy.ts) calculates a composite score for every routable provider connection. It weighs P95 latency (25%), recent error rate (20%), remaining quota percentage (15%), cost per million tokens (15%), reset-window affinity (10%), historical success rate (10%), and circuit-breaker health (5%). The highest-scoring candidates become the ordered target list.

### Can I force a specific model for certain request types?

Yes. Use **pinned-model dispatch** by including a `pinnedModel` field in your combo configuration or session context. The `tryPinnedModelDispatch` function in [`dispatchPrelude.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/dispatchPrelude.ts) routes directly to the specified model before any scoring occurs. If that model fails, standard fallback logic applies unless strict pinning is enabled.

### What recovery options exist when all providers in a combo fail?

When targets exhaust, [`pinRecovery.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/pinRecovery.ts) generates structured **recovery hints**. These may suggest increasing `max_tokens`, waiting for cooldown expiration, switching to a different combo, or adjusting `fallbackDelayMs`. The `errorResponseWithComboDiagnostics` function returns these hints alongside attempt history and excluded provider reasons, enabling client-side retry decisions.