OmniRoute Combo Routing System Architecture: A Deep Dive Into Multi‑Provider LLM Resilience

OmniRoute's combo routing system is a modular, multi-stage pipeline that distributes requests across LLM providers using configurable strategies, applies circuit-breaker and cooldown-based resilience logic, and validates response quality before returning results.

The combo routing layer sits at the heart of the OmniRoute open-source gateway. It transforms a single API request into an orchestrated sequence of provider attempts—automatically handling fallbacks, retries, and quality checks. This article examines the complete architecture, from request entry through final response, based on the actual source code in release v3.8.50.

The 10-Stage Combo Routing Pipeline

OmniRoute processes every combo request through a strictly ordered pipeline. Each stage is implemented in a dedicated module with clear responsibilities:

Stage	Module	Core Function
1	open-sse/handlers/chatCore.ts	Entry point validation
2	combo/context.ts	Context assembly
3	combo/dispatchPrelude.ts	Pre-dispatch shortcuts
4	combo/comboSetup.ts	Auto-candidate generation
5	combo/comboStructure.ts	Target resolution
6	combo/shadowRouting.ts	Shadow routing
7	combo.ts	Main execution loop
8	combo/cooldownRetry.ts	Fallback orchestration
9	comboMetrics.ts	Observability
10	open-sse/utils/error.ts	Response sanitization

Request Entry and Context Creation

All combo requests enter through handleComboChat in open-sse/handlers/chatCore.ts. This handler validates the request body, extracts the combo definition, and delegates to the combo engine.

The immediate next step is context creation in combo/context.ts. The createComboContext function packages everything the pipeline needs into a single ComboContext object:

• Original request body
• Combo definition (name, strategy, configuration)
• Resilience settings
• Logger instance
• Provider connection caches

This context object is immutable and passes through every subsequent stage, ensuring consistent state without side effects.

Pre-Dispatch Shortcuts: Pinned, Fusion, and Chaos Modes

Before the generic target loop runs, tryPinnedModelDispatch in combo/dispatchPrelude.ts checks for three special execution modes:

• Pinned-model dispatch — Routes directly to a specific model when session affinity requires it
• Fusion mode — Executes multiple models in parallel and applies a judge to select or merge responses
• Chaos mode — Intentionally fans out to multiple providers simultaneously for redundancy or A/B testing

These shortcuts bypass the standard candidate scoring when the combo configuration explicitly demands deterministic or parallel behavior.

Auto-Candidate Generation: The 13-Factor Scoring Model

When a combo uses the auto strategy, the system dynamically ranks available providers. The buildAutoCandidates function in combo/comboSetup.ts constructs AutoProviderCandidate objects by querying:

• Live provider connection caches
• Real-time quota information
• P95 latency statistics
• Cost-per-token pricing
• Circuit-breaker state
• Hidden-model filters

The actual scoring happens in autoStrategy.ts, which implements a 13-factor model:

// From autoStrategy.ts — simplified scoring weights
const score = 
  p95LatencyMs * 0.25 +
  errorRate * 0.20 +
  (1 - quotaRemaining / 100) * 0.15 +
  costPer1MTokens * 0.15 +
  resetWindowAffinity * 0.10 +
  successRate * 0.10 +
  circuitBreakerHealth * 0.05;

Higher scores indicate better candidates. The top-scoring providers become the ordered target list.

Target Resolution: From Definitions to Concrete Connections

The resolveComboTargets function in combo/comboStructure.ts transforms abstract combo definitions into executable ResolvedComboTarget[] objects. This stage handles:

• Provider wildcard expansion (e.g., "provider": "*" matches all configured providers)
• Model name resolution to actual API endpoints
• Connection ID assignment
• Step ID generation for pipeline tracking
• Execution key derivation for telemetry

The output is a fully materialized execution plan with no remaining symbolic references.

The Main Target Loop: Resilience in Action

The executeTarget function in combo.ts iterates over resolved targets, applying a comprehensive resilience stack at each attempt:

// Pseudocode representing the executeTarget checks
async function executeTarget(context, target) {
  // 1. Circuit breaker check
  if (isProviderCircuitOpen(target.provider)) return skip;
  
  // 2. Connection cooldown check
  if (isProviderInCooldown(target.connectionId)) return skip;
  
  // 3. Model lockout check
  if (isModelLocked(target.model)) return skip;
  
  // 4. Quota exhaustion check
  if (isQuotaExhausted(target)) return skip;
  
  // 5. Credential validation
  if (!credentialGate(target)) return skip;
  
  // 6. Concurrency semaphore acquire
  await rateLimitSemaphore.acquire(target);
  
  // 7. Predictive TTFT zero-latency path
  if (predictiveTtftAvailable(target)) {
    return await predictiveExecute(target);
  }
  
  // 8. Standard execution with reasoning-token buffering
  const response = await executeWithReasoningBuffer(target);
  
  // 9. Proactive compression for fallback readiness
  await prepareCompressionFallback(response);
  
  // 10. Quality validation
  if (!validateResponseQuality(response)) {
    releasePin(target);
    return failure; // Triggers next target
  }
  
  return response;
}

Each check consults a dedicated subsystem:

Check	Source File	Purpose
Circuit breaker	circuitBreaker.ts	Halt traffic to failing providers
Connection cooldown	providerCooldownTracker.ts	Respect 429 retry-after headers
Model lockout	quotaExhaustionCutoff.ts	Disable specific models per connection
Credential gate	credentialGate.ts	Validate API key permissions
Rate limit semaphore	rateLimitSemaphore.ts	Enforce global concurrency limits
Compression strategy	compression/strategySelector.ts	Prepare fallback-compressed requests

Fallback, Retry, and Recovery Orchestration

When a target fails, the combo routing system consults resolveComboCooldownWaitDecision in combo/cooldownRetry.ts. This function implements three recovery paths:

1. Immediate retry — Re-attempt the same target (up to maxRetries)
2. Cooldown-aware wait — Pause execution based on Retry-After headers, then retry the full target set
3. Diagnostic error return — Exhaustion path with structured recovery hints

The pinRecovery.ts module generates recovery hints for sticky sessions. When a pinned model fails, it suggests alternatives like:

"Increase max_tokens — reasoning model exhausted token budget"

or

"Provider cooldown expires in 47s; consider fallbackDelayMs: 5000"

Quality Guardrails Beyond HTTP Status Codes

OmniRoute treats HTTP 200 ≠ success. The validateResponseQuality function in combo/validateQuality.ts inspects every successful response for:

• Empty content
• Disallowed token patterns
• Schema violations (when responseValidation is enabled)
• Streaming truncation indicators

Failed quality checks trigger the same fallback logic as provider errors, with the additional step of pin release—allowing subsequent requests to route away from the degraded model.

Shadow Routing and Observability

The scheduleShadowRouting function in combo/shadowRouting.ts optionally dispatches duplicate requests to secondary targets. These shadow executions:

• Do not block the primary response
• Record latency and success metrics
• Enable provider performance comparison without user impact

Primary metrics collection happens in comboMetrics.ts, which aggregates per-target data including:

• Attempt order and outcome
• Excluded providers and exclusion reasons
• Fallback counts and recovery hints

All errors flow through errorResponseWithComboDiagnostics in open-sse/utils/error.ts, ensuring no stack traces leak while providing structured debugging information.

Configuring a Combo: API and Code Examples

Dashboard/JSON Definition

{
  "name": "quick-search-combo",
  "strategy": "auto",
  "models": [
    { "provider": "openai", "model": "gpt-4o-mini" },
    { "provider": "anthropic", "model": "claude-sonnet-4.6" },
    { "provider": "google", "model": "gemini-2.5-flash" }
  ],
  "config": {
    "maxRetries": 2,
    "fallbackDelayMs": 0,
    "predictiveTtftMs": 300,
    "responseValidation": true
  }
}

Client Invocation

curl -X POST http://localhost:20128/v1/chat/completions \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "quick-search-combo",
    "messages": [{"role": "user", "content": "Explain quantum tunneling."}],
    "max_tokens": 500
  }'

Custom Strategy Extension

// open-sse/services/combo/autoStrategy.ts
export const scoreAutoTargets = (candidates: AutoProviderCandidate[]) => {
  return candidates.map(c => ({
    ...c,
    score: 
      c.p95LatencyMs * 0.35 +
      c.errorRate * 0.25 +
      (1 - c.quotaRemaining / 100) * 0.20 +
      c.costPer1MTokens * 0.15 +
      (c.usageCount ?? 0) * -0.05  // Prefer less-used providers
  }));
};

Recompile to activate the new factor across all auto strategy combos.

Summary

• OmniRoute combo routing implements a 10-stage pipeline for resilient multi-provider LLM requests
• Auto-candidate generation uses 13-factor scoring including latency, cost, quota, and circuit-breaker state
• Three resilience layers—circuit breaker, connection cooldown, and model lockout—operate inside the main execution loop
• Quality validation rejects HTTP 200 responses that fail content guardrails
• Shadow routing enables passive observability without latency impact
• 19 built-in strategies include priority, weighted, round-robin, cost-optimized, and context-relay modes

Frequently Asked Questions

What makes OmniRoute's combo routing different from simple load balancing?

Simple load balancers distribute requests based on health checks and round-robin policies. OmniRoute's combo routing adds provider-aware scoring (13 factors), proactive resilience (circuit breakers, cooldowns, model lockouts), and semantic quality validation that treats malformed 200 responses as failures. The architecture specifically handles LLM-specific failure modes like quota exhaustion and reasoning-token limits.

How does the auto strategy select which provider to use?

The auto strategy in autoStrategy.ts calculates a composite score for every routable provider connection. It weighs P95 latency (25%), recent error rate (20%), remaining quota percentage (15%), cost per million tokens (15%), reset-window affinity (10%), historical success rate (10%), and circuit-breaker health (5%). The highest-scoring candidates become the ordered target list.

Can I force a specific model for certain request types?

Yes. Use pinned-model dispatch by including a pinnedModel field in your combo configuration or session context. The tryPinnedModelDispatch function in dispatchPrelude.ts routes directly to the specified model before any scoring occurs. If that model fails, standard fallback logic applies unless strict pinning is enabled.

What recovery options exist when all providers in a combo fail?

When targets exhaust, pinRecovery.ts generates structured recovery hints. These may suggest increasing max_tokens, waiting for cooldown expiration, switching to a different combo, or adjusting fallbackDelayMs. The errorResponseWithComboDiagnostics function returns these hints alongside attempt history and excluded provider reasons, enabling client-side retry decisions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}