How Fallback Routing Works in OmniRoute Across Provider Tiers: 3‑Layer Resilience Strategy

OmniRoute guarantees high‑availability LLM calls through a three‑tier fallback routing system that isolates failures at the provider, connection, and model levels before triggering automatic provider switching.

OmniRoute is an open‑source LLM gateway that implements sophisticated fallback routing to maintain service continuity when upstream providers fail. Understanding how fallback routing works in OmniRoute across provider tiers helps operators configure resilient deployments and debug routing decisions. This article examines the complete fallback hierarchy implemented in the diegosouzapw/OmniRoute codebase.

The Three‑Tier Fallback Architecture

OmniRoute processes every request through three consecutive isolation layers. Each tier targets a progressively narrower scope of failure, maximizing the chance of success before invoking cross‑provider failover.

Tier 1: Provider‑Level Circuit Breaker

The broadest tier protects against catastrophic provider outages. When repeated upstream errors (HTTP 408, 500–504) exceed providerFailureThreshold, the entire provider enters an OPEN state and is excluded from combo routing until the cooldown expires.

This prevents wasted requests to clearly degraded infrastructure and is implemented in src/shared/utils/circuitBreaker.ts. All connections for that provider—regardless of individual credential health—are temporarily bypassed.

import { getProviderStatus, canExecute } from '@/shared/utils/circuitBreaker';

// After 12 simulated 502 responses for OpenAI
if (!canExecute('openai')) {
  console.log('OpenAI is temporarily blocked at provider level');
}

The circuit breaker uses lazy recovery: timestamps are checked on each read, automatically transitioning OPEN → HALF‑OPEN → CLOSED without background timers. This eliminates "thundering‑herd" retry storms while keeping the system responsive.

Tier 2: Connection‑Level Cool‑Down

When a specific credential fails but the provider remains healthy, tier 2 isolates the individual connection (API key or OAuth token). Transient errors like HTTP 429 (rate limit), 403 (permission denied), or provider‑specific quota failures trigger this layer.

The offending connection receives a rateLimitedUntil timestamp. Other connections for the same provider continue serving traffic. This logic is split across two files:

• open-sse/services/auth.ts::markAccountUnavailable() — marks connections unavailable
• open-sse/services/providerCooldownTracker.ts — sliding‑window limiter for recovery tracking

import { markAccountUnavailable } from 'open-sse/services/auth';

await markAccountUnavailable({
  provider: 'anthropic',
  connectionId: 'key-123',
  errorCode: 429,
  retryAfter: 30, // seconds
});

Tier 3: Model‑Lockout

The most granular tier handles model‑specific failures. When errors affect only a single model—such as missing model errors or model‑level quota exhaustion—that model is skipped while the connection remains usable for other models.

This prevents over‑punishment of healthy credentials and is implemented in open-sse/services/accountFallback.ts.

Cross‑Provider Failover: Combo Routing

If all three tiers fail to yield a successful route, auto‑combo routing activates. This system selects an alternative provider/model using a 14‑factor scoring system that weighs latency, cost, capability, and current health status.

The combo router respects the tier decisions above—excluded providers, cooled connections, and locked models are automatically filtered from consideration.

Specialized Fallback Layers

Certain request types trigger additional fallback logic beyond the three‑tier hierarchy.

Web‑Search Tool Fallback

Requests containing native web_search tools receive two‑layer fallback handling in open-sse/services/webSearchRouting.ts and open-sse/services/webSearchFallback.ts:

Layer	Action
1	Convert native web_search to internal omniroute_web_search function
2	Divert entire request to pre‑configured webSearchRouteModel

import { resolveWebSearchRouteOverride } from 'open-sse/services/webSearchRouting';

const result = resolveWebSearchRouteOverride(
  'openai/gpt-4o-mini',
  requestBody,
  { webSearchRouteModel: 'openai/web-search' }
);

if (result.wasRouted) {
  console.log('Request diverted to', result.model);
}

Quota‑Text Fallback

Providers that expose quota exhaustion in plain text responses—without structured error codes—are handled by open-sse/services/quotaTextCooldowns.ts. This pattern‑matching layer catches quota messages that would otherwise trigger unnecessary retries.

Proxy Fallback

When direct upstream fetch fails, open-sse/utils/proxyFallback.ts attempts proxy selection before falling back to native fetch. This provides resilience against network‑level connectivity issues rather than provider‑level failures.

Key Implementation Files

File	Responsibility
src/shared/utils/circuitBreaker.ts	Provider‑wide circuit breaker (tier 1)
open-sse/services/auth.ts (markAccountUnavailable)	Connection‑level unavailability marking (tier 2)
open-sse/services/providerCooldownTracker.ts	Sliding‑window limiter for connection recovery
open-sse/services/accountFallback.ts	Model‑lockout logic (tier 3)
open-sse/services/webSearchRouting.ts	Web‑search route override configuration
open-sse/services/webSearchFallback.ts	Native tool conversion fallback
open-sse/services/quotaTextCooldowns.ts	Text‑based quota detection
open-sse/utils/proxyFallback.ts	HTTP proxy selection on fetch failure
open-sse/services/combo.ts	Auto‑combo provider selection with 14‑factor scoring

Summary

• Three isolated tiers handle failures at provider, connection, and model granularity before cross‑provider failover
• Lazy recovery design uses timestamp checks rather than background timers, preventing retry storms
• Specialized layers address web‑search tools, text‑based quota errors, and proxy connectivity
• Auto‑combo routing provides final fallback through 14‑factor provider scoring
• All tiers respect operator‑configured policies and retry limits as implemented in diegosouzapw/OmniRoute

Frequently Asked Questions

How does OmniRoute prevent cascading failures when a major provider goes down?

The provider‑level circuit breaker in src/shared/utils/circuitBreaker.ts automatically excludes the entire provider from routing once providerFailureThreshold errors are detected. This happens before any request is dispatched, preventing wasted attempts and allowing immediate failover to alternate providers through combo routing.

What happens when only one API key is rate‑limited but others work fine?

Connection‑level cool‑down isolates the specific credential via markAccountUnavailable() in open-sse/services/auth.ts, setting a rateLimitedUntil timestamp. The provider remains active for routing, and other connections continue serving traffic. Recovery is automatic when the timestamp expires.

Can operators configure custom behaviors for web‑search requests?

Yes. Operators can set a webSearchRouteModel parameter that triggers resolveWebSearchRouteOverride() in open-sse/services/webSearchRouting.ts. This diverts requests containing native web‑search tools to a designated model, useful for providers with specialized search‑optimized endpoints.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}