# How Fallback Routing Works in OmniRoute Across Provider Tiers: 3‑Layer Resilience Strategy

> Discover OmniRoute's 3-layer fallback routing strategy for guaranteed high-availability LLM calls. Isolate provider connection and model failures with automatic switching.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: how-to-guide
- Published: 2026-08-15

---

**OmniRoute guarantees high‑availability LLM calls through a three‑tier fallback routing system that isolates failures at the provider, connection, and model levels before triggering automatic provider switching.**

OmniRoute is an open‑source LLM gateway that implements sophisticated fallback routing to maintain service continuity when upstream providers fail. Understanding how fallback routing works in OmniRoute across provider tiers helps operators configure resilient deployments and debug routing decisions. This article examines the complete fallback hierarchy implemented in the diegosouzapw/OmniRoute codebase.

## The Three‑Tier Fallback Architecture

OmniRoute processes every request through three consecutive isolation layers. Each tier targets a progressively narrower scope of failure, maximizing the chance of success before invoking cross‑provider failover.

### Tier 1: Provider‑Level Circuit Breaker

The broadest tier protects against **catastrophic provider outages**. When repeated upstream errors (HTTP 408, 500–504) exceed `providerFailureThreshold`, the entire provider enters an **OPEN** state and is excluded from combo routing until the cooldown expires.

This prevents wasted requests to clearly degraded infrastructure and is implemented in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts). All connections for that provider—regardless of individual credential health—are temporarily bypassed.

```typescript
import { getProviderStatus, canExecute } from '@/shared/utils/circuitBreaker';

// After 12 simulated 502 responses for OpenAI
if (!canExecute('openai')) {
  console.log('OpenAI is temporarily blocked at provider level');
}

```

The circuit breaker uses **lazy recovery**: timestamps are checked on each read, automatically transitioning **OPEN → HALF‑OPEN → CLOSED** without background timers. This eliminates "thundering‑herd" retry storms while keeping the system responsive.

### Tier 2: Connection‑Level Cool‑Down

When a specific credential fails but the provider remains healthy, tier 2 isolates the **individual connection** (API key or OAuth token). Transient errors like HTTP 429 (rate limit), 403 (permission denied), or provider‑specific quota failures trigger this layer.

The offending connection receives a `rateLimitedUntil` timestamp. Other connections for the same provider continue serving traffic. This logic is split across two files:

- `open-sse/services/auth.ts::markAccountUnavailable()` — marks connections unavailable
- [`open-sse/services/providerCooldownTracker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/providerCooldownTracker.ts) — sliding‑window limiter for recovery tracking

```typescript
import { markAccountUnavailable } from 'open-sse/services/auth';

await markAccountUnavailable({
  provider: 'anthropic',
  connectionId: 'key-123',
  errorCode: 429,
  retryAfter: 30, // seconds
});

```

### Tier 3: Model‑Lockout

The most granular tier handles **model‑specific failures**. When errors affect only a single model—such as missing model errors or model‑level quota exhaustion—that model is skipped while the connection remains usable for other models.

This prevents over‑punishment of healthy credentials and is implemented in [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts).

## Cross‑Provider Failover: Combo Routing

If all three tiers fail to yield a successful route, **auto‑combo routing** activates. This system selects an alternative provider/model using a 14‑factor scoring system that weighs latency, cost, capability, and current health status.

The combo router respects the tier decisions above—excluded providers, cooled connections, and locked models are automatically filtered from consideration.

## Specialized Fallback Layers

Certain request types trigger additional fallback logic beyond the three‑tier hierarchy.

### Web‑Search Tool Fallback

Requests containing native `web_search` tools receive two‑layer fallback handling in [`open-sse/services/webSearchRouting.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/webSearchRouting.ts) and [`open-sse/services/webSearchFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/webSearchFallback.ts):

| Layer | Action |
|-------|--------|
| 1 | Convert native `web_search` to internal `omniroute_web_search` function |
| 2 | Divert entire request to pre‑configured `webSearchRouteModel` |

```typescript
import { resolveWebSearchRouteOverride } from 'open-sse/services/webSearchRouting';

const result = resolveWebSearchRouteOverride(
  'openai/gpt-4o-mini',
  requestBody,
  { webSearchRouteModel: 'openai/web-search' }
);

if (result.wasRouted) {
  console.log('Request diverted to', result.model);
}

```

### Quota‑Text Fallback

Providers that expose quota exhaustion in plain text responses—without structured error codes—are handled by [`open-sse/services/quotaTextCooldowns.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/quotaTextCooldowns.ts). This pattern‑matching layer catches quota messages that would otherwise trigger unnecessary retries.

### Proxy Fallback

When direct upstream fetch fails, [`open-sse/utils/proxyFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/utils/proxyFallback.ts) attempts proxy selection before falling back to native fetch. This provides resilience against network‑level connectivity issues rather than provider‑level failures.

## Key Implementation Files

| File | Responsibility |
|------|----------------|
| [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) | Provider‑wide circuit breaker (tier 1) |
| [`open-sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/auth.ts) (`markAccountUnavailable`) | Connection‑level unavailability marking (tier 2) |
| [`open-sse/services/providerCooldownTracker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/providerCooldownTracker.ts) | Sliding‑window limiter for connection recovery |
| [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts) | Model‑lockout logic (tier 3) |
| [`open-sse/services/webSearchRouting.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/webSearchRouting.ts) | Web‑search route override configuration |
| [`open-sse/services/webSearchFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/webSearchFallback.ts) | Native tool conversion fallback |
| [`open-sse/services/quotaTextCooldowns.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/quotaTextCooldowns.ts) | Text‑based quota detection |
| [`open-sse/utils/proxyFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/utils/proxyFallback.ts) | HTTP proxy selection on fetch failure |
| [`open-sse/services/combo.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/combo.ts) | Auto‑combo provider selection with 14‑factor scoring |

## Summary

- **Three isolated tiers** handle failures at provider, connection, and model granularity before cross‑provider failover
- **Lazy recovery** design uses timestamp checks rather than background timers, preventing retry storms
- **Specialized layers** address web‑search tools, text‑based quota errors, and proxy connectivity
- **Auto‑combo routing** provides final fallback through 14‑factor provider scoring
- All tiers respect operator‑configured policies and retry limits as implemented in diegosouzapw/OmniRoute

## Frequently Asked Questions

### How does OmniRoute prevent cascading failures when a major provider goes down?

The provider‑level circuit breaker in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) automatically excludes the entire provider from routing once `providerFailureThreshold` errors are detected. This happens before any request is dispatched, preventing wasted attempts and allowing immediate failover to alternate providers through combo routing.

### What happens when only one API key is rate‑limited but others work fine?

Connection‑level cool‑down isolates the specific credential via `markAccountUnavailable()` in [`open-sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/auth.ts), setting a `rateLimitedUntil` timestamp. The provider remains active for routing, and other connections continue serving traffic. Recovery is automatic when the timestamp expires.

### Can operators configure custom behaviors for web‑search requests?

Yes. Operators can set a `webSearchRouteModel` parameter that triggers `resolveWebSearchRouteOverride()` in [`open-sse/services/webSearchRouting.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/webSearchRouting.ts). This diverts requests containing native web‑search tools to a designated model, useful for providers with specialized search‑optimized endpoints.