# How OmniRoute's 4-Tier Fallback System Prioritizes Providers for LLM Routing

> Discover how OmniRoute's 4-tier fallback system prioritizes LLM providers by isolating failures at provider, connection, model, and policy levels. Ensure seamless routing.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: architecture
- Published: 2026-08-14

---

**OmniRoute uses a deterministic four-step fallback ladder that isolates failures at the provider, connection, model, and policy levels before selecting the next viable provider from a priority-sorted chain.**

OmniRoute's provider prioritization is implemented as a cascading filter system in the [diegosouzapw/OmniRoute](https://github.com/diegosouzapw/OmniRoute) repository. Each tier guards a different failure domain, ensuring that transient or localized issues don't trigger unnecessary provider switches while still maintaining high availability for LLM requests.

## Tier 1: Provider Circuit Breaker

The **provider circuit breaker** operates at the highest level, guarding entire provider backends (e.g., OpenAI, Anthropic) from being routed to when they are experiencing widespread issues.

In [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts), the breaker tracks upstream errors (408, 500‑504) and flips to **OPEN** state after a configurable threshold. When open, the provider is excluded from all routing decisions immediately.

This prevents cascading failures and gives troubled providers time to recover before traffic resumes.

## Tier 2: Connection Cooldown

Individual API keys or accounts within a provider can hit rate limits independently. The **connection cooldown** system in [`src/sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/auth.ts) marks connections with a `rateLimitedUntil` timestamp when they return transient errors (429, 503).

Connections whose cooldown timestamp is still in the future are skipped during provider selection, allowing other healthy connections from the same provider to continue serving requests.

## Tier 3: Model Lockout

Some failures are model-specific rather than connection-wide. The **model lockout** mechanism in [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts) isolates individual models on specific connections when quota or permission errors occur.

This granularity ensures that hitting a limit on `gpt-4o` doesn't disable the connection entirely—other models on the same key remain usable.

## Tier 4: Declarative Fallback Chain

After the first three tiers filter out unavailable providers, OmniRoute consults the **fallback chain** for the target model. This priority-sorted list is defined in [`src/domain/fallbackPolicy.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/domain/fallbackPolicy.ts) and maps model names to ordered provider arrays.

Lower `priority` values indicate higher precedence. The first enabled provider not already excluded by upstream tiers is selected:

```typescript
import {
  resolveFallbackChain,
  registerFallback,
  getNextFallback,
} from '@/domain/fallbackPolicy';

// Register priority order for gpt-4o
registerFallback('gpt-4o', [
  { provider: 'openai', priority: 0 },
  { provider: 'anthropic', priority: 1 },
  { provider: 'cohere', priority: 2, enabled: false }, // bypassed
]);

// After circuit breaker, cooldown, and lockout checks
const excluded = ['openai'];  // already attempted
const next = getNextFallback('gpt-4o', excluded);

if (next) {
  await dispatchToProvider(next, request);
} else {
  throw new Error('All fallback providers exhausted');
}

```

## The Complete Prioritization Flow

OmniRoute applies these tiers in strict sequence for every request:

1. **Circuit-breaker check** — remove providers with OPEN breakers
2. **Connection-cooldown check** — skip connections in rate-limit cooldown
3. **Model-lockout check** — exclude model+connection pairs that are locked
4. **Fallback-chain lookup** — select the lowest-priority enabled provider from the remaining set

If the chain exhausts without finding a viable provider, the request fails with a "fallback exhausted" error propagated to the client.

## Integrating Tier Checks in Request Handlers

The lower-level guards are evaluated before chain resolution:

```typescript
// From src/shared/utils/circuitBreaker.ts
if (circuitBreaker.isOpen('openai')) {
  // Entire provider excluded
}

// From src/sse/services/auth.ts
if (isConnectionOnCooldown(conn)) {
  // Specific key/account excluded
}

// From open-sse/services/accountFallback.ts
if (isModelLockedOut(conn, 'gpt-4o')) {
  // Specific model on this connection excluded
}

```

## Summary

- **Provider circuit breakers** guard entire backends against systemic failures
- **Connection cooldowns** isolate rate-limited keys without disabling providers
- **Model lockouts** provide fine-grained granularity for quota violations
- **Declarative fallback chains** determine final provider priority after all filters apply

Each tier in OmniRoute's 4-tier fallback system narrows the candidate pool before the next evaluation, ensuring deterministic, reproducible routing decisions under failure conditions.

## Frequently Asked Questions

### How does OmniRoute handle multiple rate limits on the same provider?

OmniRoute handles multiple rate limits through **connection-level isolation**. Each API key or account tracks its own `rateLimitedUntil` timestamp in [`src/sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/auth.ts). When one connection hits a 429, only that specific connection enters cooldown—other connections from the same provider remain available for routing.

### What happens if all providers in a fallback chain fail?

If all providers in a fallback chain are excluded by circuit breakers, cooldowns, lockouts, or prior attempts, OmniRoute throws a **"fallback exhausted" error** from [`src/domain/fallbackPolicy.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/domain/fallbackPolicy.ts). This error propagates to the client, indicating that no viable provider exists for the requested model at that moment.

### Can fallback chains be modified at runtime?

Yes. The `registerFallback()` function in [`src/domain/fallbackPolicy.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/domain/fallbackPolicy.ts) allows dynamic registration and updates of fallback chains. Each model can have its priority list adjusted without restarting the service, though existing in-flight requests use the chain state at the time of their evaluation.

### What's the difference between a circuit breaker and a connection cooldown?

The **circuit breaker** ([`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts)) operates at the provider level and responds to **server-side errors** (500s, timeouts) affecting the entire backend. The **connection cooldown** ([`src/sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/auth.ts)) operates at the key/account level and responds to **client-side rate limits** (429s) affecting only specific credentials.