# How OmniRoute's 3-Layer Resilience System Prevents Cascading Failures

> Learn how OmniRoute's 3 layer resilience system prevents cascading failures. It isolates problems at the provider, connection, and model levels, stopping single points of failure from spreading.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: deep-dive
- Published: 2026-08-16

---

**OmniRoute prevents cascading failures by isolating problems at three distinct scopes—provider, connection, and model—so a single point of failure cannot propagate through the request pipeline.**

When routing LLM requests across multiple providers, a single failing upstream can quickly spiral into a system-wide outage without proper isolation. OmniRoute solves this through a layered resilience architecture that operates at different granularities. This article examines how each layer works based on the source code in `diegosouzapw/OmniRoute`.

## Layer 1: Provider Circuit Breaker

The **provider circuit breaker** operates at the broadest scope, protecting the entire request pipeline when an upstream provider fails repeatedly.

### How It Works

In [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts), OmniRoute tracks provider-wide error codes (`408, 500, 502, 503, 504`) across all connections. When failures exceed the configured **providerFailureThreshold**, the breaker transitions to **OPEN** and blocks all traffic to that provider. This happens atomically—no new requests are accepted until recovery.

After **providerCooldownMs** elapses, the breaker enters **HALF_OPEN**. The next request becomes a probe: success closes the breaker, failure reopens it. This lazy recovery prevents thundering herds.

```typescript
import { getProviderStatus } from '@/shared/utils/circuitBreaker';

async function routeRequest(providerId: string) {
  const status = getProviderStatus(providerId);
  if (status === 'OPEN') {
    // Provider is circuit-broken – skip it in combo routing
    return fallbackToOtherProviders();
  }
  // Normal execution path
  return executeProviderRequest(providerId);
}

```

Without this layer, a provider experiencing a partial outage would slow every request through timeout accumulation. The circuit breaker forces an immediate hard stop.

## Layer 2: Connection Cooldown

The **connection cooldown** layer handles transient failures at the individual credential level—API keys, accounts, or endpoints within the same provider.

### How It Works

In [`src/sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/auth.ts), each connection maintains `rateLimitedUntil` metadata. When a transient error occurs (`429`, network timeout), `markAccountUnavailable()` applies exponential backoff: `baseCooldownMs * 2ⁿ` where `n` increments with repeated failures.

```typescript
import { markAccountUnavailable } from '@/sse/services/auth';

async function handleProviderResponse(resp) {
  if (resp.status === 429) {
    await markAccountUnavailable(resp.accountId, {
      errorCode: 429,
      backoffLevel: resp.retryAfter ? 1 : 0,
      rateLimitedUntil: Date.now() + (resp.retryAfter ?? 5_000),
    });
  }
}

```

Other connections for the same provider remain eligible. This isolates throttled or rate-limited keys without discarding the entire provider from rotation.

## Layer 3: Model Lockout

The **model lockout** layer provides the finest granularity, isolating failures to a specific model on a specific connection.

### How It Works

In [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts), model-specific errors (`429` for quota exhaustion, `404` for missing local models) trigger `lockoutModelOnConnection()`. Only that model becomes unavailable; other models on the same connection continue serving requests.

```typescript
import { lockoutModelOnConnection } from '@/open-sse/services/accountFallback';

async function processModelError(connId, model, err) {
  if (err.code === 429 && err.modelSpecific) {
    await lockoutModelOnConnection(connId, model, err);
  }
}

```

Without this layer, a single model's quota limit would disable an entire API key. Model lockout preserves connection utility by narrowing the failure scope to exactly what's broken.

## How the Layers Interact During Failures

OmniRoute applies these layers hierarchically during request routing:

1. **Provider check first** – If the circuit breaker is `OPEN`, skip the provider entirely.
2. **Connection filter** – Among available providers, exclude connections where `rateLimitedUntil > now()`.
3. **Model eligibility** – From remaining connections, filter out models currently locked out.

The combo routing logic in [`open-sse/services/combo.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/combo.ts) orchestrates this selection. Each layer's state is evaluated independently, so multiple failure modes can coexist without interference.

| Failure Scenario | Layer Activated | Impact Scope |
|------------------|---------------|--------------|
| OpenAI regional outage | Provider Circuit Breaker | All OpenAI traffic blocked |
| Single API key throttled | Connection Cooldown | Only that key skipped |
| GPT-4 quota exhausted on key A | Model Lockout | GPT-4 unavailable on key A; GPT-3.5 and other keys unaffected |

## Summary

- **Provider circuit breaker** in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) stops traffic to entire failing providers using threshold-based state transitions.
- **Connection cooldown** in [`src/sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/auth.ts) isolates individual credentials with exponential backoff timers.
- **Model lockout** in [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts) restricts failures to specific models without disabling whole connections.
- The three layers operate independently, ensuring no single failure mode can cascade into system-wide unavailability.

## Frequently Asked Questions

### What error codes trigger the provider circuit breaker?

The provider circuit breaker monitors HTTP `408, 500, 502, 503, 504` as defined in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts). These indicate upstream or service-level failures rather than client or rate-limiting errors.

### How does connection cooldown differ from the circuit breaker?

Connection cooldown handles transient, recoverable errors (`429`, timeouts) at the individual credential level using timed backoff. The circuit breaker handles persistent, provider-wide failures using state machine transitions. A throttled key enters cooldown; a provider returning `503` errors opens the breaker.

### Can a model be locked out on one connection but available on another?

Yes. Model lockout is scoped to `connectionId + model`. The same model remains available on other connections for the same provider, and other models remain available on the affected connection.

### How does OmniRoute recover from a circuit breaker OPEN state?

Recovery uses lazy probing. After `providerCooldownMs` expires, the breaker enters `HALF_OPEN`. The next request through becomes a probe—success closes the breaker immediately, failure reopens it for another cooldown period.