# How OmniRoute's 3-Layer Resilience System Protects Against Provider Failures

> Learn how OmniRoute's 3-layer resilience system protects against provider failures with circuit breaker, connection cooldown, and model lockout for automatic fallback and high availability.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: architecture
- Published: 2026-08-24

---

**OmniRoute uses a hierarchical circuit breaker, connection cooldown, and model lockout system to isolate failures at the provider, credential, and model levels, ensuring automatic fallback and high availability.**

OmniRoute is an open-source LLM routing proxy that maintains request throughput even when upstream providers fail. Its **3-layer resilience system** operates at progressively finer scopes—provider, connection, and model—to contain failures without disrupting healthy traffic paths.

---

## Provider Circuit Breaker: Coarse-Grained Failure Isolation

The **provider circuit breaker** guards against complete provider outages by halting all traffic to a provider that repeatedly returns upstream errors.

Located in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts), the breaker implements a standard **CLOSED → OPEN → HALF_OPEN** state machine:

- When failure threshold is hit (e.g., multiple 5xx responses), the breaker flips to **OPEN** and blocks all requests for a configurable cooldown period
- The next status read after timeout transitions to **HALF_OPEN**
- A successful probe request returns the breaker to **CLOSED**; another failure reopens it

This **lazy recovery** design requires no background timer—state transitions trigger only on demand when `canExecute()` or `getProviderStatus()` is called.

```typescript
// Example: Checking provider status before sending a request
import { getProviderStatus, canExecute } from '@/shared/utils/circuitBreaker';

if (canExecute('openai')) {
  // Provider is CLOSED or HALF_OPEN – safe to use
  await sendChatRequest(...);
} else {
  // Provider is OPEN – skip or use fallback
  await fallbackToAnotherProvider();
}

```

The circuit breaker prevents cascade failures by giving providers time to recover before traffic resumes.

---

## Connection Cooldown: Credential-Level Recovery

The **connection cooldown** layer protects individual credentials (API keys, accounts) within a provider, allowing healthy keys to continue serving traffic while problematic ones rest.

Implementation spans two files:

- `src/sse/services/auth.ts::markAccountUnavailable()` — marks failed connections with `rateLimitedUntil` timestamp and `testStatus = "unavailable"`
- `open-sse/services/accountFallback.ts::checkFallbackError()` — filters unavailable connections during credential selection

When a request fails with recoverable errors (429, 500), the connection receives an **exponential back-off** penalty:

```typescript
// Example: Marking a connection as unavailable after a 429 response
import { markAccountUnavailable } from '@/sse/services/auth';

async function handleResponse(res: Response, connId: string) {
  if (res.status === 429) {
    // Apply exponential back-off: baseCooldownMs * 2 ** failureIndex
    await markAccountUnavailable(connId, {
      errorCode: 429,
      backoffLevel: 1, // incremented internally
    });
  }
}

```

The cooldown formula `baseCooldownMs * 2 ** failureIndex` ensures aggressive back-off for repeatedly failing keys. Like the circuit breaker, recovery is **lazy**—once `rateLimitedUntil` passes, the connection becomes eligible again without active reconciliation.

---

## Model Lockout: Fine-Grained Quota Protection

The **model lockout** layer isolates failures to specific models, preventing a single model's quota exhaustion or permission error from disabling an entire connection.

Implemented in [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts), this mechanism handles model-specific errors:

- **429 quota exceeded** on a particular model
- **404 model not found** or permission denied

Only the affected model is flagged; the same connection continues serving other models.

```typescript
// Example: Locking out a single model after a quota error
import { lockModel } from '@/open-sse/services/accountFallback';

if (error.code === 429 && error.model === 'gpt-4') {
  await lockModel('openai', connectionId, 'gpt-4');
}

```

This **granular containment** preserves capacity across the model portfolio and avoids over-reaction to transient quota limits on popular models.

---

## How the Three Layers Work Together

OmniRoute's resilience system forms a **hierarchical failure containment strategy**:

| Priority | Layer | Failure Scope | Response Speed |
|----------|-------|-------------|--------------|
| 1 | Circuit Breaker | Entire provider | Fastest—immediate cut-off |
| 2 | Connection Cooldown | Single credential | Medium—back-off and retry |
| 3 | Model Lockout | Specific model | Granular—preserve connection utility |

When a provider fails completely, the **circuit breaker** triggers first, routing traffic to alternate providers via combo routing. If only one key within a provider fails, **connection cooldown** preserves other keys. If a single model hits quota, **model lockout** keeps the connection alive for remaining models.

All three mechanisms respect upstream back-off signals while maintaining **automatic fallback** through OmniRoute's routing layer.

---

## Summary

- **Provider circuit breaker** ([`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts)) stops all traffic to unhealthy providers using lazy state transitions
- **Connection cooldown** ([`src/sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/auth.ts), [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts)) applies exponential back-off to individual credentials while preserving sibling keys
- **Model lockout** ([`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts)) isolates per-model failures without disabling entire connections
- Combined, these layers enable **automatic fallback**, **high availability**, and **respectful back-off** toward upstream services

---

## Frequently Asked Questions

### How quickly does the circuit breaker react to provider failures?

The circuit breaker updates state immediately upon threshold violation. Since it uses **lazy recovery**, no polling overhead exists—status checks occur only when `canExecute()` evaluates a routing decision. Typical detection-to-isolation latency is one failed request plus the threshold count.

### What happens if all providers in a combo route are OPEN?

When all configured providers are circuit-broken, OmniRoute returns a **503 Service Unavailable** with diagnostic metadata indicating all breaker states. Administrators can configure **dead-letter routing** or **queue-and-retry** policies in [`docs/architecture/RESILIENCE_GUIDE.md`](https://github.com/diegosouzapw/OmniRoute/blob/main/docs/architecture/RESILIENCE_GUIDE.md).

### Does connection cooldown persist across server restarts?

By default, cooldown state is **in-memory only** and resets on restart. For distributed deployments, Redis-backed state can be enabled in [`src/sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/auth.ts) by configuring a shared cache provider.

### Can model lockout be configured for specific error codes?

Yes. The `lockModel()` function in [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts) accepts configurable error predicates. Modify `checkFallbackError()` to include additional codes (e.g., 402 payment required) or exclude certain transient errors from triggering lockout.