# Understanding OmniRoute's 3-Layer Resilience Mechanism: Provider, Connection & Model-Level Protection

> Discover OmniRoute's 3-layer resilience mechanism provider connection and model-level protection. Learn how it isolates failures and maintains throughput without cascading outages.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: deep-dive
- Published: 2026-08-15

---

**OmniRoute implements a three-layer resilience system—provider-level circuit breakers, connection-level cooldowns, and model-level lockouts—to isolate failures and maintain request throughput without cascading outages.**

OmniRoute's 3-layer resilience mechanism is designed to handle failures at multiple granularities, from entire AI providers down to individual models on specific API keys. This architecture ensures that a single point of failure—whether it's a provider outage, a rate-limited credential, or a quota-exhausted model—does not degrade overall routing performance. Below, we examine how each layer operates, where the code lives, and how they interact during request processing.

---

## Provider-Level Circuit Breaker

The **provider-level circuit breaker** sits at the top of the resilience stack, monitoring all traffic to a given provider (e.g., `openai`, `anthropic`). When failures exceed configurable thresholds, the breaker opens and halts all requests to that provider.

### State Machine Implementation

In [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) (lines 5–17), the breaker implements a classic **CLOSED → DEGRADED → OPEN → HALF_OPEN → CLOSED** state machine. Key capabilities include:

- **Failure-type awareness**: The `PROVIDER_FAILURE_ERROR_CODES` array (408, 429, 500, 502, 503, 504) defines which HTTP status codes count toward the failure threshold.
- **Adaptive back-off**: After each `OPEN → HALF_OPEN → OPEN` cycle, the `_effectiveResetTimeout` escalates up to a configurable multiplier (lines 55–64).
- **Per-failure-kind thresholds**: The `kindThresholds` map (lines 78–84) allows distinct thresholds for `rate_limit`, `quota_exhausted`, and other failure types, including immediate-open behavior for critical errors.
- **State persistence**: Breaker state survives restarts via `saveCircuitBreakerState` and `loadCircuitBreakerState` (lines 19–24).

The public API exposes `recordProviderFailure` and `recordProviderSuccess` (lines 87–95 and 52–57), which automatically deduplicate rapid-fire failures per connection and aggregate provider-wide network errors.

---

## Connection-Level Cooldown

The **connection-level cooldown** provides finer-grained protection than the provider breaker. It targets individual credentials, API keys, or OAuth accounts while allowing other connections for the same provider to continue serving requests.

### How It Works

Each connection record stores a `rateLimitedUntil` timestamp in the database (defined in the `AccountState` type). When a request fails for a specific key, `setConnectionRateLimitUntil` advances this timestamp:

- **5 seconds** for OAuth-based connections
- **3 seconds** for API-key-based connections

The router skips connections where `new Date(rateLimitedUntil).getTime() > Date.now()` (see [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts), lines 10, 20–27).

The cooldown mechanism is **lazy**: no background process re-enables connections. When the timestamp passes, the connection becomes eligible automatically. Repeated failures trigger exponential back-off via `calculateBackoffCooldown`, capped at `maxCooldownMs` from the `PROVIDER_PROFILE` configuration.

---

## Model-Level Lockout

The **model-level lockout** is the most granular layer. It prevents a single unavailable or quota-limited model from disabling an entire connection containing multiple models.

### Lockout Lifecycle

In [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts) (lines 53–61), the `lockModel` function creates entries tracking:

- `reason`: The failure type (e.g., `quota_exhausted`, `model_not_found`)
- `until`: Expiration timestamp
- `failureCount`: Accumulated failures for backoff calculation

The cleanup timer (`ensureCleanupTimer`, lines 17–33) automatically removes expired locks. Public functions `isModelLocked`, `getModelLockoutInfo`, and `clearModelLock` expose lock state to dashboards and monitoring tools.

This layer is especially critical for providers with **per-model quotas** (Gemini, Codex, Antigravity), where one model's exhaustion should not affect others on the same credential.

---

## How the Three Layers Interact

During request processing, OmniRoute evaluates resilience layers in strict order:

1. **Combo routing** generates candidate targets.
2. For each target, `accountFallback` checks:
   - **Provider breaker**: `isProviderInCooldown` (line 57) aborts if the breaker is OPEN.
   - **Connection cooldown**: Skips if `rateLimitedUntil` is in the future (line 10).
   - **Model lockout**: `isModelLocked` (line 36) blocks specific model-connection pairs.
3. If the request fails, the appropriate layer records it:
   - `recordProviderFailure` → circuit breaker
   - `setConnectionRateLimitUntil` → connection cooldown
   - `recordModelLockoutFailure` (lines 19–28) → model lockout
4. The router retries the next candidate, automatically respecting new cooldowns and locks.

Refer to `docs/diagrams/resilience-3layers.svg` for a visual representation of this flow.

---

## Practical Code Examples

### Check if a provider is in cooldown

```typescript
import { isProviderInCooldown } from '@/lib/usage/accountFallback';

if (isProviderInCooldown('openai')) {
  // Skip all OpenAI targets for this request
}

```

### Record a connection-level rate limit

```typescript
import { setConnectionRateLimitUntil } from '@/lib/db/providers';
import { getConnectionById } from '@/lib/db/connections';

async function onRateLimited(connectionId: string) {
  const conn = await getConnectionById(connectionId);
  const cooldownMs = conn.category === 'oauth' ? 5000 : 3000;
  setConnectionRateLimitUntil(connectionId, Date.now() + cooldownMs);
}

```

### Lock a model after quota exhaustion

```typescript
import { lockModel } from '@/lib/usage/accountFallback';

function handleQuotaExhausted(provider: string, connectionId: string, model: string) {
  const ONE_HOUR = 60 * 60 * 1000;
  lockModel(provider, connectionId, model, 'quota_exhausted', ONE_HOUR);
}

```

### Query active model lockouts for dashboards

```typescript
import { getAllModelLockouts } from '@/lib/usage/accountFallback';

const lockouts = getAllModelLockouts();
lockouts.forEach(l => {
  console.log(`${l.provider}:${l.connectionId}:${l.model} locked for ${l.reason}, ${l.remainingMs}ms left`);
});

```

---

## Key Source Files

| File | Purpose |
|------|---------|
| [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) | Provider-level circuit breaker: state machine, persistence, per-failure-kind thresholds. |
| [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts) | Connection cooldown, model lockouts, provider profile resolution, router integration. |
| `docs/diagrams/resilience-3layers.svg` | Visual diagram of layer interactions. |
| [`src/lib/resilience/settings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/settings.ts) | Default resilience configuration: cooldown durations, thresholds, back-off steps. |
| [`src/lib/resilience/modelLockoutSettings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/modelLockoutSettings.ts) | Per-model lockout settings and maximum cooldown limits. |

---

## Summary

- **Provider-level circuit breakers** isolate entire providers during systemic failures, with adaptive back-off and state persistence across restarts.
- **Connection-level cooldowns** temporarily disable individual credentials while preserving provider access through alternate keys.
- **Model-level lockouts** prevent single-model issues from poisoning multi-model connections.

These layers operate sequentially during routing decisions and record failures independently, enabling automatic recovery without manual intervention.

---

## Frequently Asked Questions

### What triggers a provider-level circuit breaker to open?

The circuit breaker opens when failures exceeding the configured threshold occur within a time window. In [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts), failures are counted for HTTP status codes in `PROVIDER_FAILURE_ERROR_CODES` (408, 429, 500, 502, 503, 504). Per-failure-kind thresholds in `kindThresholds` (lines 78–84) allow `quota_exhausted` or `rate_limit` errors to trigger faster opens than transient network errors.

### How does OmniRoute handle multiple API keys for the same provider?

Each key is a separate connection with its own `rateLimitedUntil` timestamp. When one key hits a rate limit, `setConnectionRateLimitUntil` disables only that connection (typically 3–5 seconds), while other connections for the same provider continue serving requests. This is implemented in [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts) (lines 20–27).

### Can model lockouts persist longer than connection cooldowns?

Yes. Model lockouts default to longer durations—often 1 hour for `quota_exhausted`—while connection cooldowns typically cap at seconds or minutes. The `lockModel` function (lines 53–61) accepts arbitrary `until` timestamps, and [`modelLockoutSettings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/modelLockoutSettings.ts) configures maximum limits per provider.

### What happens to in-flight requests when a breaker opens?

In-flight requests complete normally; the breaker only affects new routing decisions. The `isProviderInCooldown` check (line 57) filters targets before request dispatch, so open breakers cause immediate failover to alternate providers without aborting active connections.