# How OmniRoute's Three-Layer Resilience System Works: Circuit Breakers, Cooldowns, and Model Lockouts

> Discover OmniRoute's three-layer resilience system: circuit breakers, cooldowns, and model lockouts. Learn how it protects AI service integrations from cascading outages and isolates failures.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: internals
- Published: 2026-08-23

---

**OmniRoute protects request throughput through a hierarchical three-layer resilience architecture that isolates failures at the provider, connection, and model levels to prevent cascading outages across AI service integrations.**

OmniRoute (available at diegosouzapw/OmniRoute) is an open-source request router designed for AI model providers. Its three-layer resilience system ensures that transient upstream failures, rate limits, or quota exhaustion on specific credentials do not degrade the entire routing pipeline. The implementation spans [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) and [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts), creating a defense-in-depth strategy against provider instability.

## Provider-Level Circuit Breaker

The outermost layer resides in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) (lines 4-17) and implements a classic **CLOSED → DEGRADED → OPEN → HALF_OPEN → CLOSED** state machine. This protects all connections belonging to a provider (e.g., `openai` or `anthropic`) by stopping traffic to any provider experiencing repeated upstream failures.

**Failure-type awareness** drives the breaker logic. The `PROVIDER_FAILURE_ERROR_CODES` array defines which HTTP status codes increment the failure threshold, including `408`, `429`, `500`, `502`, `503`, and `504` (lines 20-27). When failure counts exceed configured thresholds, the breaker transitions to **OPEN**, rejecting requests immediately until the reset timeout expires.

The implementation features **adaptive back-off** through the `_effectiveResetTimeout` mechanism (lines 55-64). After each `OPEN → HALF_OPEN → OPEN` cycle, the timeout escalates according to a configurable multiplier, preventing aggressive retry storms against still-unhealthy providers.

**Per-failure-kind thresholds** allow granular control via `kindThresholds` (lines 78-84). Different failure types—such as `rate_limit` or `quota_exhausted`—can trigger distinct thresholds and immediate-open behavior. **State persistence** ensures that breaker status survives process restarts through `saveCircuitBreakerState` and `loadCircuitBreakerState` (lines 19-24).

Helpers such as `recordProviderFailure` (lines 87-95) and `recordProviderSuccess` (lines 52-57) automatically deduplicate rapid-fire failures per connection and manage per-provider network error accounting.

## Connection-Level Cooldown

While the provider breaker handles aggregate failures, the connection-level cooldown isolates individual credentials or API keys. Implemented in [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts) (lines 20-27), this layer tracks a `rateLimitedUntil` timestamp on each connection record.

When a request fails for a specific key, `setConnectionRateLimitUntil` advances the timestamp (typically 5 seconds for OAuth connections or 3 seconds for API-key based connections). The router skips any connection where `new Date(rateLimitedUntil).getTime() > Date.now()`, creating a **lazy cooling** mechanism that requires no additional housekeeping—connections automatically become eligible once the timestamp passes.

Repeated failures trigger **exponential back-off** via `calculateBackoffCooldown`, respecting the `maxCooldownMs` and `maxBackoffSteps` values defined in `PROVIDER_PROFILE` configurations. This prevents hammering a temporarily rate-limited key while allowing faster recovery than provider-level outages.

## Model-Level Lockout

The innermost layer addresses per-model quota limitations on specific connections. When a provider uses per-model quotas (e.g., Gemini, Codex), OmniRoute tracks failures per **provider:connection:model** tuple in [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts) (lines 53-61).

The `lockModel` function creates a lock entry containing the `reason`, `until` timestamp, and `failureCount`. This prevents the router from using a specific model on a particular connection when only that model is unavailable or quota-limited, avoiding the unnecessary disablement of an entire connection for a single model failure.

Lock state is exposed through `isModelLocked`, `getModelLockoutInfo`, and `clearModelLock` for dashboard visibility. An automated cleanup timer (`ensureCleanupTimer`, lines 17-33) purges expired locks without manual intervention.

## How the Three Layers Interact

During request processing, OmniRoute evaluates resilience layers in strict order before attempting a connection. The `accountFallback` service performs three checks for each routing target:

1. **Provider breaker status** – `isProviderInCooldown` (line 57) aborts the target if the provider-level circuit breaker is **OPEN**.
2. **Connection cooldown** – The router skips connections with a future `rateLimitedUntil` timestamp (line 10).
3. **Model lockout** – `isModelLocked` (line 36) prevents use of a specific model on that connection.

If a request fails, the system records the failure at the appropriate granularity. `recordProviderFailure` increments the circuit breaker, `setConnectionRateLimitUntil` updates the connection timestamp, and `recordModelLockoutFailure` (lines 19-28) creates or extends a model lock. The router then attempts the next candidate, automatically honoring new cooldowns and locks.

This hierarchical approach ensures that a quota limit on `gpt-4` for one API key does not trigger a provider-wide outage, while a total provider failure does not permanently disable individual connections that might recover faster.

## Code Examples

### Checking Provider Circuit Breaker Status

Before routing, verify whether a provider is currently isolated:

```typescript
import { isProviderInCooldown } from '@/lib/usage/accountFallback';

if (isProviderInCooldown('openai')) {
  // Skip all OpenAI targets for this request
  return [];
}

```

### Recording Connection-Level Rate Limits

Handle 429 responses by cooling down the specific credential:

```typescript
import { setConnectionRateLimitUntil } from '@/lib/db/providers';
import { getConnectionById } from '@/lib/db/connections';

async function onRateLimited(connectionId: string) {
  const conn = await getConnectionById(connectionId);
  // 5 seconds base cooldown for OAuth, 3 seconds for API-key
  const cooldownMs = conn.category === 'oauth' ? 5000 : 3000;
  setConnectionRateLimitUntil(connectionId, Date.now() + cooldownMs);
}

```

### Locking Models After Quota Exhaustion

When a specific model hits its quota on a connection:

```typescript
import { lockModel } from '@/lib/usage/accountFallback';

function handleQuotaExhausted(provider: string, connectionId: string, model: string) {
  // 1 hour lock for quota exhausted
  const ONE_HOUR = 60 * 60 * 1000;
  lockModel(provider, connectionId, model, 'quota_exhausted', ONE_HOUR);
}

```

### Querying Active Model Lockouts

For dashboard monitoring or debugging:

```typescript
import { getAllModelLockouts } from '@/lib/usage/accountFallback';

const lockouts = getAllModelLockouts();
lockouts.forEach(l => {
  console.log(`${l.provider}:${l.connectionId}:${l.model} locked for ${l.reason}, ${l.remainingMs}ms left`);
});

```

## Summary

- **Provider-level circuit breakers** in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) use a finite state machine (CLOSED → DEGRADED → OPEN → HALF_OPEN → CLOSED) to halt traffic to failing providers, with adaptive back-off and persistent state across restarts.
- **Connection-level cooldowns** managed in [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts) isolate individual API keys through lazy timestamp evaluation (`rateLimitedUntil`), preventing bad credentials from affecting healthy ones.
- **Model-level lockouts** provide granular protection for per-model quotas, tracking failures per provider:connection:model tuple without disabling entire connections.
- The layers interact hierarchically during request routing, with failure recording automatically escalating protection at the appropriate granularity and recovery occurring independently at each level.

## Frequently Asked Questions

### How does OmniRoute prevent a single bad API key from disabling an entire provider?

OmniRoute uses **connection-level cooldowns** to isolate individual credentials. When a specific key returns rate limits or errors, `setConnectionRateLimitUntil` sets a future timestamp on that connection only ([`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts), lines 20-27). The provider-level circuit breaker tracks aggregate failures separately, ensuring that one problematic key does not trip the provider-wide breaker unless the failure pattern indicates a service-wide outage.

### What happens when the circuit breaker enters the HALF_OPEN state?

In **HALF_OPEN** state (defined in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts), lines 4-17), the breaker allows a limited number of test requests to pass through to probe provider health. If these succeed, the state transitions back to **CLOSED**; if they fail, it returns to **OPEN** with an increased `_effectiveResetTimeout` (lines 55-64). This adaptive back-off prevents retry storms while gradually restoring traffic to recovered services.

### Can different failure types trigger different circuit breaker thresholds?

Yes. The `kindThresholds` configuration (lines 78-84) allows distinct thresholds and immediate-open behavior for specific failure kinds such as `rate_limit` or `quota_exhausted`. This enables aggressive circuit breaking for permanent errors like authentication failures while tolerating transient 503 errors with higher thresholds.

### How long do model-level lockouts persist?

Model lockouts persist for the duration specified when calling `lockModel` (lines 53-61), after which the `ensureCleanupTimer` (lines 17-33) automatically purges expired entries. There is no hardcoded default; the lockout duration is caller-defined (e.g., one hour for quota exhaustion), and the cleanup timer ensures memory does not accumulate stale lockouts.