# OmniRoute 3-Layer Resilience Model: How Circuit Breaker, Connection Cooldown, and Model Lockout Work

> Discover OmniRoute's 3-layer resilience model: Circuit Breaker, Connection Cooldown, and Model Lockout. Ensure continuous LLM service availability by isolating failures effectively.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: deep-dive
- Published: 2026-08-22

---

**OmniRoute's three-layer resilience architecture isolates failures at the provider, connection, and model levels through hierarchical circuit breaking, exponential backoff cooldowns, and success-decay lockouts to ensure continuous LLM service availability.**

The diegosouzapw/OmniRoute repository implements this sophisticated **OmniRoute 3-layer resilience model** to handle cascading failures in multi-provider LLM routing. By separating concerns across provider-wide outages, individual API credential issues, and per-model quotas, the system minimizes downtime and automates recovery without manual intervention.

## Layer 1: Provider Circuit Breaker

The **Provider Circuit Breaker** operates at the entire provider scope (e.g., `openai`, `anthropic`), guarding against repeated upstream service errors such as 5xx responses and timeouts that affect all API keys for that provider.

According to the source code in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts), the breaker maintains four states: **CLOSED**, **DEGRADED**, **OPEN**, and **HALF_OPEN**. When the failure count crosses the configurable `failureThreshold`, the state transitions from CLOSED → DEGRADED → OPEN. While OPEN, the provider is completely excluded from combo routing.

Recovery happens lazily. When code calls `getStatus()`, `canExecute()`, or `getRetryAfterMs()` after the `resetTimeout` expires, the breaker automatically moves to HALF_OPEN and allows a probe request. A successful probe closes the breaker; another failure re-opens it.

Default thresholds vary by provider type:

- **OAuth**: Degraded at 5 failures, opens at 8 failures, resets after 60 seconds
- **API-key**: Degraded at 7 failures, opens at 12 failures, resets after 30 seconds
- **Local**: Opens at 2 failures, resets after 15 seconds

## Layer 2: Connection Cooldown

The **Connection Cooldown** layer isolates transient errors specific to a single API key or account, such as rate limits, temporary authentication errors, or network hiccups. This prevents one bad credential from triggering a provider-wide circuit breaker.

When `checkFallbackError()` in [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts) classifies an error as transient, the system calls `markAccountUnavailable()` to set a back-off timestamp (`rateLimitedUntil`). The cooldown duration grows exponentially using `baseCooldownMs * 2ⁿ`, starting at 5 seconds for OAuth keys and 3 seconds for API-key keys.

Recovery is lazy: once `rateLimitedUntil` passes, the connection becomes eligible again. A successful request clears all error fields via `clearAccountError()`. The helper function `canExecute()` in [`src/sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/auth.ts) checks eligibility without requiring manual intervention.

## Layer 3: Model Lockout

The **Model Lockout** mechanism targets failures isolated to a specific model within a provider and connection triple (provider + connection ID + model name). This handles per-model quotas, missing models, or mode-specific permission errors.

When `recordModelLockoutFailure()` detects a model-scoped error, it invokes `lockModel()` to create a temporary lockout entry. Unlike the circuit breaker, model lockouts implement **success-decay** recovery via `decayModelFailureCount()`, which halves the failure count on successful requests and eventually deletes the lockout entry entirely.

Model lockout settings are defined in [`src/lib/resilience/modelLockoutSettings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/modelLockoutSettings.ts), with default configurations allowing operators to enable or disable the feature and define specific error codes that trigger lockouts.

## How the Layers Interact During Routing

When OmniRoute processes a request, it evaluates resilience layers in strict hierarchical order:

1. **Circuit Breaker Check**: If the provider state is OPEN, the provider is omitted from the candidate list entirely.
2. **Connection Cooldown Filter**: For remaining providers, connections with `rateLimitedUntil` timestamps in the future are filtered out.
3. **Model Lockout Exclusion**: For surviving connections, models with active lockout entries are excluded.

Only if all connections for a provider are filtered out does the combo engine fall back to the next provider or invoke emergency fallback logic. This sequential evaluation ensures that granular issues (single model) don't trigger broader isolation (entire provider) unless necessary.

## Inspecting and Managing Resilience State

### Checking Provider Circuit Breaker Status

To programmatically inspect circuit breaker states:

```typescript
import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';
import { Provider } from '@/shared/constants/providers';

async function logProviderStatus(provider: Provider) {
  const cb = await getCircuitBreaker(provider);
  console.log(`${provider} → state: ${cb.state}, failures: ${cb.failureCount}`);
}

// Example: check OpenAI status
await logProviderStatus('openai');

```

### Resetting Breakers via Admin API

To manually reset a provider's circuit breaker:

```bash
curl -X POST \
  -H "Authorization: Bearer $ADMIN_JWT" \
  https://localhost:20128/api/resilience/reset \
  -d '{"provider":"anthropic"}' \
  -H "Content-Type: application/json"

```

This endpoint is implemented in [`src/sse/handlers/chatHelpers.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/handlers/chatHelpers.ts) via the `resetCircuitBreaker()` function.

### Detecting Connection Cooldowns

Use the `canExecute()` helper to check connection availability:

```typescript
import { canExecute } from '@/sse/services/auth';

async function executeIfAvailable(conn) {
  if (!canExecute(conn)) {
    console.warn(`Connection ${conn.id} is in cooldown until ${new Date(conn.rateLimitedUntil)}`);
    return; // skip or fallback
  }
  // normal executor logic …
}

```

### Listing Active Model Lockouts

Query active model lockouts via the REST API:

```bash
curl -H "Authorization: Bearer $ADMIN_JWT" \
  https://localhost:20128/api/resilience/model-cooldowns

```

This endpoint is defined in [`src/app/api/v1/resilience/model-cooldowns/route.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/app/api/v1/resilience/model-cooldowns/route.ts).

### Clearing Specific Model Lockouts

To manually clear a model lockout:

```bash
curl -X DELETE \
  -H "Authorization: Bearer $ADMIN_JWT" \
  -H "Content-Type: application/json" \
  -d '{"provider":"openai","connectionId":"abc123","model":"gpt-4o"}' \
  https://localhost:20128/api/resilience/model-cooldowns

```

The underlying logic resides in [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts) in the `clearModelLock()` function.

## Summary

- **Provider Circuit Breakers** protect entire LLM providers from cascading failures using state machines (CLOSED → DEGRADED → OPEN → HALF_OPEN) with lazy recovery mechanisms implemented in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts).
- **Connection Cooldowns** isolate individual API keys through exponential back-off strategies, preventing single-credential issues from affecting provider-wide availability.
- **Model Lockouts** provide granular protection at the model level using success-decay algorithms that automatically heal after successful requests.
- The three layers operate sequentially during request routing, ensuring that isolation occurs at the most specific applicable scope.
- All layers support both automatic lazy recovery and manual administrative intervention via REST APIs and database queries.

## Frequently Asked Questions

### How do I determine which resilience layer is blocking my requests?

Check the scope of the failure. If **all keys for a provider are skipped**, inspect the circuit breaker state in `domain_circuit_breakers`. If **only one API key fails**, examine the connection's `rateLimitedUntil` field for a cooldown issue. If **only a specific model fails**, look for lockout entries in [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts). Reading raw database columns won't trigger lazy recovery; use helper functions like `getStatus()` or `canExecute()` instead.

### Why does the circuit breaker use lazy recovery instead of scheduled jobs?

Lazy recovery reduces system complexity and resource usage. The breaker transitions from OPEN to HALF_OPEN only when `getStatus()`, `canExecute()`, or `getRetryAfterMs()` is called after the `resetTimeout` expires. This ensures that recovery checks happen precisely when needed—during actual routing decisions—rather than consuming resources with background polling.

### Can I adjust the exponential backoff for connection cooldowns?

Yes. The base cooldown values are defined in [`src/lib/resilience/settings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/settings.ts)—5 seconds for OAuth keys and 3 seconds for API-key keys. The backoff follows the formula `baseCooldownMs * 2ⁿ` where *n* is the retry count. Modify these base values or the classification logic in `checkFallbackError()` to customize cooldown behavior for specific error types.

### What triggers a model lockout versus a connection cooldown?

Connection cooldowns handle transient, credential-specific errors like HTTP 429 rate limits or temporary network failures detected by `checkFallbackError()`. Model lockouts trigger on persistent model-specific errors such as quota exhaustion for individual models (`recordModelLockoutFailure()`), missing model IDs, or mode-specific permission denials that affect only the provider+connection+model triple.