# OmniRoute 3-Layer Resilience System: How Circuit Breaker, Connection Cooldown, and Model Lockout Work

> Discover how OmniRoute's 3-layer resilience system (Circuit Breaker, Connection Cooldown, Model Lockout) isolates failures and enables automatic recovery at provider, credential, and model levels.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: internals
- Published: 2026-08-10

---

**OmniRoute uses three independent resilience layers—Provider Circuit Breaker, Connection Cooldown, and Model Lockout—to isolate failures at the provider, credential, and model levels, ensuring automatic recovery without manual intervention.**

The **3-layer resilience system** in the [diegosouzapw/OmniRoute](https://github.com/diegosouzapw/OmniRoute) repository protects request routing by operating at distinct scopes to handle different failure classes. Each layer maintains its own state machine and recovery logic, allowing the system to gracefully degrade when upstream AI providers experience outages, rate limits, or model-specific errors. Understanding how these mechanisms interact is essential for operating a production OmniRoute deployment.

## Layer 1: Provider Circuit Breaker

The **Provider Circuit Breaker** guards against repeated upstream service errors that affect all API keys of a specific provider, such as `openai` or `anthropic`.

### State Transitions and Failure Thresholds

Implemented in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts), the breaker tracks provider-level failures across all connections. When the failure count crosses the `failureThreshold`, the state transitions from **CLOSED → DEGRADED → OPEN**. While **OPEN**, the provider is completely excluded from combo routing.

Configuration defaults vary by provider type:
- **OAuth**: Degrades at 5 failures, opens at 8 failures, resets after 60 seconds
- **API-key**: Degrades at 7 failures, opens at 12 failures, resets after 30 seconds  
- **Local**: Opens at 2 failures, resets after 15 seconds

### Lazy Recovery Mechanism

Recovery occurs lazily when `getStatus()`, `canExecute()`, or `getRetryAfterMs()` is called after the `resetTimeout` expires. This automatically moves the breaker to **HALF_OPEN**, allowing a single probe request. A successful probe closes the breaker; another failure re-opens it.

```typescript
import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';
import { Provider } from '@/shared/constants/providers';

async function logProviderStatus(provider: Provider) {
  const cb = await getCircuitBreaker(provider);
  console.log(`${provider} → state: ${cb.state}, failures: ${cb.failureCount}`);
}

// Example: check OpenAI status
await logProviderStatus('openai');

```

To manually reset a breaker via the admin API:

```bash
curl -X POST \
  -H "Authorization: Bearer $ADMIN_JWT" \
  https://localhost:20128/api/resilience/reset \
  -d '{"provider":"anthropic"}' \
  -H "Content-Type: application/json"

```

*Implementation reference: [`src/sse/handlers/chatHelpers.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/handlers/chatHelpers.ts) → `resetCircuitBreaker()`*

## Layer 2: Connection Cooldown

The **Connection Cooldown** layer isolates transient errors specific to a single credential or API key, such as rate limits, temporary authentication failures, or network hiccups.

### Exponential Back-off Strategy

When `checkFallbackError()` in [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts) classifies an error as transient, the connection is marked unavailable via `markAccountUnavailable()`. The system sets a `rateLimitedUntil` timestamp calculated as `baseCooldownMs * 2ⁿ`, where base values are 5 seconds for OAuth keys and 3 seconds for API-key keys.

```typescript
import { canExecute } from '@/sse/services/auth';

async function executeIfAvailable(conn) {
  if (!canExecute(conn)) {
    console.warn(`Connection ${conn.id} is in cooldown until ${new Date(conn.rateLimitedUntil)}`);
    return; // skip or fallback
  }
  // normal executor logic …
}

```

*Helper defined in `src/sse/services/auth.ts::canExecute`*

### Integration with Request Routing

During the routing phase in [`src/sse/handlers/chat.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/handlers/chat.ts), connections with a future `rateLimitedUntil` timestamp are filtered from the candidate pool. Recovery happens automatically once the timestamp passes, and a successful request clears all error fields via `clearAccountError()`.

## Layer 3: Model Lockout

The **Model Lockout** layer handles failures isolated to a specific provider-connection-model triple, such as per-model quota exhaustion or missing model permissions.

### Per-Model Failure Isolation

When `recordModelLockoutFailure()` detects a model-scoped error, it invokes `lockModel()` in [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts) to create a lockout entry. This prevents subsequent requests from attempting that specific model on that connection for a configurable duration.

### Success-Decay Recovery

Model lockout offers two recovery paths:
1. **Timer expiration**: The lockout automatically expires after the cooldown period
2. **Success-decay**: A successful request triggers `decayModelFailureCount()`, which halves the failure count and eventually deletes the lockout entry entirely

List active lockouts via the REST endpoint:

```bash
curl -H "Authorization: Bearer $ADMIN_JWT" \
  https://localhost:20128/api/resilience/model-cooldowns

```

*Endpoint implemented in [`src/app/api/v1/resilience/model-cooldowns/route.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/app/api/v1/resilience/model-cooldowns/route.ts)*

Clear a specific lockout manually:

```bash
curl -X DELETE \
  -H "Authorization: Bearer $ADMIN_JWT" \
  -H "Content-Type: application/json" \
  -d '{"provider":"openai","connectionId":"abc123","model":"gpt-4o"}' \
  https://localhost:20128/api/resilience/model-cooldowns

```

*Lockout removal logic lives in `open-sse/services/accountFallback.ts::clearModelLock()`*

## How the Layers Interact in the Routing Pipeline

When OmniRoute evaluates a request, it applies the three layers in strict sequence:

1. **Circuit Breaker** – If the provider state is **OPEN**, the provider is omitted from combo candidates entirely
2. **Connection Cooldown** – For remaining providers, connections with future `rateLimitedUntil` timestamps are filtered out
3. **Model Lockout** – For surviving connections, currently locked models are excluded

Only if all connections of a provider are filtered does the combo engine fall back to the next provider or invoke emergency fallback logic. This hierarchical filtering ensures that model-specific issues don't trigger unnecessary provider-wide circuit breaks.

## Configuration Defaults and Tuning

Default settings are defined across several configuration files:

| Layer | Configuration File | Key Parameters |
|-------|-------------------|----------------|
| Circuit Breaker | [`src/lib/resilience/settings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/settings.ts) | Thresholds and timeouts per provider type |
| Connection Cooldown | [`src/lib/resilience/settings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/settings.ts) | `baseCooldownMs` (5s OAuth, 3s API-key) |
| Model Lockout | [`src/lib/resilience/modelLockoutSettings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/modelLockoutSettings.ts) | Enabled flag, error codes, duration parameters |

Model lockout defaults to `enabled: false` and must be explicitly activated in [`modelLockoutSettings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/modelLockoutSettings.ts) before the system creates per-model lockout entries.

## Debugging and Operational Commands

When troubleshooting why a provider is being skipped:

- **All keys for a provider are skipped** – Check the circuit breaker state in `domain_circuit_breakers` and each connection's `rateLimitedUntil`/`testStatus`
- **Only one key fails** – Investigate connection cooldown in [`src/sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/auth.ts); this indicates a credential-specific issue rather than a provider-wide outage
- **Only one model fails** – Search for lockout entries in [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts) using the provider-connection-model triple
- **State should have recovered but didn't** – Verify code is reading status via `getStatus()` or `canExecute()` helpers, which trigger lazy recovery. Reading raw database columns directly will not refresh expired entries.

## Summary

- **Provider Circuit Breaker** operates at the provider level with states CLOSED, DEGRADED, OPEN, and HALF_OPEN, using lazy recovery after `resetTimeout` expires
- **Connection Cooldown** manages per-credential back-off with exponential growth (`baseCooldownMs * 2ⁿ`), filtering unavailable connections during routing
- **Model Lockout** isolates per-model failures with dual recovery paths: timer expiration and success-decay via `decayModelFailureCount()`
- The routing pipeline evaluates layers sequentially: breaker first, then cooldown, then lockout
- Configuration files in `src/lib/resilience/` control thresholds, with different defaults for OAuth, API-key, and Local provider types

## Frequently Asked Questions

### How do I know which resilience layer is blocking my request?

Check the symptoms: if **all keys** for a provider are skipped, the **Circuit Breaker** is open; if **only one key** fails, it is a **Connection Cooldown**; if **only one model** fails on an otherwise working key, it is **Model Lockout**. Inspect `domain_circuit_breakers` for breaker states, `rateLimitedUntil` timestamps for cooldowns, and the model-cooldowns API for lockouts.

### What triggers the transition from DEGRADED to OPEN in the circuit breaker?

The breaker moves from **DEGRADED** to **OPEN** when the `failureCount` reaches the provider-specific open threshold (8 for OAuth, 12 for API-key). While DEGRADED, the provider still receives traffic but is marked as unstable; once OPEN, it is excluded from routing entirely until the `resetTimeout` triggers lazy recovery.

### Can I disable the model lockout layer?

Yes. Model lockout defaults to `enabled: false` in [`src/lib/resilience/modelLockoutSettings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/modelLockoutSettings.ts). When disabled, model-specific errors fall through to the Connection Cooldown or Circuit Breaker layers instead of creating per-model lockout entries.

### Why is my provider still marked as OPEN after the reset timeout passed?

OmniRoute uses **lazy recovery**: the breaker only checks the timeout when `getStatus()`, `canExecute()`, or `getRetryAfterMs()` is invoked. If you are checking raw database columns directly without calling these helper functions, the state machine will not transition to HALF_OPEN. Use the provided utility functions to ensure proper state evaluation.