# OmniRoute 3-Layer Resilience System: Circuit Breakers, Connection Cooldown, and Model Lockout

> Discover OmniRoute's 3-layer resilience system: circuit breakers, connection cooldown, and model lockout. Prevent cascading outages and isolate upstream failures effectively.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: architecture
- Published: 2026-07-05

---

**OmniRoute implements a hierarchical 3-layer resilience system using provider-wide circuit breakers, per-failure-kind connection cooldowns, and model-level lockouts to isolate upstream failures and prevent cascading outages.**

OmniRoute is an open-source LLM routing framework that maintains high availability through granular fault isolation. The 3-layer resilience system operates at distinct scopes—entire providers, specific error types, and individual model connections—to ensure that transient failures or rate limits do not compromise the entire routing pipeline. According to the diegosouzapw/OmniRoute source code, these protective mechanisms are implemented across [`circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/circuitBreaker.ts), [`lockoutPolicy.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/lockoutPolicy.ts), and the account-fallback service.

## Layer 1: Provider-Wide Circuit Breaker

The first line of defense guards all traffic destined for a specific provider (e.g., OpenAI, Gemini) regardless of which model is requested. This layer is implemented in [[`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts)](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) and tracks consecutive failures using a deterministic state machine.

The breaker maintains four states with the following transitions:

- **CLOSED** → **DEGRADED** → **OPEN** → **HALF_OPEN** → **CLOSED**

When the failure count exceeds the configurable `failureThreshold` (default 5), the breaker transitions to **OPEN**, immediately short-circuiting further requests to that provider. After the `resetTimeout` duration elapses, the state shifts to **HALF_OPEN**, allowing a limited number of probe requests defined by `halfOpenRequests`. Successful probes restore the **CLOSED** state, while failed probes reopen the circuit. This prevents "thundering herd" scenarios during upstream outages.

## Layer 2: Connection Cooldown (Per-Failure-Kind Back-Off)

The second layer provides fine-grained throttling within the same `CircuitBreaker` class by applying **per-failure-kind cooldowns** rather than uniform delays. This prevents aggressive retry loops when a provider returns specific error types like rate limits or quota exhaustion.

The implementation uses a `cooldownByKind` map and `classifyError` logic to categorize failures (e.g., `rate_limit`, `quota_exhausted`, `transient`). Each kind can override the generic `resetTimeout` with a custom duration. The breaker automatically selects the most restrictive timeout via `_effectiveCooldown` based on the last failure’s classification. For example, a generic 30-second timeout might extend to 120 seconds specifically for HTTP 429 responses, ensuring respectful back-off behavior without penalizing the entire provider for transient network blips.

## Layer 3: Model-Level Lockout

The third layer isolates specific **model-connection pairs** (e.g., *gemini-1.5-flash* on a specific account) that exhibit repeated failures, without disabling the entire provider. This granularity is handled by [[`src/domain/lockoutPolicy.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/domain/lockoutPolicy.ts)](https://github.com/diegosouzapw/OmniRoute/blob/main/src/domain/lockoutPolicy.ts) and consumed by the account-fallback service at [[`src/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/services/accountFallback.ts)](https://github.com/diegosouzapw/OmniRoute/blob/main/src/services/accountFallback.ts).

The lockout mechanism counts failures per unique identifier within a sliding `attemptWindowMs`. When failures exceed `maxAttempts`, a lockout record is persisted to the `domain_lockout_state` table for the configured `lockoutDurationMs`. While locked, the model is excluded from combo resolution and routing selection. The lockout clears automatically upon a successful request via `recordSuccess` or manually through `forceUnlock`. This ensures that a single misbehaving model cannot degrade the entire provider’s circuit while preserving other healthy endpoints on the same account.

## How the Resilience Layers Interact

The three layers operate hierarchically to provide defense in depth:

1. **Circuit-Breaker Evaluation**: Before any request executes, the system checks if the provider’s circuit is **OPEN**. If so, the provider is excluded entirely from combo resolution, failing fast to alternatives.

2. **Cooldown Application**: When the circuit is **CLOSED** or **DEGRADED**, the connection cooldown layer may still impose a delay. The next request to that provider is held according to the `_effectiveCooldown` derived from the last error classification, preventing hammering during rate-limit scenarios.

3. **Lockout Consultation**: Even when a provider is healthy, the routing logic consults the lockout policy via the account-fallback service. If a specific model-connection pair is locked, it is skipped during target resolution, ensuring requests route only to viable endpoints.

Together, these layers provide **provider-wide** protection against systemic outages, **failure-type** throttling for respectful retry behavior, and **model-specific** isolation for granular fault containment.

## Implementation Examples

### Configuring a Circuit Breaker with Per-Kind Cooldowns

```typescript
import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';

// Initialize breaker for OpenAI with custom rate-limit handling
const breaker = getCircuitBreaker('openai', {
  failureThreshold: 5,
  resetTimeout: 30_000,      // 30 seconds base cooldown
  halfOpenRequests: 1,
  // Aggressive back-off for rate limits specifically
  cooldownByKind: { rate_limit: 120_000 },
});

async function callOpenAI(payload: any) {
  if (!breaker.canExecute()) {
    throw new Error('OpenAI is currently unavailable (circuit open)');
  }

  return breaker.execute(async () => {
    // Actual HTTP fetch implementation
    return fetch('https://api.openai.com/v1/chat/completions', {
      method: 'POST',
      body: JSON.stringify(payload),
    });
  });
}

```

### Applying Model-Level Lockouts

```typescript
import { markAccountUnavailable } from '@/services/accountFallback';

// Trigger lockout after repeated 429 errors for a specific model
await markAccountUnavailable({
  provider: 'gemini',
  connectionId: 'conn-123',
  model: 'gemini-1.5-flash',
  reason: 'rate_limit',
});

// Subsequent combo resolution automatically skips locked models
const combo = await resolveComboTargets(...);

```

### Monitoring Resilience State

```typescript
import { getAllCircuitBreakerStatuses } from '@/shared/utils/circuitBreaker';
import { getAllModelLockouts } from '@/domain/lockoutPolicy';

// Export current health for dashboards or alerting
console.log('Provider breakers:', getAllCircuitBreakerStatuses());
console.log('Model lockouts:', await getAllModelLockouts());

```

## Summary

- **Provider-Wide Circuit Breaker**: Implements a state machine (CLOSED → DEGRADED → OPEN → HALF_OPEN) in [`circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/circuitBreaker.ts) to cut off all traffic to failing providers after a configurable failure threshold.
- **Connection Cooldown**: Uses the `cooldownByKind` map within the same breaker to apply specific back-off durations for different error classifications, preventing aggressive retries during rate limits.
- **Model-Level Lockout**: Persists failure counts per model-connection pair via [`lockoutPolicy.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/lockoutPolicy.ts), temporarily excluding specific endpoints from routing while keeping the provider circuit healthy.
- **Hierarchical Defense**: The layers cascade from broad provider exclusion to specific error throttling to granular model isolation, ensuring high availability across the OmniRoute pipeline.

## Frequently Asked Questions

### What causes the circuit breaker to transition from CLOSED to OPEN?

The breaker opens when consecutive failures exceed the `failureThreshold` parameter (default 5). Each failed request increments an internal counter; when the threshold is crossed, the state becomes **OPEN** and remains so for the `resetTimeout` duration before entering **HALF_OPEN** mode to test recovery.

### How does connection cooldown differ from the circuit breaker timeout?

While the circuit breaker’s `resetTimeout` defines how long the entire provider remains unavailable after an open event, **connection cooldown** operates within closed circuits to delay specific request types. The `cooldownByKind` configuration allows longer back-offs for rate limits (e.g., 120 seconds) compared to transient errors (e.g., 5 seconds), whereas the breaker itself is binary—either allowing or blocking all traffic.

### Can a specific model be locked out while its provider’s circuit remains closed?

Yes. The model-level lockout in [`lockoutPolicy.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/lockoutPolicy.ts) functions independently of the circuit breaker. A model can accumulate failures and become locked via `markAccountUnavailable` while the provider-wide circuit in [`circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/circuitBreaker.ts) remains **CLOSED**, allowing other models on the same provider to continue serving requests without interruption.

### Where is resilience state persisted across restarts?

OmniRoute persists circuit-breaker state to SQLite via [[`src/lib/db/domainState.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/db/domainState.ts)](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/db/domainState.ts) and model lockout records via [[`src/lib/db/lockoutState.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/db/lockoutState.ts)](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/db/lockoutState.ts). This ensures that failure counts, cooldown timers, and lockout durations survive application restarts and maintain consistent protection across deployments.