# OmniRoute's 3-Layer Resilience System: Circuit Breaker, Connection Cooldown, and Model Lockout

> Explore OmniRoute's 3-layer resilience system: circuit breaker, connection cooldown, and model lockout. Prevent AI routing pipeline outages. Learn how it works.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: deep-dive
- Published: 2026-07-08

---

**OmniRoute implements a three-layer resilience system that isolates failures at the provider, connection, and model levels to prevent cascading outages in AI routing pipelines.**

OmniRoute is an open-source routing pipeline that protects AI model integrations through hierarchical fault tolerance. The **OmniRoute's 3-layer resilience system** combines circuit breakers, connection cooldowns, and model lockouts to handle upstream provider failures without manual intervention. This architecture ensures that transient errors like rate limits or quota exhaustion do not cascade through the entire service.

## Layer 1: Provider-Wide Circuit Breaker

The first layer guards all requests routed through a specific provider using a state-machine-based circuit breaker implemented in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts).

**State Machine and Thresholds**

The circuit breaker tracks consecutive failures against a configurable `failureThreshold` (default **5**). It transitions through four states:

- **CLOSED**: Normal operation; requests flow through
- **DEGRADED**: Early warning state before full isolation
- **OPEN**: Circuit tripped; all requests short-circuit immediately
- **HALF_OPEN**: Probe period allowing limited test requests (`halfOpenRequests`)

When failures exceed the threshold, the breaker moves from **CLOSED** to **OPEN**, blocking further traffic to that provider. After the `resetTimeout` expires, it enters **HALF_OPEN** to test recovery. Successful probes close the circuit; failures reopen it.

```typescript
import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';

const breaker = getCircuitBreaker('openai', {
  failureThreshold: 5,
  resetTimeout: 30_000,      // 30 seconds
  halfOpenRequests: 1
});

async function callProvider() {
  if (!breaker.canExecute()) {
    throw new Error('Provider unavailable (circuit open)');
  }
  return breaker.execute(async () => {
    // HTTP request logic here
  });
}

```

## Layer 2: Connection Cooldown (Per-Failure-Kind Back-off)

The second layer implements **connection cooldown**—fine-grained back-off logic that applies different timeouts based on specific error types rather than treating all failures equally.

**Cooldown by Failure Kind**

Within the same [`circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/circuitBreaker.ts) file, the `cooldownByKind` map and `classifyError` logic allow each failure type (e.g., *rate-limit*, *quota-exhausted*, *transient*) to specify its own cooldown duration. The breaker selects the most restrictive timeout (`_effectiveCooldown`) for the last failure kind encountered.

This prevents "hammering" a provider when it returns HTTP 429 status codes or quota-exhausted errors. You can override the generic `resetTimeout` with specific durations for sensitive error types:

```typescript
const breaker = getCircuitBreaker('anthropic', {
  resetTimeout: 30_000,
  cooldownByKind: {
    rate_limit: 120_000,      // 2 minutes for rate limits
    quota_exhausted: 300_000  // 5 minutes for quota issues
  }
});

```

## Layer 3: Model-Level Lockout

The third layer provides granular protection at the individual model-connection level, implemented in [`src/domain/lockoutPolicy.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/domain/lockoutPolicy.ts) and consumed by the account fallback service at [`src/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/services/accountFallback.ts).

**Lockout Mechanics**

Model lockout tracks failures per identifier using a sliding window (`attemptWindowMs`). When `maxAttempts` is exceeded within the window, a **lockout** record is persisted to the `domain_lockout_state` table with a configurable `lockoutDurationMs`.

While locked, the specific model-connection pair (e.g., *Gemini-1.5-flash* on account *conn-123*) is excluded from combo resolution. The lockout clears automatically after the duration expires, or immediately upon `recordSuccess` or explicit `forceUnlock`.

```typescript
import { markAccountUnavailable } from '@/services/accountFallback';

// Lock out a specific model after repeated 429 errors
await markAccountUnavailable({
  provider: 'gemini',
  connectionId: 'conn-123',
  model: 'gemini-1.5-flash',
  reason: 'rate_limit'
});

// The lockout is automatically consulted during combo resolution
const combo = await resolveComboTargets(...); // Excludes locked models

```

## How the Three Layers Interact

The **OmniRoute's 3-layer resilience system** operates hierarchically to minimize blast radius:

1. **Circuit-breaker** acts as the first gate. If the breaker is **OPEN**, the provider is excluded entirely from combo resolution.
2. **Connection-cooldown** applies when the breaker is **CLOSED** or **DEGRADED**, delaying the next request based on the specific failure kind via `cooldownByKind`.
3. **Model-lockout** provides the final safety net, isolating individual misbehaving models even when their parent provider is healthy.

This hierarchy ensures that systemic outages trigger provider-wide protection, while transient spikes affect only specific error types, and chronic model issues result in targeted lockouts.

## Implementation Reference

**Key Files and Functions**

- **[`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts)**: Core circuit breaker implementation with `getCircuitBreaker()`, `canExecute()`, and `execute()` methods
- **[`src/domain/lockoutPolicy.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/domain/lockoutPolicy.ts)**: Model lockout logic with `recordFailure()`, `recordSuccess()`, and `forceUnlock()` functions
- **[`src/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/services/accountFallback.ts)**: Routing logic that consults lockout state via `markAccountUnavailable()`
- **[`src/lib/db/domainState.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/db/domainState.ts)**: SQLite persistence for circuit breaker states
- **[`src/lib/db/lockoutState.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/db/lockoutState.ts)**: SQLite persistence for model lockout records

**Monitoring Resilience State**

To inspect current system health for dashboards or debugging:

```typescript
import { getAllCircuitBreakerStatuses } from '@/shared/utils/circuitBreaker';
import { getAllModelLockouts } from '@/domain/lockoutPolicy';

console.log('Provider breakers:', getAllCircuitBreakerStatuses());
console.log('Model lockouts:', await getAllModelLockouts());

```

## Summary

- **Provider-Wide Circuit Breaker**: Tracks consecutive failures (default threshold of 5) across a provider, transitioning through CLOSED → DEGRADED → OPEN → HALF_OPEN states to prevent cascading failures
- **Connection Cooldown**: Implements per-failure-kind back-off via `cooldownByKind` configuration, applying specific timeouts for rate limits versus quota exhaustion
- **Model Lockout**: Isolates individual model-connection pairs using sliding window counting (`maxAttempts` within `attemptWindowMs`) stored in `domain_lockout_state`
- **Hierarchical Protection**: Layers operate sequentially—circuit breakers protect providers, cooldowns throttle specific error types, and lockouts target individual models

## Frequently Asked Questions

### What triggers the circuit breaker to open?

The circuit breaker opens when consecutive failures exceed the `failureThreshold` parameter (default **5**), as tracked in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts). Once opened, it remains in the **OPEN** state for the duration specified by `resetTimeout` before attempting recovery via the **HALF_OPEN** state.

### How does connection cooldown differ from the circuit breaker?

While the circuit breaker operates at the provider level and uses a binary open/closed state, **connection cooldown** provides granular back-off per failure type. The `cooldownByKind` map in [`circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/circuitBreaker.ts) allows different timeouts for specific errors (e.g., 120 seconds for rate limits versus 30 seconds for generic timeouts), whereas the circuit breaker uses a single `resetTimeout` for all failures.

### Can you manually unlock a model before the lockout expires?

Yes. According to the implementation in [`src/domain/lockoutPolicy.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/domain/lockoutPolicy.ts), you can clear a lockout immediately by calling `recordSuccess()` after a successful request, or by invoking `forceUnlock()` to explicitly remove the restriction. Otherwise, the lockout automatically expires after `lockoutDurationMs`.

### How long do model lockouts typically last?

The duration is configurable per provider via `lockoutDurationMs` in the lockout policy configuration. The default values depend on the provider's reliability profile, but the system stores these durations in [`src/lib/db/lockoutState.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/db/lockoutState.ts) and consults them during the combo resolution phase in [`src/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/services/accountFallback.ts).