# How OmniRoute Implements Its 3-Layer Resilience System: Circuit Breakers, Cooldown, and Lockout

> Learn how OmniRoute's 3-layer resilience system uses circuit breakers, cooldown, and lockout to prevent failures and ensure LLM service availability.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: architecture
- Published: 2026-08-06

---

**OmniRoute protects LLM request routing with three coordinated resilience layers—circuit breakers at the provider level, cooldown at the connection level, and lockout at the model level—to prevent cascading failures and maintain service availability.**

OmniRoute's **3-layer resilience system** ensures robust request processing across multiple AI providers. By combining **circuit breakers** for provider-level protection, **connection cooldown** for credential-level isolation, and **model lockout** for fine-grained error handling, the system gracefully degrades only the problematic components while keeping healthy paths open. This implementation is live in the `diegosouzapw/OmniRoute` repository across TypeScript services in `src/` and `open-sse/` directories.

---

## Architecture Overview: Three Resilience Layers

The resilience strategy operates at progressively narrower scopes, allowing precise failure containment:

| Layer | Scope | Trigger | Recovery |
|-------|-------|---------|----------|
| **Provider Circuit Breaker** | Entire provider (e.g., `openai`, `anthropic`) | Repeated upstream failures | Timeout-based reset with HALF_OPEN probing |
| **Connection Cooldown** | Individual credential/account/key | Rate limits, transient errors | Exponential backoff per-connection |
| **Model Lockout** | Specific model on a connection | Model-specific quota errors | Manual or policy-based clearance |

Each layer consults the next in sequence during request routing, creating a defense-in-depth pattern.

---

## Layer 1: Provider Circuit Breaker

The **circuit breaker** stops all traffic to a provider that repeatedly fails at the upstream level, preventing a single unhealthy provider from throttling the entire service.

### State Machine Implementation

In [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts), the `CircuitBreaker` class maintains three states:

- **CLOSED**: Normal operation, requests allowed
- **OPEN**: Provider failing, requests rejected immediately
- **HALF_OPEN**: Probe request allowed to test recovery

```ts
// src/shared/utils/circuitBreaker.ts (excerpt)
const DEFAULTS = {
  oauth: { threshold: 3, resetTimeoutMs: 60_000 },
  apiKey: { threshold: 5, resetTimeoutMs: 30_000 },
};

```

The `oauth` threshold (3 failures) is stricter than `apiKey` (5 failures) because OAuth-based providers typically indicate more systemic issues when they fail.

### Integration Points

The breaker is consulted at two critical execution points:

- [`open-sse/handlers/chatHelpers.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/handlers/chatHelpers.ts) – Validates provider health before request execution
- [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts) – Checks `canExecute()` and `getRetryAfterMs()` during fallback decisions

### Lazy Recovery Mechanism

OmniRoute uses **lazy recovery**: the breaker doesn't use timers to transition states. Instead, `getStatus()` automatically flips **OPEN** → **HALF_OPEN** when `resetTimeoutMs` has elapsed since the last failure. A single successful probe transitions to **CLOSED**; any failure reopens the circuit.

```ts
// Example: manually checking a provider before sending a request
import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';
import { Provider } from '@/shared/constants/providers';

async function safeExecute(provider: Provider, exec: () => Promise<any>) {
  const cb = getCircuitBreaker(provider);
  if (!cb.canExecute()) {
    throw new Error(`Provider ${provider} is circuit‑open`);
  }

  try {
    const result = await exec();
    cb.recordSuccess();        // resets failure count
    return result;
  } catch (err) {
    cb.recordFailure(err);     // may transition to OPEN
    throw err;
  }
}

```

---

## Layer 2: Connection Cooldown

While the circuit breaker handles provider-wide failures, **connection cooldown** isolates individual bad credentials without disabling the entire provider. This allows other accounts or API keys for the same provider to continue serving traffic.

### Cooldown State Management

In [`src/sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/auth.ts), the `markAccountUnavailable()` function implements exponential backoff:

```ts
// src/sse/services/auth.ts (excerpt)
function markAccountUnavailable(account, error) {
  const backoff = baseCooldownMs * 2 ** account.backoffLevel;
  account.rateLimitedUntil = new Date(Date.now() + backoff);
  account.backoffLevel++;
}

```

The backoff multiplier (`2 ** failureIndex`) ensures that repeatedly failing connections are sidelined for progressively longer periods, starting from `baseCooldownMs`.

### Credential Selection Logic

The `selectAvailableCredentials()` function (also in [`auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/auth.ts)) filters out credentials where `rateLimitedUntil` is still in the future. This check is invoked by `open-sse/services/accountFallback.ts::checkFallbackError()` when evaluating fallback candidates.

```ts
// Example: using the connection selector that respects cooldowns
import { selectAvailableCredentials } from '@/sse/services/auth';

async function pickCredential(provider: string) {
  const creds = await selectAvailableCredentials(provider);
  if (!creds.length) {
    throw new Error(`All accounts for ${provider} are on cooldown`);
  }
  return creds[0]; // simple first‑available strategy
}

```

---

## Layer 3: Model Lockout

The finest-grained layer, **model lockout**, isolates a single unavailable or quota-limited model while leaving other models on the same connection operational. This handles per-model rate limits common in LLM APIs (e.g., GPT-4 vs. GPT-3.5 on the same OpenAI organization).

### Implementation in Account Fallback Service

In [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts), the `checkFallbackError()` function distinguishes between connection-level and model-specific errors:

```ts
// open-sse/services/accountFallback.ts (excerpt)
if (isTransientProviderError(err)) {
  // apply connection cooldown
  markAccountUnavailable(conn, err);
} else if (isModelSpecificError(err)) {
  // apply model lockout
  conn.modelLockout[modelId] = true;
}

```

### Error Classification Logic

- **HTTP 429 with model-specific indicator** → **Model lockout** (e.g., quota exhausted for `gpt-4-turbo` only)
- **HTTP 500-599 or network timeouts** → **Connection cooldown** (affects the entire credential)

```ts
// Example: handling a model‑specific error to trigger lockout
import { handleModelError } from '@/open-sse/services/accountFallback';

function onModelResponse(err: any, modelId: string, conn: any) {
  if (err.code === 429 && err.isModelSpecific) {
    conn.modelLockout[modelId] = true; // lockout this model only
  } else {
    markAccountUnavailable(conn, err); // fallback to connection cooldown
  }
}

```

---

## Request Flow: How Layers Interact

The [`open-sse/services/combo.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/combo.ts) router orchestrates all three resilience checks:

1. **Request validation** → Zod schema, optional auth, policy checks
2. **Target resolution** → `resolveComboTargets()` enumerates candidate providers
3. **Circuit breaker check** → `handleSingleModel()` calls `circuitBreaker.canExecute()`; **OPEN** providers are skipped
4. **Credential filtering** → Connection selector excludes `rateLimitedUntil` timestamps in the future
5. **Execution & error handling** → On failure, `accountFallback.checkFallbackError()` applies **cooldown** or **lockout** based on error classification
6. **Retry with fallback** → Router proceeds to next candidate if current fails

This sequential validation ensures that no request reaches a known-bad provider, credential, or model.

---

## Configuration Constants

Default thresholds are centralized in [`open-sse/config/constants.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/config/constants.ts):

| Parameter | OAuth Value | API Key Value | Rationale |
|-----------|-------------|---------------|-----------|
| Failure threshold | 3 | 5 | OAuth failures indicate broader issues |
| Reset timeout | 60,000 ms | 30,000 ms | OAuth providers recover more slowly |

These defaults can be overridden per-provider via environment configuration.

---

## Summary

OmniRoute's **3-layer resilience system** provides comprehensive failure isolation:

- **Provider circuit breakers** prevent cascading outages by blocking traffic to failing upstreams, with configurable thresholds and lazy recovery
- **Connection cooldown** implements exponential backoff for individual credentials, preserving provider capacity through alternative accounts
- **Model lockout** enables per-model isolation without connection disruption, critical for multi-model provider integrations

Together, these mechanisms ensure that OmniRoute routes around failures at the appropriate granularity—provider, credential, or model—maintaining service availability even during partial upstream outages.

---

## Frequently Asked Questions

### What triggers a circuit breaker to open in OmniRoute?

A circuit breaker opens when failure count exceeds the configured threshold (3 for OAuth, 5 for API keys) within a single operational window. Failures are recorded via `recordFailure()` in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts). The breaker stays open until `resetTimeoutMs` elapses, then transitions to HALF_OPEN on the next status check.

### How does connection cooldown differ from circuit breaker state?

**Circuit breaker** operates at the provider level and uses fixed timeouts (30-60 seconds). **Connection cooldown** operates per-credential with exponential backoff starting from `baseCooldownMs`. A provider can have its circuit closed while individual connections are on cooldown, allowing other credentials for that provider to serve requests.

### Can a model be locked out while its connection remains active?

Yes. Model lockout in [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts) marks only `conn.modelLockout[modelId] = true`, leaving the connection and other models available. This commonly occurs when a specific model hits quota limits (HTTP 429) while the underlying credential remains valid for other models.

### What happens when all resilience layers have excluded all candidates?

If no providers pass circuit breaker checks, no credentials escape cooldown, and no models avoid lockout, the router returns an aggregated error indicating exhaustion of all fallback paths. This is surfaced in [`open-sse/services/combo.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/combo.ts) when `resolveComboTargets()` yields an empty candidate set.