# How OmniRoute Handles Circuit Breakers and Resilience: A Complete Technical Guide

> Discover how OmniRoute ensures resilience with its three-layer architecture: provider circuit breakers, connection cooldowns, and model lockouts. Learn about lazy recovery and database persistence for robust failure isolation.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: deep-dive
- Published: 2026-08-07

---

**OmniRoute implements a three-layer resilience architecture—provider circuit breakers, connection cooldowns, and model lockouts—with lazy recovery and database persistence to isolate failures at progressively finer scopes.**

OmniRoute is an open-source AI model routing proxy that combines multiple LLM providers behind a unified API. Its **circuit breaker and resilience** system prevents cascading failures when upstream providers (OpenAI, Anthropic, etc.) experience outages, rate limits, or quota exhaustion. The implementation spans [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts), [`src/sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/auth.ts), and multiple supporting modules that work together to maintain service availability.

## Three-Layer Resilience Architecture

OmniRoute organizes failure isolation into three distinct scopes, each with dedicated implementation files and state management strategies.

| Layer | Scope | Core File | Purpose |
|-------|-------|-----------|---------|
| **Provider Circuit Breaker** | Entire provider (e.g., `openai`, `anthropic`) | [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) | Blocks traffic to failing providers after repeated 5xx errors |
| **Connection Cooldown** | Individual API key / credential | [`src/sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/auth.ts) | Temporarily excludes bad credentials while keeping other keys active |
| **Model Lockout** | Provider + connection + model triple | [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts) | Prevents a single model from blocking an otherwise healthy connection |

All three layers share a **lazy-recovery philosophy**: state refreshes on demand during status checks rather than using background timers, reducing CPU overhead and complexity.

## Provider Circuit Breaker Implementation

The circuit breaker in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) provides the outermost protection layer. It tracks failure rates per provider and transitions through four states to balance availability against error propagation.

### State Machine Design

The breaker implements **CLOSED → DEGRADED → OPEN → HALF_OPEN** transitions:

- **CLOSED**: Normal operation; requests pass through
- **DEGRADED**: Elevated failure rate detected; requests still allowed but monitored
- **OPEN**: Provider blocked; all requests fail fast with `retryAfterMs` guidance
- **HALF_OPEN**: Probe mode; limited test requests allowed to verify recovery

```typescript
// src/shared/utils/circuitBreaker.ts – core state transition logic
import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';

const openAiBreaker = getCircuitBreaker('openai', {
  failureThreshold: 8,           // Transition to OPEN after 8 failures
  resetTimeout: 30_000,          // 30s base cooldown
  halfOpenRequests: 2,           // Allow 2 probe requests in HALF_OPEN
  maxBackoffMultiplier: 4,       // Cap exponential backoff at 4x
  
  // Classify errors: only upstream 5xx triggers breaker
  isFailure: (err) => err?.statusCode >= 500,
});

```

### Adaptive Back-Off and Persistence

The `_effectiveResetTimeout()` method implements **exponential back-off**: each `OPEN → HALF_OPEN → OPEN` cycle multiplies the reset timeout up to `maxBackoffMultiplier`. Breaker state persists to the `domain_circuit_breakers` table via `saveCircuitBreakerState()` and `loadCircuitBreakerState()`, ensuring recovery after process restarts.

A global `Map<string, CircuitBreaker>` registry caps at 500 entries, with idle CLOSED breakers automatically evicted via `evictColdBreakersIfNeeded()`.

### Integration Points

The breaker wires into the request pipeline through [`src/sse/handlers/chatHelpers.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/handlers/chatHelpers.ts) and [`src/sse/handlers/chat.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/handlers/chat.ts). Operational visibility comes from:

- `GET /api/monitoring/health` – exposes breaker states
- `POST /api/resilience/reset` – manual reset endpoint

```typescript
// Wrapping an upstream request with circuit breaker protection
async function fetchOpenAIChat(payload: any) {
  return openAiBreaker.execute(async () => {
    const resp = await fetch('https://api.openai.com/v1/chat/completions', {
      method: 'POST',
      headers: { Authorization: `Bearer ${process.env.OPENAI_API_KEY}` },
      body: JSON.stringify(payload),
    });
    if (!resp.ok) throw new Error(`HTTP ${resp.status}`);
    return resp.json();
  });
}

// Health check integration
const status = openAiBreaker.getStatus();
console.log('State:', status.state, 'Retry after:', status.retryAfterMs);

```

## Connection Cooldown: Per-Credential Isolation

When a specific API key hits rate limits or network errors, **connection cooldown** isolates that credential without affecting other keys for the same provider.

### Implementation in auth.ts

The `markAccountUnavailable()` function in [`src/sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/auth.ts) records failure metadata:

- `rateLimitedUntil`: Timestamp when the connection becomes eligible again
- `testStatus`: Current verification state of the credential
- `backoffLevel`: Exponential backoff iteration count

`checkFallbackError()` in [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts) evaluates these fields during request routing. The cooldown uses **lazy expiration**: connections are excluded while `rateLimitedUntil` is in the future, then automatically become eligible without timer cleanup.

### Back-Off Configuration

Default cooldown bases are defined in [`src/lib/resilience/settings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/settings.ts):

- **OAuth keys**: 5 seconds base
- **API keys**: 3 seconds base

Cooldown duration calculates as `baseCooldownMs × 2ⁿ` where *n* is the failure index.

```typescript
// Simplified connection cooldown usage
import { markAccountUnavailable } from '@/sse/services/auth';

async function handleRequest(connId: string, fn: () => Promise<any>) {
  try {
    return await fn();
  } catch (err) {
    if (isRateLimitOrNetworkError(err)) {
      await markAccountUnavailable(connId, err); // Triggers exponential backoff
    }
    throw err;
  }
}

```

## Model Lockout: Fine-Grained Quota Protection

The innermost resilience layer prevents individual models from degrading connection health. A model returning 404 (unknown model) or 429 (per-model quota) triggers lockout for the specific `provider:connectionId:model` triple.

### Lockout Mechanics

`lockModel()` and `clearModelLock()` in [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts) manage an in-memory `Map` of lockout entries. Configuration lives in [`src/lib/resilience/modelLockoutSettings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/modelLockoutSettings.ts):

```typescript
// DEFAULT_MODEL_LOCKOUT_SETTINGS structure
{
  enabled: boolean,
  errorCodes: number[],        // e.g., [404, 429]
  baseCooldownMs: number,
  maxBackoffMultiplier: number,
  failureThreshold: number,    // Lock after N failures
}

```

### Recovery via Decay

Unlike the provider breaker, model lockout includes **failure count decay**: successful calls increment a decay counter (`decayModelFailureCount`), allowing models to recover before the lockout timer expires. This prevents permanent exclusion of temporarily strained models.

Users can view and clear lockouts via **Settings → Model Lockouts** in the UI, or programmatically through the API.

```typescript
// Model lockout trigger on model-specific errors
import { lockModel, clearModelLock } from '@/open-sse/services/accountFallback';

async function requestModel(provider: string, connId: string, model: string, fn: () => Promise<any>) {
  try {
    return await fn();
  } catch (err) {
    if (lockoutSettings.errorCodes.includes(err.statusCode)) {
      await lockModel(provider, connId, model, err);
    }
    throw err;
  }
}

// Manual recovery
await clearModelLock(provider, connId, model);

```

## Per-Failure-Kind Thresholds

The provider circuit breaker supports granular failure classification through `kindThresholds` and `cooldownByKind`. Failures are categorized (e.g., `rate_limit`, `quota_exhausted`, `transient`) with distinct:

- Threshold counts for state transitions
- Immediate-open flags for critical error types
- Custom cooldown durations per kind

This prevents, for example, rate-limit 429s from opening the breaker (handled by connection cooldown instead) while treating 503 service-unavailable errors as breaker-triggering.

## Key Source Files and References

| Component | File Path | Purpose |
|-----------|-----------|---------|
| Circuit breaker core | [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) | State machine, persistence, registry |
| Resilience documentation | [`docs/architecture/RESILIENCE_GUIDE.md`](https://github.com/diegosouzapw/OmniRoute/blob/main/docs/architecture/RESILIENCE_GUIDE.md) | Architecture overview and configuration |
| Chat handler integration | [`src/sse/handlers/chatHelpers.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/handlers/chatHelpers.ts) | Request pipeline wiring |
| Connection cooldown | [`src/sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/auth.ts) | `markAccountUnavailable()` implementation |
| Model lockout logic | [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts) | `lockModel()`, `clearModelLock()` |
| Model lockout settings | [`src/lib/resilience/modelLockoutSettings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/modelLockoutSettings.ts) | `DEFAULT_MODEL_LOCKOUT_SETTINGS` |
| Health API | [`src/app/api/monitoring/health/route.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/app/api/monitoring/health/route.ts) | Breaker state exposure |
| Reset API | [`src/app/api/resilience/reset/route.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/app/api/resilience/reset/route.ts) | Manual breaker reset endpoint |

## Summary

OmniRoute's **circuit breaker and resilience** system provides defense in depth through three coordinated mechanisms:

- **Provider circuit breakers** ([`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts)) stop traffic to failing providers with a four-state machine, adaptive exponential back-off, and database persistence
- **Connection cooldowns** ([`src/sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/auth.ts)) isolate bad API keys using lazy timestamp-based exclusion without background timers
- **Model lockouts** ([`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts)) prevent individual models from blocking healthy connections, with decay-based recovery

All layers implement **lazy recovery**—state evaluation happens on demand during routing decisions, eliminating timer management overhead. The system exposes operational visibility through health endpoints and supports manual intervention via reset APIs and UI controls.

## Frequently Asked Questions

### What happens when a provider circuit breaker opens?

When the breaker opens, all new requests to that provider fail fast with a `retryAfterMs` value indicating when to retry. The breaker enters **HALF_OPEN** state after the reset timeout, allows a limited number of probe requests, and transitions back to **CLOSED** if they succeed or **OPEN** with increased back-off if they fail.

### How does OmniRoute distinguish between provider failures and model-specific errors?

The system uses **error classification** and **scope-based handling**. Provider-level 5xx errors trigger the circuit breaker in [`circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/circuitBreaker.ts). Connection-level 429s trigger cooldown in [`auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/auth.ts). Model-specific 404s or per-model 429s trigger lockout in [`accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/accountFallback.ts). The `isFailure` predicate and `kindThresholds` configuration control this routing.

### Can circuit breaker state survive process restarts?

**Yes.** The `saveCircuitBreakerState()` and `loadCircuitBreakerState()` functions persist breaker state to the `domain_circuit_breakers` database table. This ensures that accumulated failure history and back-off levels are not lost when the OmniRoute process restarts.

### What's the difference between DEGRADED and OPEN states?

**DEGRADED** is a warning state where elevated failure rates are detected but requests still flow through; it allows quick recovery without full blocking. **OPEN** is a hard stop where all requests fail immediately. The DEGRADED state provides early indication before escalating to OPEN, reducing unnecessary traffic disruption.