# OmniRoute Built-in Resilience Features: A 3-Layer Protection System Explained

> Discover OmniRoute's 3-layer protection system: circuit breakers, cooldowns, and lockouts safeguard your requests from unreliable providers and ensure robust operation.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: deep-dive
- Published: 2026-08-25

---

**OmniRoute protects request processing from flaky upstream providers through a provider-level circuit breaker, connection-level cooldowns, and model-level lockouts, plus additional runtime guards for quota control, queue admission, stream health, and status correction.**

The OmniRoute proxy—maintained by diegosouzapw/OmniRoute—implements a sophisticated **three-layer resilience stack** that isolates failures at different granularities. Each layer operates independently, ensuring that a problem with one provider, credential, or model does not cascade into system-wide degradation. Understanding these built-in resilience features of OmniRoute helps operators configure effective failure boundaries and debug issues faster.

## Provider Circuit Breaker: Stopping Catastrophic Outages

The **provider circuit breaker** in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) halts traffic to entire upstream providers (e.g., `openai`, `anthropic`) when they return repeated 5xx errors.

### State Machine and Transitions

The breaker implements a five-state lifecycle:

- `CLOSED` → normal operation
- `DEGRADED` → elevated failure rate threshold reached
- `OPEN` → circuit tripped, requests blocked
- `HALF_OPEN` → probe request allowed after timeout
- `CLOSED` → recovery confirmed

Lazy recovery triggers automatically: when `OPEN` timeout expires, calls to `getStatus()`, `canExecute()`, or `getRetryAfterMs()` transition the breaker to `HALF_OPEN` and admit a single probe request.

### Trigger Conditions and Scope

Only upstream 5xx errors (`408, 500, 502, 503, 504`) open the breaker. Credential-level failures (`401`, `403`, `429`) bypass this layer and are handled by connection cooldowns instead.

```ts
// Core implementation – src/shared/utils/circuitBreaker.ts
export class CircuitBreaker {
  private state: BreakerState = BreakerState.CLOSED;
  private failureCount = 0;
  private lastFailureTime?: number;

  // State transitions (lines 70-84, 110-135)
  // Lazy recovery via getStatus() / canExecute() / getRetryAfterMs()
}

```

### Global Registry Management

The `getCircuitBreaker(name)` function lazily initializes breakers and evicts cold, closed entries when the registry exceeds `MAX_REGISTRY_SIZE` (lines 45-52). This prevents memory leaks in long-running deployments.

## Connection Cooldown: Isolating Bad Credentials

While the circuit breaker protects at provider scope, **connection cooldown** targets individual credentials. When a specific API key hits rate limits or transient failures, OmniRoute marks it unavailable without affecting other keys for the same provider.

### Implementation in auth.ts

The `markAccountUnavailable` function in [`src/sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/auth.ts) updates `provider_connections` with cooldown metadata:

```ts
// Mark connection unavailable – src/sse/services/auth.ts
export async function markAccountUnavailable(
  connectionId: string,
  errorContext: ErrorContext
): Promise<void> {
  const cooldownMs = calculateBackoff(failureIndex);
  await updateProviderConnection(connectionId, {
    rateLimitedUntil: new Date(Date.now() + cooldownMs).toISOString(),
    testStatus: "unavailable",
    lastError: errorContext.message,
    backoffLevel: failureIndex,
  });
}

```

The selector checks `new Date(rateLimitedUntil).getTime() > Date.now()` when choosing connections (lines 64-70).

### Exponential Back-off with Thundering Herd Protection

Cooldown duration follows `baseCooldownMs * 2 ** failureIndex`. A synchronization guard prevents concurrent failures from double-incrementing `backoffLevel`, ensuring predictable recovery times even under high concurrency.

### Fallback Integration

The `checkFallbackError` function in [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts) coordinates cooldown triggers with the broader fallback system, ensuring rapid failover to healthy connections.

## Model Lockout: Granular Quota Isolation

**Model lockout** provides the finest-grained failure isolation: when a specific model (e.g., `gpt-4-turbo`) exhausts quota on a connection, other models on that same credential remain operational.

### Scope and Keying

Lockouts are keyed by `${provider}:${connectionId}:${model}` triple. This prevents over-quota scenarios on one model from disabling an entire connection with multiple model access.

### Implementation in accountFallback.ts

```ts
// Lock a failing model – open-sse/services/accountFallback.ts
function lockModel(connectionId: string, model: string, reason: string): void {
  const key = `${provider}:${connectionId}:${model}`;
  const record = modelLockoutMap.get(key) || { failureCount: 0, lockedAt: null };
  record.failureCount++;
  if (record.failureCount >= MODEL_LOCKOUT_THRESHOLD) {
    record.lockedAt = Date.now();
  }
  modelLockoutMap.set(key, record);
}

```

### Success-Decay Recovery

Unlike time-based expiration alone, successful requests automatically halve the stored `failureCount`. This allows organic recovery before the hard lockout timer expires, reducing false positives for intermittently flaky models.

## Quota-Share Concurrency Control

Shared accounts—common with providers like GLM and MiniMax—require serialized access to prevent parallel requests from exhausting quota instantly.

### Configuration and Runtime

Operators set `provider_connections.max_concurrent` via Dashboard → Settings → Resilience. At runtime, a per-connection semaphore (`qsconn:<connectionId>`) enforces the limit:

- Positive `max_concurrent`: requests queue and execute serially
- Zero or null: no concurrency restriction applies

The implementation spans [`src/lib/resilience/settings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/settings.ts) for configuration and [`open-sse/services/combo.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/combo.ts) for semaphore acquisition.

## Request Queue Admission Control

OmniRoute rejects overload before expensive work begins. The **admission control** layer in [`open-sse/services/rateLimitManager/admission.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/rateLimitManager/admission.ts) monitors Bottleneck queue depth against `resilienceSettings.requestQueue.maxQueueDepth`.

### Fast-Fail Behavior

When `queueLength >= maxQueueDepth`, the system throws immediately:

```ts
// Admission check – open-sse/services/rateLimitManager/admission.ts
if (queueLength >= maxQueueDepth) {
  throw new RateLimitError(
    "Request queue at capacity",
    "RATE_LIMIT_QUEUE_FULL"
  );
}

```

This occurs **before** translation, compression, or upstream connection attempts—conserving resources during traffic spikes. Default `maxQueueDepth` is 0 (disabled); `maxWaitMs` defaults to 15000ms.

## Slow-Stream Throughput Watchdog

Streaming responses can stall indefinitely without closing the connection. The **throughput watchdog** in [`open-sse/services/streamRecovery.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/streamRecovery.ts) monitors byte rate after a configurable warm-up period.

### Activation and Measurement

Controlled by `resilienceSettings.streamRecovery.throughputWatchdog` (default disabled). Post warm-up, measured delta throughput below the configured minimum triggers stream abortion, freeing resources and allowing fallback to alternative providers.

## Upstream Status Restatement

Some gateways misreport temporary quota exhaustion as `403 Forbidden` instead of `429 Too Many Requests`. OmniRoute corrects these before resilience classification.

### Rule-Based Rewriting

[`open-sse/config/upstreamStatusRestatement.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/config/upstreamStatusRestatement.ts) defines pattern-matching rules that rewrite matched responses to `429`. This occurs in [`chatCore.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/chatCore.ts) before fallback logic executes, ensuring cooldown and lockout paths handle the error correctly.

## Debugging and Operational Hooks

OmniRoute exposes resilience state for monitoring and manual intervention:

| Endpoint/Purpose | Location |
|------------------|----------|
| Health check all breakers | `GET /api/monitoring/health` in [`src/sse/handlers/chatHelpers.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/handlers/chatHelpers.ts) |
| Reset all breaker states | `POST /api/resilience/reset` → `resetAllCircuitBreakers()` in [`circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/circuitBreaker.ts) |
| Dashboard configuration | Settings panels for all three core mechanisms |

### Query Circuit Breaker Health Programmatically

```ts
import { getAllCircuitBreakerStatuses } from "@/shared/utils/circuitBreaker";

async function auditResilience(): Promise<void> {
  const statuses = await getAllCircuitBreakerStatuses();
  console.table(statuses.map(s => ({
    provider: s.name,
    state: s.state,
    failures: s.failureCount,
    retryAfter: s.retryAfterMs,
  })));
}

```

Source: `getAllCircuitBreakerStatuses()` at lines 621-627 in [`circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/circuitBreaker.ts).

### Manual Breaker Reset

```ts
import { getCircuitBreaker } from "@/shared/utils/circuitBreaker";

async function resetProvider(name: string): Promise<void> {
  const breaker = getCircuitBreaker(name);
  breaker.reset(); // Clears state and persistent DB entry
}

```

Source: `reset()` method at lines 401-410.

### Force Connection Cooldown After 429

```ts
import { markAccountUnavailable } from "@/sse/services/auth";

async function applyCooldown(
  connId: string,
  retryAfterSec: number
): Promise<void> {
  await markAccountUnavailable(connId, {
    message: "Manual cooldown after 429",
    retryAfter: retryAfterSec,
  });
}

```

Source: `markAccountUnavailable` at lines 380-390 in [`auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/auth.ts).

### Programmatic Model Lockout

```ts
import { lockModel } from "@/sse/services/accountFallback";

function isolateFailingModel(connId: string, model: string): void {
  lockModel(connId, model, "operator_triggered_lockout");
}

```

Source: `lockModel` in [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts).

## Summary

OmniRoute's built-in resilience features operate through composable, layered mechanisms:

- **Provider circuit breaker** ([`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts)) stops traffic to failing upstreams at the 5xx level
- **Connection cooldown** ([`src/sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/auth.ts)) isolates individual credentials with exponential back-off
- **Model lockout** ([`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts)) preserves connection usability when only specific models fail
- **Quota-share concurrency control** serializes access to shared accounts
- **Queue admission control** ([`open-sse/services/rateLimitManager/admission.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/rateLimitManager/admission.ts)) rejects overload before expensive processing
- **Slow-stream watchdog** ([`open-sse/services/streamRecovery.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/streamRecovery.ts)) aborts stalled SSE connections
- **Status restatement** ([`open-sse/config/upstreamStatusRestatement.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/config/upstreamStatusRestatement.ts)) corrects misreported errors for proper classification

All layers are configurable via environment, settings files, or the Dashboard UI, with programmatic hooks for health monitoring and manual state reset.

## Frequently Asked Questions

### What triggers an OmniRoute circuit breaker to open?

Only upstream 5xx HTTP status codes (`408, 500, 502, 503, 504`) increment the failure counter toward `OPEN` state. Client errors like `401`, `403`, and `429` are intentionally excluded—these are handled by connection cooldown and model lockout layers instead, preserving granular failure isolation.

### How does OmniRoute handle rate limits differently from circuit breaker trips?

Rate limits (`429`) trigger **connection cooldown**, not circuit breaker opening. A single API key enters exponential back-off while other keys for the same provider remain active. This per-credential isolation prevents one exhausted quota from disabling an entire provider endpoint.

### Can operators manually reset circuit breaker states?

Yes. The `POST /api/resilience/reset` endpoint clears all breaker states, or use `getCircuitBreaker(name).reset()` programmatically to target a specific provider. This is useful after confirmed upstream recovery or during incident response drills.

### What happens when a model-specific quota is exhausted?

OmniRoute applies **model lockout** for the `provider:connection:model` triple. Other models on the same credential continue serving requests. The lockout includes success-decay recovery—successful requests on other models reduce the failure count organically before the timer expires.