# OmniRoute 3-Layer Resilience System Architecture: Provider, Connection & Model Protection Explained

> Explore OmniRoute's 3-layer resilience architecture. Discover provider circuit breakers, connection cooldowns, and model lockouts for uninterrupted AI service availability.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: architecture
- Published: 2026-08-05

---

**OmniRoute implements a stacked, self-healing resilience pipeline that isolates failures at three distinct scopes—provider-level circuit-breakers, per-connection cooldowns, and model-specific lockouts—to ensure continuous availability across upstream AI service disruptions.**

This article examines the architecture of OmniRoute's 3-layer resilience system as implemented in the `diegosouzapw/OmniRoute` repository (release v3.8.50). The design protects every request through progressive failure isolation, from entire providers down to individual model endpoints.

---

## Overview of the Three Protection Layers

The resilience system operates as independent, composable layers that can trigger simultaneously without interference:

| Layer | Scope | Failure Type | Source Location |
|-------|-------|--------------|-----------------|
| **Layer 1: Provider Circuit-Breaker** | Entire provider (e.g., `openai`, `anthropic`) | Repeated 5xx failures, systemic outages | [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) |
| **Layer 2: Connection Cool-down** | Individual connection/account/key (OAuth, API key) | Transient rate limits (429), per-key quotas | [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts) |
| **Layer 3: Model Lockout** | Provider + connection + model triple | Model-specific 404s, quota exhaustion, mode denials | [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts) |

Each layer maintains separate state machinery, allowing granular recovery: a provider can be `OPEN` (skipped entirely) while a specific connection on that same provider merely enters cooldown, and a single model can be locked without affecting sibling models on the same connection.

---

## Layer 1: Provider Circuit-Breaker

The outermost layer guards against cascading failures when an entire provider experiences systemic issues. Implemented in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts), this is a **classic circuit-breaker with degradation state**.

### State Machine

The `CircuitBreaker` class implements five states with progressive escalation:

- **`CLOSED`** — Normal operation, requests pass through
- **`DEGRADED`** — Elevated failure rate detected; requests still pass but are monitored
- **`OPEN`** — Provider bypassed entirely; requests fail fast with cached fallback or error
- **`HALF_OPEN`** — Probe requests allowed to test recovery
- **Return to `CLOSED`** — Successful probes restore full service

### Key Configuration Parameters

Configured per provider via `getCircuitBreaker(provider, options)`:

```typescript
const cb = getCircuitBreaker("openai", {
  failureThreshold: 8,        // trips to DEGRADED
  degradationThreshold: 12,   // trips to OPEN
  resetTimeout: 60_000,       // base time before HALF_OPEN
  halfOpenRequests: 2,        // probes needed to close
  maxBackoffMultiplier: 4,    // caps exponential back-off
});

```

### Escalating Back-Off

Each `OPEN → HALF_OPEN → OPEN` cycle doubles the reset timeout (capped by `maxBackoffMultiplier`). This prevents thundering-herd problems during prolonged outages. State persists to `domain_circuit_breakers` via `saveCircuitBreakerState`/`loadCircuitBreakerState`, surviving process restarts.

### Usage Example

```typescript
import { getCircuitBreaker } from "@/shared/utils/circuitBreaker";

const cb = getCircuitBreaker("anthropic");

// Execute through breaker; automatically handles state transitions
const result = await cb.execute(async () => {
  return await fetchAnthropicChat(completionRequest);
});

// Monitor health in dashboards
const { state, retryAfterMs, failureCount } = cb.getStatus();

```

---

## Layer 2: Connection Cool-down

When failures are isolated to a specific credential rather than a whole provider, **Layer 2** applies targeted back-off through `markAccountUnavailable()` in [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts).

### Trigger Conditions

Any error classified as:
- HTTP 429 (rate limit)
- Transient per-connection errors
- Quota exhaustion signals

### Cool-down Mechanics

```typescript
// Base cooldown varies by credential type
const baseCooldownMs = isOAuth ? 5000 : 3000;  // 5s OAuth, 3s API key

// Exponential: cooldown = base * 2^n, where n = failure index
const cooldownMs = baseCooldownMs * Math.pow(2, failureIndex);

```

A `rateLimitedUntil` timestamp is set per connection. The system guarantees **single back-off increment per failure burst** to prevent stampede effects.

### Session Affinity (v3.8.50)

Multi-turn interactions maintain context through session pinning. Headers `X-Session-Id`, `x-codex-session-id`, or `x-omniroute-session` bind a client to the same connection for the conversation duration, preventing mid-dialog account switches that would lose state.

### Terminal States

Certain conditions persist until manually cleared:
- `banned` — administratively blocked
- `expired` — credential lifecycle ended
- `credits_exhausted` — billing threshold reached

### Implementation Example

```typescript
import { markAccountUnavailable } from "@/sse/services/auth";

// After catching a 429 in request handler
await markAccountUnavailable({
  provider: "openai",
  connectionId: "sk-abc123...",
  errorCode: 429,
  errorMessage: "Rate limit exceeded: 60 rpm",
});

```

---

## Layer 3: Model Lockout

The finest-grained layer protects against **model-specific failures**—quota exhaustion on `gpt-4o-mini` while `gpt-4o` remains available, or 404s when a model is deprecated. Implemented in [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts) via `lockModel()` and `clearModelLock()`.

### Lock Scope

Keys are composite: `${provider}:${connectionId}:${model}`. This precision avoids over-penalization.

### Recovery via Success-Decay

Unlike the circuit-breaker's time-based recovery, model lockouts implement **active decay**:

```typescript
// On successful request to locked model
const newCount = Math.floor(currentFailureCount / 2);

if (newCount === 0) {
  clearModelLock({ provider, connectionId, model });
}

```

Each success halves the failure count; full clearance occurs before timer expiry if traffic resumes healthily.

### Lockout Configuration

Default settings in [`src/lib/resilience/modelLockoutSettings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/modelLockoutSettings.ts):

```typescript
export const defaultModelLockoutSettings = {
  enabled: true,
  defaultCooldownMs: 120_000,    // 2 minute base
  maxCooldownMs: 900_000,        // 15 minute cap
  decayOnSuccess: true,
};

```

### Usage Pattern

```typescript
import { lockModel, clearModelLock } from "@/open-sse/services/accountFallback";

// Lock after model-specific failure
if (err.status === 404 && err.model === "claude-3-opus-20240229") {
  await lockModel({
    provider: "anthropic",
    connectionId: "key-456",
    model: "claude-3-opus-20240229",
    reason: "model_deprecated",
    expiresInMs: 300_000,  // 5 minutes
  });
}

// Decay on success (called by request handler)
await clearModelLock({
  provider: "anthropic",
  connectionId: "key-456", 
  model: "claude-3-opus-20240229",
});

```

---

## Supporting Mechanisms

### Quota-Share Concurrency Control

When `max_concurrent` is configured for a connection, requests serialize through Bottleneck rate limiters. This prevents account flooding when multiple clients share pooled credentials.

### Request-Queue Admission Control

Two tunable guards in [`src/lib/resilience/settings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/settings.ts):
- **`maxWaitMs`** — Maximum time a request may queue locally before rejection
- **`maxQueueDepth`** — Optional cap on pending requests per connection

These settings expose through Dashboard → Settings for runtime adjustment.

---

## Administrative Operations

### Reset Circuit-Breakers

```typescript
import { resetAllCircuitBreakers } from "@/shared/utils/circuitBreaker";

// Emergency clear: wipes DB state and in-memory registry
resetAllCircuitBreakers();

```

### Query System Health

```typescript
// Per-provider status
const openaiCB = getCircuitBreaker("openai");
console.log(openaiCB.getStatus());  
// { state: 'HALF_OPEN', retryAfterMs: 15000, failureCount: 7 }

```

---

## Summary

OmniRoute's 3-layer resilience system architecture delivers production-grade fault tolerance through progressive isolation:

- **Provider circuit-breakers** ([`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts)) halt traffic to failing services with escalating back-off and persistent state
- **Connection cool-downs** ([`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts)) apply targeted exponential back-off to rate-limited credentials, with session affinity preserving multi-turn context
- **Model lockouts** (same file) enable surgical restriction of failing endpoints with success-decay recovery

All layers operate independently, ensuring that a failure at one scope never over-penalizes healthy resources at narrower scopes. Configuration propagates through [`src/lib/resilience/settings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/settings.ts) with runtime dashboard controls.

---

## Frequently Asked Questions

### How does OmniRoute prevent a single rate-limited key from blocking all traffic?

The Connection Cool-down layer (Layer 2) isolates failures to individual credentials. When `markAccountUnavailable()` detects a 429, only that specific `connectionId` enters cooldown; other connections on the same provider continue serving requests. The `baseCooldownMs * 2^n` formula ensures transient issues resolve automatically without operator intervention.

### What triggers the circuit-breaker to move from HALF_OPEN back to OPEN?

Probe requests during `HALF_OPEN` state must succeed. If `halfOpenRequests` configured probes (default: 2) fail, the breaker reopens and doubles the `resetTimeout` (capped by `maxBackoffMultiplier`). This escalating delay prevents retry storms against recovering but unstable providers, as implemented in `getCircuitBreaker()` within [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts).

### Can model lockouts expire before the configured timer?

Yes. While `expiresInMs` sets a maximum lockout duration, the **success-decay** mechanism actively clears lockouts earlier. Each successful request to a locked model halves its failure count via `Math.floor(count/2)`; when the count reaches zero, `clearModelLock()` removes the entry immediately. This rewards transient recovery without waiting for timer expiry.

### Where are resilience settings configured and persisted?

Runtime configuration lives in [`src/lib/resilience/settings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/settings.ts) and [`src/lib/resilience/modelLockoutSettings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/modelLockoutSettings.ts), exposed through the Dashboard Settings UI. Circuit-breaker state persists to `domain_circuit_breakers` table via `saveCircuitBreakerState()`; connection cooldowns and terminal states (`banned`, `expired`, `credits_exhausted`) also survive restarts. Model lockouts currently remain in-memory only, using a `Map` keyed by `provider:connectionId:model`.