# OmniRoute 3-Layer Resilience Model: How Provider, Connection, and Model-Level Failures Are Isolated

> Discover OmniRoute's 3-layer resilience model. Isolate provider, connection, and model failures for robust system operation. Learn how hierarchical failure handling protects your application.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: architecture
- Published: 2026-08-26

---

**OmniRoute implements a three-layer resilience architecture that isolates provider-wide outages, credential-specific throttling, and per-model quota limits through hierarchical failure handling.**

The OmniRoute 3-layer resilience model is a hierarchical failure-isolation system designed to keep AI routing functional during upstream disruptions. As implemented in `diegosouzapw/OmniRoute`, each layer targets a specific scope—from entire providers down to individual models—ensuring that localized failures don't cascade into system-wide outages.

## OmniRoute Resilience Model Overview

The architecture operates on a single principle: handle the **broadest failures first**, then narrow focus to preserve as many healthy resources as possible. This lazy-recovery design avoids background timers; components self-heal automatically when traffic resumes after timeout expiration.

| Layer | Scope | State Transition Trigger |
|-------|-------|--------------------------|
| Provider Circuit Breaker | Entire provider (e.g., *openai*, *anthropic*) | Consecutive upstream server errors |
| Connection Cooldown | Individual API key or OAuth account | Rate limits or transient credential errors |
| Model Lockout | Provider + connection + model triple | Per-model quota exhaustion |

## Layer 1: Provider Circuit Breaker

The provider circuit breaker acts as the outermost guard, stopping all traffic to a provider that exhibits systemic failure.

### Implementation Details

Core logic resides in [[`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts)](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.51/src/shared/utils/circuitBreaker.ts), with integration points in:

- [`src/sse/handlers/chatHelpers.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/handlers/chatHelpers.ts)
- [`src/sse/handlers/chat.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/handlers/chat.ts)

Configuration thresholds and reset timers are defined in [[`open-sse/config/constants.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/config/constants.ts)](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.51/open-sse/config/constants.ts) and exposed via the Dashboard at **Settings → Resilience**.

The circuit breaker implements standard states: `CLOSED` (normal operation), `OPEN` (traffic blocked), `HALF_OPEN` (test request allowed), and `DEGRADED` (reduced capacity).

```typescript
// Inspecting a provider's circuit-breaker state
import { getCircuitBreaker } from '@/src/shared/utils/circuitBreaker';

const cb = getCircuitBreaker('openai');
console.log(cb.getStatus()); // CLOSED | DEGRADED | OPEN | HALF_OPEN

```

When `OPEN`, the provider is excluded from combo routing entirely—no requests reach that provider's connection pool.

## Layer 2: Connection Cooldown

When a provider remains `CLOSED` at the circuit-breaker level but individual credentials fail, connection cooldown isolates the problematic credential while preserving other connections for the same provider.

### Key Functions and Files

- **Marking unavailable**: `src/sse/services/auth.ts::markAccountUnavailable()`
- **Cooldown calculation**: `open-sse/services/accountFallback.ts::checkFallbackError()`
- **Settings management**: [`src/lib/resilience/settings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/settings.ts)

### Default Cooldown Behavior

| Credential Type | Base Cooldown | Back-off Formula |
|-----------------|---------------|------------------|
| OAuth accounts | 5,000 ms | `baseCooldownMs * 2^n` |
| API keys | 3,000 ms | `baseCooldownMs * 2^n` |

```typescript
// Manually forcing a connection cooldown (useful for testing)
import { markAccountUnavailable } from '@/src/sse/services/auth';

await markAccountUnavailable({
  provider: 'openai',
  connectionId: 'conn-123',
  reason: 'rate_limit',
  retryAfterMs: 15000,
});

```

This fine-grained approach prevents a single rate-limited API key from disabling an entire provider's capacity.

## Layer 3: Model Lockout

The innermost layer addresses per-model quota limits—a scenario where a connection works for most models but fails for one specific model due to provider-side quotas.

### Implementation in accountFallback.ts

Model lockout logic lives in [[`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts)](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.51/open-sse/services/accountFallback.ts) via:

- `lockModel()` — isolates a failing model
- `clearModelLock()` — restores model availability

Configuration defaults are in [[`src/lib/resilience/modelLockoutSettings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/modelLockoutSettings.ts)](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.51/src/lib/resilience/modelLockoutSettings.ts).

### Configuration Defaults

| Setting | Value |
|---------|-------|
| Enabled by default | `false` |
| Initial lockout duration | 120 seconds |
| Maximum duration | 30 minutes |
| Back-off strategy | Exponential growth |

The UI for toggling this feature is at [`src/app/(dashboard)/dashboard/settings/components/ModelLockoutCard.tsx`](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.51/src/app/(dashboard)/dashboard/settings/components/ModelLockoutCard.tsx).

```typescript
// Triggering a model lockout via the API
// POST /api/resilience/model-cooldowns
{
  "provider": "openai",
  "connection": "conn-123",
  "model": "gpt-4o-mini"
}

```

## How the 3 Layers Interact

Request processing follows strict evaluation order:

1. **Circuit breaker check** — If `OPEN`, reject immediately; provider excluded from routing
2. **Connection availability** — If credential is rate-limited, skip to next available credential
3. **Model lockout verification** — If specific model is locked, fall back to alternative model on same connection

This hierarchy guarantees maximum resource utilization. A provider-wide outage triggers layer 1. A single bad API key triggers layer 2 while other keys continue serving. A model quota issue triggers layer 3 while other models on the same connection remain operational.

## Key Source Files Reference

| Component | File Path | Purpose |
|-----------|-----------|---------|
| Circuit breaker core | [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) | State machine implementation |
| Credential marking | [`src/sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/auth.ts) | `markAccountUnavailable()` function |
| Fallback logic | [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts) | Cooldown calculation and model lockout |
| Circuit constants | [`open-sse/config/constants.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/config/constants.ts) | Threshold configuration |
| Model lockout settings | [`src/lib/resilience/modelLockoutSettings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/modelLockoutSettings.ts) | Default durations and limits |
| Dashboard UI | `src/app/(dashboard)/dashboard/settings/components/ModelLockoutCard.tsx` | User-facing controls |
| Architecture docs | [`docs/architecture/RESILIENCE_GUIDE.md`](https://github.com/diegosouzapw/OmniRoute/blob/main/docs/architecture/RESILIENCE_GUIDE.md) | Complete design documentation |

## Summary

- **Provider Circuit Breaker** ([`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts)) stops all traffic to failing providers
- **Connection Cooldown** ([`src/sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/auth.ts), [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts)) isolates bad credentials without affecting provider-wide availability
- **Model Lockout** ([`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts)) prevents per-model quota issues from disabling entire connections
- **Lazy recovery** eliminates background timers—components heal automatically when traffic resumes
- **Hierarchical evaluation** maximizes healthy resource pool at each routing decision

## Frequently Asked Questions

### How do I configure circuit breaker thresholds in OmniRoute?

Thresholds and reset timers are defined in [`open-sse/config/constants.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/config/constants.ts) and surfaced through the Dashboard at **Settings → Resilience**. Modify constants directly for deployment-wide changes, or use the UI for per-organization tuning.

### What happens when multiple resilience layers trigger simultaneously?

The layers evaluate in strict order: circuit breaker first, then connection cooldown, then model lockout. A provider in `OPEN` state bypasses all downstream checks. This ensures the broadest failure scope is handled with highest priority, preserving granular resources when possible.

### Why is model lockout disabled by default?

Model lockout carries a risk of over-isolation—legitimate temporary errors could lock models unnecessarily. Operators must explicitly enable it in [`src/lib/resilience/modelLockoutSettings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/modelLockoutSettings.ts) or via the Dashboard after evaluating their specific failure patterns and quota structures.

### How does lazy recovery work without background timers?

Each layer stores expiration timestamps (`rateLimitedUntil` for connections, lock timeouts for models, state transition times for circuit breakers). When the next request arrives, the system checks `Date.now()` against these timestamps and promotes state automatically. This eliminates timer overhead while ensuring immediate recovery when traffic resumes.