OmniRoute 3-Layer Resilience Model: How Provider, Connection, and Model-Level Failures Are Isolated

OmniRoute implements a three-layer resilience architecture that isolates provider-wide outages, credential-specific throttling, and per-model quota limits through hierarchical failure handling.

The OmniRoute 3-layer resilience model is a hierarchical failure-isolation system designed to keep AI routing functional during upstream disruptions. As implemented in diegosouzapw/OmniRoute, each layer targets a specific scope—from entire providers down to individual models—ensuring that localized failures don't cascade into system-wide outages.

OmniRoute Resilience Model Overview

The architecture operates on a single principle: handle the broadest failures first, then narrow focus to preserve as many healthy resources as possible. This lazy-recovery design avoids background timers; components self-heal automatically when traffic resumes after timeout expiration.

Layer	Scope	State Transition Trigger
Provider Circuit Breaker	Entire provider (e.g., openai, anthropic)	Consecutive upstream server errors
Connection Cooldown	Individual API key or OAuth account	Rate limits or transient credential errors
Model Lockout	Provider + connection + model triple	Per-model quota exhaustion

Layer 1: Provider Circuit Breaker

The provider circuit breaker acts as the outermost guard, stopping all traffic to a provider that exhibits systemic failure.

Implementation Details

Core logic resides in [src/shared/utils/circuitBreaker.ts](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.51/src/shared/utils/circuitBreaker.ts), with integration points in:

• src/sse/handlers/chatHelpers.ts
• src/sse/handlers/chat.ts

Configuration thresholds and reset timers are defined in [open-sse/config/constants.ts](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.51/open-sse/config/constants.ts) and exposed via the Dashboard at Settings → Resilience.

The circuit breaker implements standard states: CLOSED (normal operation), OPEN (traffic blocked), HALF_OPEN (test request allowed), and DEGRADED (reduced capacity).

// Inspecting a provider's circuit-breaker state
import { getCircuitBreaker } from '@/src/shared/utils/circuitBreaker';

const cb = getCircuitBreaker('openai');
console.log(cb.getStatus()); // CLOSED | DEGRADED | OPEN | HALF_OPEN

When OPEN, the provider is excluded from combo routing entirely—no requests reach that provider's connection pool.

Layer 2: Connection Cooldown

When a provider remains CLOSED at the circuit-breaker level but individual credentials fail, connection cooldown isolates the problematic credential while preserving other connections for the same provider.

Key Functions and Files

• Marking unavailable: src/sse/services/auth.ts::markAccountUnavailable()
• Cooldown calculation: open-sse/services/accountFallback.ts::checkFallbackError()
• Settings management: src/lib/resilience/settings.ts

Default Cooldown Behavior

Credential Type	Base Cooldown	Back-off Formula
OAuth accounts	5,000 ms	baseCooldownMs * 2^n
API keys	3,000 ms	baseCooldownMs * 2^n

// Manually forcing a connection cooldown (useful for testing)
import { markAccountUnavailable } from '@/src/sse/services/auth';

await markAccountUnavailable({
  provider: 'openai',
  connectionId: 'conn-123',
  reason: 'rate_limit',
  retryAfterMs: 15000,
});

This fine-grained approach prevents a single rate-limited API key from disabling an entire provider's capacity.

Layer 3: Model Lockout

The innermost layer addresses per-model quota limits—a scenario where a connection works for most models but fails for one specific model due to provider-side quotas.

Implementation in accountFallback.ts

Model lockout logic lives in [open-sse/services/accountFallback.ts](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.51/open-sse/services/accountFallback.ts) via:

• lockModel() — isolates a failing model
• clearModelLock() — restores model availability

Configuration defaults are in [src/lib/resilience/modelLockoutSettings.ts](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.51/src/lib/resilience/modelLockoutSettings.ts).

Configuration Defaults

Setting	Value
Enabled by default	false
Initial lockout duration	120 seconds
Maximum duration	30 minutes
Back-off strategy	Exponential growth

The UI for toggling this feature is at src/app/(dashboard)/dashboard/settings/components/ModelLockoutCard.tsx.

// Triggering a model lockout via the API
// POST /api/resilience/model-cooldowns
{
  "provider": "openai",
  "connection": "conn-123",
  "model": "gpt-4o-mini"
}

How the 3 Layers Interact

Request processing follows strict evaluation order:

1. Circuit breaker check — If OPEN, reject immediately; provider excluded from routing
2. Connection availability — If credential is rate-limited, skip to next available credential
3. Model lockout verification — If specific model is locked, fall back to alternative model on same connection

This hierarchy guarantees maximum resource utilization. A provider-wide outage triggers layer 1. A single bad API key triggers layer 2 while other keys continue serving. A model quota issue triggers layer 3 while other models on the same connection remain operational.

Key Source Files Reference

Component	File Path	Purpose
Circuit breaker core	src/shared/utils/circuitBreaker.ts	State machine implementation
Credential marking	src/sse/services/auth.ts	markAccountUnavailable() function
Fallback logic	open-sse/services/accountFallback.ts	Cooldown calculation and model lockout
Circuit constants	open-sse/config/constants.ts	Threshold configuration
Model lockout settings	src/lib/resilience/modelLockoutSettings.ts	Default durations and limits
Dashboard UI	src/app/(dashboard)/dashboard/settings/components/ModelLockoutCard.tsx	User-facing controls
Architecture docs	docs/architecture/RESILIENCE_GUIDE.md	Complete design documentation

Summary

• Provider Circuit Breaker (src/shared/utils/circuitBreaker.ts) stops all traffic to failing providers
• Connection Cooldown (src/sse/services/auth.ts, open-sse/services/accountFallback.ts) isolates bad credentials without affecting provider-wide availability
• Model Lockout (open-sse/services/accountFallback.ts) prevents per-model quota issues from disabling entire connections
• Lazy recovery eliminates background timers—components heal automatically when traffic resumes
• Hierarchical evaluation maximizes healthy resource pool at each routing decision

Frequently Asked Questions

How do I configure circuit breaker thresholds in OmniRoute?

Thresholds and reset timers are defined in open-sse/config/constants.ts and surfaced through the Dashboard at Settings → Resilience. Modify constants directly for deployment-wide changes, or use the UI for per-organization tuning.

What happens when multiple resilience layers trigger simultaneously?

The layers evaluate in strict order: circuit breaker first, then connection cooldown, then model lockout. A provider in OPEN state bypasses all downstream checks. This ensures the broadest failure scope is handled with highest priority, preserving granular resources when possible.

Why is model lockout disabled by default?

Model lockout carries a risk of over-isolation—legitimate temporary errors could lock models unnecessarily. Operators must explicitly enable it in src/lib/resilience/modelLockoutSettings.ts or via the Dashboard after evaluating their specific failure patterns and quota structures.

How does lazy recovery work without background timers?

Each layer stores expiration timestamps (rateLimitedUntil for connections, lock timeouts for models, state transition times for circuit breakers). When the next request arrives, the system checks Date.now() against these timestamps and promotes state automatically. This eliminates timer overhead while ensuring immediate recovery when traffic resumes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}