OmniRoute 3-Layer Resilience System Architecture: Provider, Connection & Model Protection Explained

OmniRoute implements a stacked, self-healing resilience pipeline that isolates failures at three distinct scopes—provider-level circuit-breakers, per-connection cooldowns, and model-specific lockouts—to ensure continuous availability across upstream AI service disruptions.

This article examines the architecture of OmniRoute's 3-layer resilience system as implemented in the diegosouzapw/OmniRoute repository (release v3.8.50). The design protects every request through progressive failure isolation, from entire providers down to individual model endpoints.

---

Overview of the Three Protection Layers

The resilience system operates as independent, composable layers that can trigger simultaneously without interference:

Layer	Scope	Failure Type	Source Location
Layer 1: Provider Circuit-Breaker	Entire provider (e.g., openai, anthropic)	Repeated 5xx failures, systemic outages	src/shared/utils/circuitBreaker.ts
Layer 2: Connection Cool-down	Individual connection/account/key (OAuth, API key)	Transient rate limits (429), per-key quotas	open-sse/services/accountFallback.ts
Layer 3: Model Lockout	Provider + connection + model triple	Model-specific 404s, quota exhaustion, mode denials	open-sse/services/accountFallback.ts

Each layer maintains separate state machinery, allowing granular recovery: a provider can be OPEN (skipped entirely) while a specific connection on that same provider merely enters cooldown, and a single model can be locked without affecting sibling models on the same connection.

---

Layer 1: Provider Circuit-Breaker

The outermost layer guards against cascading failures when an entire provider experiences systemic issues. Implemented in src/shared/utils/circuitBreaker.ts, this is a classic circuit-breaker with degradation state.

State Machine

The CircuitBreaker class implements five states with progressive escalation:

• CLOSED — Normal operation, requests pass through
• DEGRADED — Elevated failure rate detected; requests still pass but are monitored
• OPEN — Provider bypassed entirely; requests fail fast with cached fallback or error
• HALF_OPEN — Probe requests allowed to test recovery
• Return to CLOSED — Successful probes restore full service

Key Configuration Parameters

Configured per provider via getCircuitBreaker(provider, options):

const cb = getCircuitBreaker("openai", {
  failureThreshold: 8,        // trips to DEGRADED
  degradationThreshold: 12,   // trips to OPEN
  resetTimeout: 60_000,       // base time before HALF_OPEN
  halfOpenRequests: 2,        // probes needed to close
  maxBackoffMultiplier: 4,    // caps exponential back-off
});

Escalating Back-Off

Each OPEN → HALF_OPEN → OPEN cycle doubles the reset timeout (capped by maxBackoffMultiplier). This prevents thundering-herd problems during prolonged outages. State persists to domain_circuit_breakers via saveCircuitBreakerState/loadCircuitBreakerState, surviving process restarts.

Usage Example

import { getCircuitBreaker } from "@/shared/utils/circuitBreaker";

const cb = getCircuitBreaker("anthropic");

// Execute through breaker; automatically handles state transitions
const result = await cb.execute(async () => {
  return await fetchAnthropicChat(completionRequest);
});

// Monitor health in dashboards
const { state, retryAfterMs, failureCount } = cb.getStatus();

---

Layer 2: Connection Cool-down

When failures are isolated to a specific credential rather than a whole provider, Layer 2 applies targeted back-off through markAccountUnavailable() in open-sse/services/accountFallback.ts.

Trigger Conditions

Any error classified as:

• HTTP 429 (rate limit)
• Transient per-connection errors
• Quota exhaustion signals

Cool-down Mechanics

// Base cooldown varies by credential type
const baseCooldownMs = isOAuth ? 5000 : 3000;  // 5s OAuth, 3s API key

// Exponential: cooldown = base * 2^n, where n = failure index
const cooldownMs = baseCooldownMs * Math.pow(2, failureIndex);

A rateLimitedUntil timestamp is set per connection. The system guarantees single back-off increment per failure burst to prevent stampede effects.

Session Affinity (v3.8.50)

Multi-turn interactions maintain context through session pinning. Headers X-Session-Id, x-codex-session-id, or x-omniroute-session bind a client to the same connection for the conversation duration, preventing mid-dialog account switches that would lose state.

Terminal States

Certain conditions persist until manually cleared:

• banned — administratively blocked
• expired — credential lifecycle ended
• credits_exhausted — billing threshold reached

Implementation Example

import { markAccountUnavailable } from "@/sse/services/auth";

// After catching a 429 in request handler
await markAccountUnavailable({
  provider: "openai",
  connectionId: "sk-abc123...",
  errorCode: 429,
  errorMessage: "Rate limit exceeded: 60 rpm",
});

---

Layer 3: Model Lockout

The finest-grained layer protects against model-specific failures—quota exhaustion on gpt-4o-mini while gpt-4o remains available, or 404s when a model is deprecated. Implemented in open-sse/services/accountFallback.ts via lockModel() and clearModelLock().

Lock Scope

Keys are composite: ${provider}:${connectionId}:${model}. This precision avoids over-penalization.

Recovery via Success-Decay

Unlike the circuit-breaker's time-based recovery, model lockouts implement active decay:

// On successful request to locked model
const newCount = Math.floor(currentFailureCount / 2);

if (newCount === 0) {
  clearModelLock({ provider, connectionId, model });
}

Each success halves the failure count; full clearance occurs before timer expiry if traffic resumes healthily.

Lockout Configuration

Default settings in src/lib/resilience/modelLockoutSettings.ts:

export const defaultModelLockoutSettings = {
  enabled: true,
  defaultCooldownMs: 120_000,    // 2 minute base
  maxCooldownMs: 900_000,        // 15 minute cap
  decayOnSuccess: true,
};

Usage Pattern

import { lockModel, clearModelLock } from "@/open-sse/services/accountFallback";

// Lock after model-specific failure
if (err.status === 404 && err.model === "claude-3-opus-20240229") {
  await lockModel({
    provider: "anthropic",
    connectionId: "key-456",
    model: "claude-3-opus-20240229",
    reason: "model_deprecated",
    expiresInMs: 300_000,  // 5 minutes
  });
}

// Decay on success (called by request handler)
await clearModelLock({
  provider: "anthropic",
  connectionId: "key-456", 
  model: "claude-3-opus-20240229",
});

---

Supporting Mechanisms

Quota-Share Concurrency Control

When max_concurrent is configured for a connection, requests serialize through Bottleneck rate limiters. This prevents account flooding when multiple clients share pooled credentials.

Request-Queue Admission Control

Two tunable guards in src/lib/resilience/settings.ts:

• maxWaitMs — Maximum time a request may queue locally before rejection
• maxQueueDepth — Optional cap on pending requests per connection

These settings expose through Dashboard → Settings for runtime adjustment.

---

Administrative Operations

Reset Circuit-Breakers

import { resetAllCircuitBreakers } from "@/shared/utils/circuitBreaker";

// Emergency clear: wipes DB state and in-memory registry
resetAllCircuitBreakers();

Query System Health

// Per-provider status
const openaiCB = getCircuitBreaker("openai");
console.log(openaiCB.getStatus());  
// { state: 'HALF_OPEN', retryAfterMs: 15000, failureCount: 7 }

---

Summary

OmniRoute's 3-layer resilience system architecture delivers production-grade fault tolerance through progressive isolation:

• Provider circuit-breakers (src/shared/utils/circuitBreaker.ts) halt traffic to failing services with escalating back-off and persistent state
• Connection cool-downs (open-sse/services/accountFallback.ts) apply targeted exponential back-off to rate-limited credentials, with session affinity preserving multi-turn context
• Model lockouts (same file) enable surgical restriction of failing endpoints with success-decay recovery

All layers operate independently, ensuring that a failure at one scope never over-penalizes healthy resources at narrower scopes. Configuration propagates through src/lib/resilience/settings.ts with runtime dashboard controls.

---

Frequently Asked Questions

How does OmniRoute prevent a single rate-limited key from blocking all traffic?

The Connection Cool-down layer (Layer 2) isolates failures to individual credentials. When markAccountUnavailable() detects a 429, only that specific connectionId enters cooldown; other connections on the same provider continue serving requests. The baseCooldownMs * 2^n formula ensures transient issues resolve automatically without operator intervention.

What triggers the circuit-breaker to move from HALF_OPEN back to OPEN?

Probe requests during HALF_OPEN state must succeed. If halfOpenRequests configured probes (default: 2) fail, the breaker reopens and doubles the resetTimeout (capped by maxBackoffMultiplier). This escalating delay prevents retry storms against recovering but unstable providers, as implemented in getCircuitBreaker() within src/shared/utils/circuitBreaker.ts.

Can model lockouts expire before the configured timer?

Yes. While expiresInMs sets a maximum lockout duration, the success-decay mechanism actively clears lockouts earlier. Each successful request to a locked model halves its failure count via Math.floor(count/2); when the count reaches zero, clearModelLock() removes the entry immediately. This rewards transient recovery without waiting for timer expiry.

Where are resilience settings configured and persisted?

Runtime configuration lives in src/lib/resilience/settings.ts and src/lib/resilience/modelLockoutSettings.ts, exposed through the Dashboard Settings UI. Circuit-breaker state persists to domain_circuit_breakers table via saveCircuitBreakerState(); connection cooldowns and terminal states (banned, expired, credits_exhausted) also survive restarts. Model lockouts currently remain in-memory only, using a Map keyed by provider:connectionId:model.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}