How OmniRoute's 3-Layer Resilience System Prevents Cascading Failures

OmniRoute prevents cascading failures by isolating problems at three distinct scopes—provider, connection, and model—so a single point of failure cannot propagate through the request pipeline.

When routing LLM requests across multiple providers, a single failing upstream can quickly spiral into a system-wide outage without proper isolation. OmniRoute solves this through a layered resilience architecture that operates at different granularities. This article examines how each layer works based on the source code in diegosouzapw/OmniRoute.

Layer 1: Provider Circuit Breaker

The provider circuit breaker operates at the broadest scope, protecting the entire request pipeline when an upstream provider fails repeatedly.

How It Works

In src/shared/utils/circuitBreaker.ts, OmniRoute tracks provider-wide error codes (408, 500, 502, 503, 504) across all connections. When failures exceed the configured providerFailureThreshold, the breaker transitions to OPEN and blocks all traffic to that provider. This happens atomically—no new requests are accepted until recovery.

After providerCooldownMs elapses, the breaker enters HALF_OPEN. The next request becomes a probe: success closes the breaker, failure reopens it. This lazy recovery prevents thundering herds.

import { getProviderStatus } from '@/shared/utils/circuitBreaker';

async function routeRequest(providerId: string) {
  const status = getProviderStatus(providerId);
  if (status === 'OPEN') {
    // Provider is circuit-broken – skip it in combo routing
    return fallbackToOtherProviders();
  }
  // Normal execution path
  return executeProviderRequest(providerId);
}

Without this layer, a provider experiencing a partial outage would slow every request through timeout accumulation. The circuit breaker forces an immediate hard stop.

Layer 2: Connection Cooldown

The connection cooldown layer handles transient failures at the individual credential level—API keys, accounts, or endpoints within the same provider.

How It Works

In src/sse/services/auth.ts, each connection maintains rateLimitedUntil metadata. When a transient error occurs (429, network timeout), markAccountUnavailable() applies exponential backoff: baseCooldownMs * 2ⁿ where n increments with repeated failures.

import { markAccountUnavailable } from '@/sse/services/auth';

async function handleProviderResponse(resp) {
  if (resp.status === 429) {
    await markAccountUnavailable(resp.accountId, {
      errorCode: 429,
      backoffLevel: resp.retryAfter ? 1 : 0,
      rateLimitedUntil: Date.now() + (resp.retryAfter ?? 5_000),
    });
  }
}

Other connections for the same provider remain eligible. This isolates throttled or rate-limited keys without discarding the entire provider from rotation.

Layer 3: Model Lockout

The model lockout layer provides the finest granularity, isolating failures to a specific model on a specific connection.

How It Works

In open-sse/services/accountFallback.ts, model-specific errors (429 for quota exhaustion, 404 for missing local models) trigger lockoutModelOnConnection(). Only that model becomes unavailable; other models on the same connection continue serving requests.

import { lockoutModelOnConnection } from '@/open-sse/services/accountFallback';

async function processModelError(connId, model, err) {
  if (err.code === 429 && err.modelSpecific) {
    await lockoutModelOnConnection(connId, model, err);
  }
}

Without this layer, a single model's quota limit would disable an entire API key. Model lockout preserves connection utility by narrowing the failure scope to exactly what's broken.

How the Layers Interact During Failures

OmniRoute applies these layers hierarchically during request routing:

1. Provider check first – If the circuit breaker is OPEN, skip the provider entirely.
2. Connection filter – Among available providers, exclude connections where rateLimitedUntil > now().
3. Model eligibility – From remaining connections, filter out models currently locked out.

The combo routing logic in open-sse/services/combo.ts orchestrates this selection. Each layer's state is evaluated independently, so multiple failure modes can coexist without interference.

Failure Scenario	Layer Activated	Impact Scope
OpenAI regional outage	Provider Circuit Breaker	All OpenAI traffic blocked
Single API key throttled	Connection Cooldown	Only that key skipped
GPT-4 quota exhausted on key A	Model Lockout	GPT-4 unavailable on key A; GPT-3.5 and other keys unaffected

Summary

• Provider circuit breaker in src/shared/utils/circuitBreaker.ts stops traffic to entire failing providers using threshold-based state transitions.
• Connection cooldown in src/sse/services/auth.ts isolates individual credentials with exponential backoff timers.
• Model lockout in open-sse/services/accountFallback.ts restricts failures to specific models without disabling whole connections.
• The three layers operate independently, ensuring no single failure mode can cascade into system-wide unavailability.

Frequently Asked Questions

What error codes trigger the provider circuit breaker?

The provider circuit breaker monitors HTTP 408, 500, 502, 503, 504 as defined in src/shared/utils/circuitBreaker.ts. These indicate upstream or service-level failures rather than client or rate-limiting errors.

How does connection cooldown differ from the circuit breaker?

Connection cooldown handles transient, recoverable errors (429, timeouts) at the individual credential level using timed backoff. The circuit breaker handles persistent, provider-wide failures using state machine transitions. A throttled key enters cooldown; a provider returning 503 errors opens the breaker.

Can a model be locked out on one connection but available on another?

Yes. Model lockout is scoped to connectionId + model. The same model remains available on other connections for the same provider, and other models remain available on the affected connection.

How does OmniRoute recover from a circuit breaker OPEN state?

Recovery uses lazy probing. After providerCooldownMs expires, the breaker enters HALF_OPEN. The next request through becomes a probe—success closes the breaker immediately, failure reopens it for another cooldown period.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}