How OmniRoute Handles Upstream Provider Failures and Retries: A Three-Layer Resilience System

OmniRoute protects against upstream provider outages through a three-layer resilience architecture that combines provider-level circuit breakers, per-credential connection cool-downs, and model-specific lockouts to ensure high availability.

OmniRoute is an open-source request routing layer designed to distribute LLM traffic across multiple providers. When upstream services fail, the system must distinguish between transient errors and systemic outages to maintain throughput. This article examines the specific mechanisms implemented in the diegosouzapw/OmniRoute repository that detect, classify, and recover from upstream failures.

The Three-Layer Resilience Architecture

OmniRoute classifies failures by scope and severity, applying distinct recovery strategies through three coordinated layers.

Provider Circuit Breaker

The provider circuit breaker protects the entire request pipeline from cascading failures when a provider becomes systemically unhealthy. According to src/shared/utils/circuitBreaker.ts, the breaker tracks consecutive upstream HTTP 5xx responses (including 408, 500, 501, 502, 503, and 504).

• Threshold: The circuitBreakerThreshold defaults to 8 consecutive failures for API-key providers, though this is configurable per provider.
• States: The breaker transitions from CLOSED (normal operation) to OPEN (blocking all requests) when the threshold is exceeded.
• Recovery: After circuitBreakerReset (approximately 30 seconds), the state shifts to HALF-OPEN, allowing a single probe request. A successful response closes the breaker; another failure reopens it.

In src/lib/resilience/settings.ts, the getCircuitBreaker(provider, ...) function retrieves the configured breaker instance for providers like openai or anthropic.

Connection Cool-down

When failures are isolated to a specific credential rather than the entire provider, OmniRoute applies connection cool-down. This layer handles transient errors such as 429 Too Many Requests or 401 Unauthorized by marking individual accounts unavailable without penalizing the provider.

• Implementation: The markAccountUnavailable() function in src/sse/services/auth.ts records the error state.
• Back-off: The cool-down period grows exponentially using the formula baseCooldownMs × 2ⁿ, where n represents the failure index.
• Tracking: Each connection stores rateLimitedUntil timestamps and backoffLevel counters to determine eligibility for selection.

Model Lockout

The model lockout layer isolates failures specific to individual models (such as quota exhaustion or 404 Not Found errors) while keeping the underlying connection available for other models. This prevents a single unavailable model from triggering a full connection cool-down.

Request Flow and Error Classification

Understanding how these layers interact requires tracing the request lifecycle through the codebase:

1. Entry Point: Client requests hit Next.js API routes (e.g., /v1/chat/completions) and pass through Zod validation and policy checks.
2. Core Handling: handleChatCore() in open-sse/handlers/chatCore.ts delegates to the combo router (open-sse/services/combo.ts).
3. Routing Decision: The handleSingleModel() function checks both the provider breaker state and per-connection availability before dispatching.
4. Execution: Executors in open-sse/executors/baseExecutor.ts perform the actual HTTP calls and parse responses.

Error classification occurs at the executor level. The system distinguishes between:

• Circuit-breaker eligible: 408, 500-504 (recorded via circuitBreaker.recordFailure())
• Cool-down eligible: 429, 401, other non-5xx errors (trigger markAccountUnavailable())
• Model-specific: 404 or quota errors (isolated to the specific model)

Retry Logic and Back-off Strategies

The retry implementation in open-sse/executors/baseExecutor.ts respects upstream signaling while preventing thundering herds.

Header-Aware Retries

When executors encounter a 429 response, they parse the Retry-After or reset headers to compute concrete wait times:

import { BaseExecutor } from '@/open-sse/executors/baseExecutor';

class OpenAIExecutor extends BaseExecutor {
  async execute(request: Request): Promise<Response> {
    const resp = await fetch(this.endpoint, request);
    if (resp.status === 429) {
      const retryAfter = resp.headers.get('Retry-After');
      const waitMs = retryAfter ? parseInt(retryAfter, 10) * 1000 : this.baseBackoffMs;
      await new Promise(r => setTimeout(r, waitMs));
      return this.execute(request); // retry with exponential backoff
    }
    return resp;
  }
}

Exponential Back-off

For repeated failures on the same connection, the executor applies jittered exponential back-off using the formula baseCooldownMs * 2 ** failureIndex. The MAX_RETRIES constant limits total retry attempts; after exhaustion, the failure propagates to the combo router, which may fall back to alternative targets based on the selected strategy (fill-first, weighted, or auto).

Programmatic Cool-down Example

Developers can manually mark accounts unavailable when handling provider-specific errors:

import { markAccountUnavailable } from '@/src/sse/services/auth';

async function handleProviderError(accountId: string, err: any) {
  if (err.status === 429) {
    await markAccountUnavailable(accountId, {
      errorCode: 429,
      rateLimitedUntil: Date.now() + 5000, // 5 second base cooldown
      backoffLevel: 1,
    });
  }
}

Configuration and Monitoring

All resilience parameters are configurable through environment variables prefixed with OMNIROUTE_PROVIDER_BREAKER_* and OMNIROUTE_CIRCUIT_BREAKER_*. The health endpoint at src/app/api/monitoring/health/route.ts exposes real-time breaker states and connection cool-down status, enabling operational visibility into which providers are currently isolated.

To check a provider's circuit breaker status programmatically:

import { getCircuitBreaker } from '@/src/lib/resilience/settings';
import { ProviderId } from '@/src/types/provider';

const breaker = getCircuitBreaker('openai' as ProviderId);
if (breaker.isOpen()) {
  console.log('OpenAI is currently blocked – will fallback to another provider');
}

Key implementation files include:

• src/shared/utils/circuitBreaker.ts – Core state machine and persistence
• src/lib/resilience/settings.ts – Breaker configuration factory
• open-sse/services/combo.ts – Combo routing and fallback strategies
• src/sse/services/auth.ts – Credential selection and cool-down management

Summary

• Three-layer protection: Provider circuit breakers handle systemic 5xx failures, connection cool-downs manage per-credential 429/401 errors, and model lockouts isolate specific model unavailability.
• Intelligent retry logic: Executors in open-sse/executors/baseExecutor.ts respect Retry-After headers while applying exponential back-off with jitter.
• State management: The circuit breaker transitions through CLOSED → OPEN → HALF-OPEN states with configurable thresholds (default 8 failures) and reset windows (~30 seconds).
• Operational visibility: Runtime status is available through the /api/monitoring/health endpoint and configurable via environment variables.

Frequently Asked Questions

What is the difference between the provider circuit breaker and connection cool-down?

The provider circuit breaker affects all traffic to a specific provider (e.g., all OpenAI requests) when systemic failures (HTTP 5xx) exceed the threshold. The connection cool-down applies only to a single credential or API key within a provider, triggered by transient errors like 429 or 401. This distinction ensures that one rate-limited key does not block an entire provider, while systemic outages trigger immediate fallback to alternative providers.

How does OmniRoute handle 429 Too Many Requests errors?

When executors receive a 429 response, they first attempt to parse the Retry-After header to determine the specific wait time. If no header is present, the system applies exponential back-off using baseCooldownMs * 2 ** failureIndex. Additionally, the connection is marked unavailable via markAccountUnavailable() in src/sse/services/auth.ts until the back-off period expires, preventing immediate reuse of the exhausted credential.

Can I configure the retry thresholds and circuit breaker settings?

Yes. OmniRoute exposes configuration through environment variables such as OMNIROUTE_PROVIDER_BREAKER_THRESHOLD and OMNIROUTE_CIRCUIT_BREAKER_RESET_MS. These values are consumed by src/lib/resilience/settings.ts to instantiate breakers with custom parameters. You can also adjust MAX_RETRIES in the executor base class and modify baseCooldownMs for specific provider implementations.

How can I monitor which providers are currently unavailable?

The health endpoint defined in src/app/api/monitoring/health/route.ts provides real-time visibility into circuit breaker states (CLOSED, OPEN, HALF-OPEN) and connection cool-down timers. This endpoint reports which providers are circuit-open and which specific credentials are rate-limited, allowing operators to diagnose routing decisions and provider health without inspecting internal logs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}