# How OmniRoute Handles Upstream Provider Failures and Retries: A Three-Layer Resilience System

> Learn how OmniRoute ensures high availability by handling upstream provider failures with a three-layer resilience system including circuit breakers and cool-downs.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: deep-dive
- Published: 2026-08-31

---

**OmniRoute protects against upstream provider outages through a three-layer resilience architecture that combines provider-level circuit breakers, per-credential connection cool-downs, and model-specific lockouts to ensure high availability.**

OmniRoute is an open-source request routing layer designed to distribute LLM traffic across multiple providers. When upstream services fail, the system must distinguish between transient errors and systemic outages to maintain throughput. This article examines the specific mechanisms implemented in the `diegosouzapw/OmniRoute` repository that detect, classify, and recover from upstream failures.

## The Three-Layer Resilience Architecture

OmniRoute classifies failures by scope and severity, applying distinct recovery strategies through three coordinated layers.

### Provider Circuit Breaker

The **provider circuit breaker** protects the entire request pipeline from cascading failures when a provider becomes systemically unhealthy. According to [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts), the breaker tracks consecutive upstream HTTP **5xx responses** (including 408, 500, 501, 502, 503, and 504).

- **Threshold**: The `circuitBreakerThreshold` defaults to **8 consecutive failures** for API-key providers, though this is configurable per provider.
- **States**: The breaker transitions from **CLOSED** (normal operation) to **OPEN** (blocking all requests) when the threshold is exceeded.
- **Recovery**: After `circuitBreakerReset` (approximately **30 seconds**), the state shifts to **HALF-OPEN**, allowing a single probe request. A successful response closes the breaker; another failure reopens it.

In [`src/lib/resilience/settings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/settings.ts), the `getCircuitBreaker(provider, ...)` function retrieves the configured breaker instance for providers like `openai` or `anthropic`.

### Connection Cool-down

When failures are isolated to a specific credential rather than the entire provider, OmniRoute applies **connection cool-down**. This layer handles transient errors such as **429 Too Many Requests** or **401 Unauthorized** by marking individual accounts unavailable without penalizing the provider.

- **Implementation**: The `markAccountUnavailable()` function in [`src/sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/auth.ts) records the error state.
- **Back-off**: The cool-down period grows exponentially using the formula `baseCooldownMs × 2ⁿ`, where `n` represents the failure index.
- **Tracking**: Each connection stores `rateLimitedUntil` timestamps and `backoffLevel` counters to determine eligibility for selection.

### Model Lockout

The **model lockout** layer isolates failures specific to individual models (such as quota exhaustion or 404 Not Found errors) while keeping the underlying connection available for other models. This prevents a single unavailable model from triggering a full connection cool-down.

## Request Flow and Error Classification

Understanding how these layers interact requires tracing the request lifecycle through the codebase:

1. **Entry Point**: Client requests hit Next.js API routes (e.g., `/v1/chat/completions`) and pass through Zod validation and policy checks.
2. **Core Handling**: `handleChatCore()` in [`open-sse/handlers/chatCore.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/handlers/chatCore.ts) delegates to the combo router ([`open-sse/services/combo.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/combo.ts)).
3. **Routing Decision**: The `handleSingleModel()` function checks both the provider breaker state and per-connection availability before dispatching.
4. **Execution**: Executors in [`open-sse/executors/baseExecutor.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/executors/baseExecutor.ts) perform the actual HTTP calls and parse responses.

Error classification occurs at the executor level. The system distinguishes between:
- **Circuit-breaker eligible**: 408, 500-504 (recorded via `circuitBreaker.recordFailure()`)
- **Cool-down eligible**: 429, 401, other non-5xx errors (trigger `markAccountUnavailable()`)
- **Model-specific**: 404 or quota errors (isolated to the specific model)

## Retry Logic and Back-off Strategies

The retry implementation in [`open-sse/executors/baseExecutor.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/executors/baseExecutor.ts) respects upstream signaling while preventing thundering herds.

### Header-Aware Retries

When executors encounter a 429 response, they parse the `Retry-After` or `reset` headers to compute concrete wait times:

```typescript
import { BaseExecutor } from '@/open-sse/executors/baseExecutor';

class OpenAIExecutor extends BaseExecutor {
  async execute(request: Request): Promise<Response> {
    const resp = await fetch(this.endpoint, request);
    if (resp.status === 429) {
      const retryAfter = resp.headers.get('Retry-After');
      const waitMs = retryAfter ? parseInt(retryAfter, 10) * 1000 : this.baseBackoffMs;
      await new Promise(r => setTimeout(r, waitMs));
      return this.execute(request); // retry with exponential backoff
    }
    return resp;
  }
}

```

### Exponential Back-off

For repeated failures on the same connection, the executor applies jittered exponential back-off using the formula `baseCooldownMs * 2 ** failureIndex`. The `MAX_RETRIES` constant limits total retry attempts; after exhaustion, the failure propagates to the combo router, which may fall back to alternative targets based on the selected strategy (fill-first, weighted, or auto).

### Programmatic Cool-down Example

Developers can manually mark accounts unavailable when handling provider-specific errors:

```typescript
import { markAccountUnavailable } from '@/src/sse/services/auth';

async function handleProviderError(accountId: string, err: any) {
  if (err.status === 429) {
    await markAccountUnavailable(accountId, {
      errorCode: 429,
      rateLimitedUntil: Date.now() + 5000, // 5 second base cooldown
      backoffLevel: 1,
    });
  }
}

```

## Configuration and Monitoring

All resilience parameters are configurable through environment variables prefixed with `OMNIROUTE_PROVIDER_BREAKER_*` and `OMNIROUTE_CIRCUIT_BREAKER_*`. The health endpoint at [`src/app/api/monitoring/health/route.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/app/api/monitoring/health/route.ts) exposes real-time breaker states and connection cool-down status, enabling operational visibility into which providers are currently isolated.

To check a provider's circuit breaker status programmatically:

```typescript
import { getCircuitBreaker } from '@/src/lib/resilience/settings';
import { ProviderId } from '@/src/types/provider';

const breaker = getCircuitBreaker('openai' as ProviderId);
if (breaker.isOpen()) {
  console.log('OpenAI is currently blocked – will fallback to another provider');
}

```

Key implementation files include:
- [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) – Core state machine and persistence
- [`src/lib/resilience/settings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/settings.ts) – Breaker configuration factory
- [`open-sse/services/combo.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/combo.ts) – Combo routing and fallback strategies
- [`src/sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/auth.ts) – Credential selection and cool-down management

## Summary

- **Three-layer protection**: Provider circuit breakers handle systemic 5xx failures, connection cool-downs manage per-credential 429/401 errors, and model lockouts isolate specific model unavailability.
- **Intelligent retry logic**: Executors in [`open-sse/executors/baseExecutor.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/executors/baseExecutor.ts) respect `Retry-After` headers while applying exponential back-off with jitter.
- **State management**: The circuit breaker transitions through CLOSED → OPEN → HALF-OPEN states with configurable thresholds (default 8 failures) and reset windows (~30 seconds).
- **Operational visibility**: Runtime status is available through the `/api/monitoring/health` endpoint and configurable via environment variables.

## Frequently Asked Questions

### What is the difference between the provider circuit breaker and connection cool-down?

The **provider circuit breaker** affects all traffic to a specific provider (e.g., all OpenAI requests) when systemic failures (HTTP 5xx) exceed the threshold. The **connection cool-down** applies only to a single credential or API key within a provider, triggered by transient errors like 429 or 401. This distinction ensures that one rate-limited key does not block an entire provider, while systemic outages trigger immediate fallback to alternative providers.

### How does OmniRoute handle 429 Too Many Requests errors?

When executors receive a 429 response, they first attempt to parse the `Retry-After` header to determine the specific wait time. If no header is present, the system applies exponential back-off using `baseCooldownMs * 2 ** failureIndex`. Additionally, the connection is marked unavailable via `markAccountUnavailable()` in [`src/sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/auth.ts) until the back-off period expires, preventing immediate reuse of the exhausted credential.

### Can I configure the retry thresholds and circuit breaker settings?

Yes. OmniRoute exposes configuration through environment variables such as `OMNIROUTE_PROVIDER_BREAKER_THRESHOLD` and `OMNIROUTE_CIRCUIT_BREAKER_RESET_MS`. These values are consumed by [`src/lib/resilience/settings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/settings.ts) to instantiate breakers with custom parameters. You can also adjust `MAX_RETRIES` in the executor base class and modify `baseCooldownMs` for specific provider implementations.

### How can I monitor which providers are currently unavailable?

The health endpoint defined in [`src/app/api/monitoring/health/route.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/app/api/monitoring/health/route.ts) provides real-time visibility into circuit breaker states (CLOSED, OPEN, HALF-OPEN) and connection cool-down timers. This endpoint reports which providers are circuit-open and which specific credentials are rate-limited, allowing operators to diagnose routing decisions and provider health without inspecting internal logs.