# How OmniRoute Implements Automatic Failover Across Multiple AI Providers

> Discover how OmniRoute's three-layer system with circuit breakers, cooldowns, and lockouts ensures automatic failover across multiple AI providers for uninterrupted service.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: how-to-guide
- Published: 2026-08-13

---

**OmniRoute implements automatic failover across multiple AI providers through a three-layer resilience system: provider-wide circuit breakers, connection-level cooldowns, and model-level lockouts, all orchestrated by a combo engine that continuously filters and reroutes requests to healthy targets.**

Production AI applications depend on uninterrupted access to language models, but provider outages, rate limits, and quota exhaustion are inevitable. OmniRoute, an open-source LLM routing platform by diegosouzapw, solves this through **automatic failover**—a mechanism that transparently reroutes failed requests to alternative providers without client intervention. This article examines exactly how the system implements this resilience, drawing from the v3.8.50 source code.

## The Three Layers of Resilience

OmniRoute's automatic failover operates through three complementary layers that function independently yet integrate during request processing.

### Provider-Wide Circuit Breaker

The **circuit breaker** pattern prevents cascading failures by tracking upstream errors per provider. Located in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts), this component maintains three states:

- **CLOSED**: Normal operation; requests flow through
- **OPEN**: Failure threshold exceeded; all calls short-circuited immediately
- **HALF_OPEN**: After a configurable back-off timeout, a single probe request tests recovery

```typescript
import { getCircuitBreaker } from '@/src/shared/utils/circuitBreaker';

const cb = getCircuitBreaker('openai');
console.log(cb.getStatus());
// → { name: 'openai', state: 'CLOSED', failureCount: 0, … }

```

When failures accumulate, the breaker opens. The state persists in `domainState` ([`src/lib/db/domainState.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/db/domainState.ts)) across restarts, ensuring durability. Successful probes transition the breaker back to closed.

### Connection Cooldown (Provider-Level)

Individual credentials—not entire providers—can enter cooldown. The [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts) module tracks each OAuth token or API key via `rateLimitedUntil` timestamps:

```typescript
// Connection cooldown helpers in accountFallback.ts
isProviderInCooldown(connection)      // Check if credential is temporarily blocked
recordProviderCooldown(connection, error)  // Set cooldown on 429/quota errors
clearAccountError(connection)         // Clear on successful use

```

This granular approach allows one exhausted API key to rest while others for the same provider continue serving traffic.

### Model-Level Lockout

Some providers quota-limit specific models. The `isModelLocked()` and `recordModelLockoutFailure()` functions ([`accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/accountFallback.ts)) isolate failing models without disabling the entire provider. This proves essential when providers like OpenAI restrict GPT-4 separately from GPT-3.5.

## The Combo Engine: Orchestrating Automatic Failover

The **combo engine** in [`open-sse/services/combo.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/combo.ts) integrates these layers during request processing. Here's the complete flow:

### Step 1: Build the Candidate Pool

`buildAutoCandidates()` gathers all viable provider-model combinations, annotating each with:

- Circuit breaker state (`circuitBreakerState`)
- Connection cooldown status (`isProviderInCooldown`)
- Model lockout flag (`isModelLocked`)
- Quota percentages and reset windows
- Cost-per-token and latency statistics

The scoring logic in [`open-sse/services/autoCombo/scoring.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/autoCombo/scoring.ts) generates `ProviderCandidate` objects containing this metadata.

### Step 2: Filter Unavailable Candidates

Before attempting any target, the engine applies sequential filters:

```typescript
// Pseudocode representing the filtering logic in combo.ts
if (cb.getStatus().state === "OPEN") {
  skipProvider();           // Entire provider unavailable
}
if (isProviderInCooldown(connection)) {
  skipConnection();         // Specific credential resting
}
if (isModelLocked(model)) {
  skipModel();              // Single model restricted
}

```

### Step 3: Select and Dispatch

Based on the combo strategy—**auto**, priority, weighted, or round-robin—the engine orders remaining candidates. The first healthy target receives the request via `handleSingleModel`.

### Step 4: Handle Failures and Fail Over

When a call returns a retry-eligible error (429, 502, 503, quota exceeded), the engine:

1. Records the failure (`recordProviderFailure`, `recordModelLockoutFailure`)
2. Updates circuit breaker counters (`CircuitBreaker._onFailure`)
3. Proceeds to the next candidate in the ranked list

This creates **automatic failover**—the request transparently reroutes without client awareness.

### Step 5: Update State

Successful requests clear connection cooldowns (`clearAccountError`) and update circuit breaker success counters (`_onSuccess`). Failed requests persist state to `domainState` for cross-session durability.

```typescript
// Example: making a chat request with automatic failover
import { handleComboChat } from '@/open-sse/services/combo';

await handleComboChat({
  body,                    // Standard OpenAI-compatible request
  combo: {
    name: 'default',
    strategy: 'auto',      // Uses full candidate pool with resilience filtering
    config: { maxRetries: 2 },
  },
  handleSingleModel,       // Your upstream fetch implementation
  log,
});

```

## Key Implementation Files

| File | Responsibility |
|------|---------------|
| [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) | Circuit breaker with state persistence and exponential back-off |
| [`open-sse/services/combo.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/combo.ts) | Core combo engine orchestrating failover |
| [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts) | Connection cooldowns and model lockouts |
| [`open-sse/services/providerCooldownTracker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/providerCooldownTracker.ts) | Global provider cooldown tracking |
| [`open-sse/services/autoCombo/scoring.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/autoCombo/scoring.ts) | Candidate scoring with resilience metadata |
| [`src/lib/db/domainState.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/db/domainState.ts) | Persistent state storage across restarts |

## Design Benefits of Lazy State Management

All three resilience layers use **lazy evaluation**:

- Circuit breakers re-open only when timeouts elapse—no polling required
- Cooldown timestamps become eligible automatically when `Date.now()` exceeds them
- No background jobs or complex scheduling needed

This prevents "sticky-down" scenarios where temporary outages permanently blacklist providers. The system self-heals without manual intervention.

## Summary

- **Three independent layers** provide defense in depth: circuit breakers (provider), cooldowns (connection), lockouts (model)
- **Lazy state management** eliminates polling overhead and enables automatic recovery
- **The combo engine** in [`combo.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/combo.ts) filters candidates at request time, ensuring automatic failover to the healthiest available target
- **Persistent state** in [`domainState.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/domainState.ts) maintains resilience across service restarts
- **Client-transparent operation** means applications receive consistent responses regardless of upstream provider changes

## Frequently Asked Questions

### How do I reset a circuit breaker after investigating a provider issue?

Import `resetAllCircuitBreakers` or use `getCircuitBreaker(name).reset()` from [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) to manually clear state. This is useful after verifying that a provider has recovered from an outage.

### What's the difference between connection cooldown and model lockout?

Connection cooldown affects a specific credential (API key or OAuth token) and typically triggers on rate limits or quota exhaustion. Model lockout restricts a single model identifier while the provider's other models remain available—useful when providers impose per-model quotas.

### Does automatic failover increase latency?

The failover itself adds minimal overhead—candidate filtering occurs in-memory during `buildAutoCandidates()`. However, if the primary provider fails and a secondary provider has higher latency, that provider's natural response time applies. The auto strategy weights latency statistics to prefer faster alternatives when available.

### What happens if all providers fail?

The combo engine returns a `ComboDiagnostics` payload containing the attempted candidate pool, failure reasons per target, and recovery hints. This enables debugging and alerts without silent failures.