How OmniRoute Implements Automatic Failover Across Multiple AI Providers

OmniRoute implements automatic failover across multiple AI providers through a three-layer resilience system: provider-wide circuit breakers, connection-level cooldowns, and model-level lockouts, all orchestrated by a combo engine that continuously filters and reroutes requests to healthy targets.

Production AI applications depend on uninterrupted access to language models, but provider outages, rate limits, and quota exhaustion are inevitable. OmniRoute, an open-source LLM routing platform by diegosouzapw, solves this through automatic failover—a mechanism that transparently reroutes failed requests to alternative providers without client intervention. This article examines exactly how the system implements this resilience, drawing from the v3.8.50 source code.

The Three Layers of Resilience

OmniRoute's automatic failover operates through three complementary layers that function independently yet integrate during request processing.

Provider-Wide Circuit Breaker

The circuit breaker pattern prevents cascading failures by tracking upstream errors per provider. Located in src/shared/utils/circuitBreaker.ts, this component maintains three states:

• CLOSED: Normal operation; requests flow through
• OPEN: Failure threshold exceeded; all calls short-circuited immediately
• HALF_OPEN: After a configurable back-off timeout, a single probe request tests recovery

import { getCircuitBreaker } from '@/src/shared/utils/circuitBreaker';

const cb = getCircuitBreaker('openai');
console.log(cb.getStatus());
// → { name: 'openai', state: 'CLOSED', failureCount: 0, … }

When failures accumulate, the breaker opens. The state persists in domainState (src/lib/db/domainState.ts) across restarts, ensuring durability. Successful probes transition the breaker back to closed.

Connection Cooldown (Provider-Level)

Individual credentials—not entire providers—can enter cooldown. The open-sse/services/accountFallback.ts module tracks each OAuth token or API key via rateLimitedUntil timestamps:

// Connection cooldown helpers in accountFallback.ts
isProviderInCooldown(connection)      // Check if credential is temporarily blocked
recordProviderCooldown(connection, error)  // Set cooldown on 429/quota errors
clearAccountError(connection)         // Clear on successful use

This granular approach allows one exhausted API key to rest while others for the same provider continue serving traffic.

Model-Level Lockout

Some providers quota-limit specific models. The isModelLocked() and recordModelLockoutFailure() functions (accountFallback.ts) isolate failing models without disabling the entire provider. This proves essential when providers like OpenAI restrict GPT-4 separately from GPT-3.5.

The Combo Engine: Orchestrating Automatic Failover

The combo engine in open-sse/services/combo.ts integrates these layers during request processing. Here's the complete flow:

Step 1: Build the Candidate Pool

buildAutoCandidates() gathers all viable provider-model combinations, annotating each with:

• Circuit breaker state (circuitBreakerState)
• Connection cooldown status (isProviderInCooldown)
• Model lockout flag (isModelLocked)
• Quota percentages and reset windows
• Cost-per-token and latency statistics

The scoring logic in open-sse/services/autoCombo/scoring.ts generates ProviderCandidate objects containing this metadata.

Step 2: Filter Unavailable Candidates

Before attempting any target, the engine applies sequential filters:

// Pseudocode representing the filtering logic in combo.ts
if (cb.getStatus().state === "OPEN") {
  skipProvider();           // Entire provider unavailable
}
if (isProviderInCooldown(connection)) {
  skipConnection();         // Specific credential resting
}
if (isModelLocked(model)) {
  skipModel();              // Single model restricted
}

Step 3: Select and Dispatch

Based on the combo strategy—auto, priority, weighted, or round-robin—the engine orders remaining candidates. The first healthy target receives the request via handleSingleModel.

Step 4: Handle Failures and Fail Over

When a call returns a retry-eligible error (429, 502, 503, quota exceeded), the engine:

1. Records the failure (recordProviderFailure, recordModelLockoutFailure)
2. Updates circuit breaker counters (CircuitBreaker._onFailure)
3. Proceeds to the next candidate in the ranked list

This creates automatic failover—the request transparently reroutes without client awareness.

Step 5: Update State

Successful requests clear connection cooldowns (clearAccountError) and update circuit breaker success counters (_onSuccess). Failed requests persist state to domainState for cross-session durability.

// Example: making a chat request with automatic failover
import { handleComboChat } from '@/open-sse/services/combo';

await handleComboChat({
  body,                    // Standard OpenAI-compatible request
  combo: {
    name: 'default',
    strategy: 'auto',      // Uses full candidate pool with resilience filtering
    config: { maxRetries: 2 },
  },
  handleSingleModel,       // Your upstream fetch implementation
  log,
});

Key Implementation Files

File	Responsibility
src/shared/utils/circuitBreaker.ts	Circuit breaker with state persistence and exponential back-off
open-sse/services/combo.ts	Core combo engine orchestrating failover
open-sse/services/accountFallback.ts	Connection cooldowns and model lockouts
open-sse/services/providerCooldownTracker.ts	Global provider cooldown tracking
open-sse/services/autoCombo/scoring.ts	Candidate scoring with resilience metadata
src/lib/db/domainState.ts	Persistent state storage across restarts

Design Benefits of Lazy State Management

All three resilience layers use lazy evaluation:

• Circuit breakers re-open only when timeouts elapse—no polling required
• Cooldown timestamps become eligible automatically when Date.now() exceeds them
• No background jobs or complex scheduling needed

This prevents "sticky-down" scenarios where temporary outages permanently blacklist providers. The system self-heals without manual intervention.

Summary

• Three independent layers provide defense in depth: circuit breakers (provider), cooldowns (connection), lockouts (model)
• Lazy state management eliminates polling overhead and enables automatic recovery
• The combo engine in combo.ts filters candidates at request time, ensuring automatic failover to the healthiest available target
• Persistent state in domainState.ts maintains resilience across service restarts
• Client-transparent operation means applications receive consistent responses regardless of upstream provider changes

Frequently Asked Questions

How do I reset a circuit breaker after investigating a provider issue?

Import resetAllCircuitBreakers or use getCircuitBreaker(name).reset() from src/shared/utils/circuitBreaker.ts to manually clear state. This is useful after verifying that a provider has recovered from an outage.

What's the difference between connection cooldown and model lockout?

Connection cooldown affects a specific credential (API key or OAuth token) and typically triggers on rate limits or quota exhaustion. Model lockout restricts a single model identifier while the provider's other models remain available—useful when providers impose per-model quotas.

Does automatic failover increase latency?

The failover itself adds minimal overhead—candidate filtering occurs in-memory during buildAutoCandidates(). However, if the primary provider fails and a secondary provider has higher latency, that provider's natural response time applies. The auto strategy weights latency statistics to prefer faster alternatives when available.

What happens if all providers fail?

The combo engine returns a ComboDiagnostics payload containing the attempted candidate pool, failure reasons per target, and recovery hints. This enables debugging and alerts without silent failures.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}