How OmniRoute's Key Rotation and Failover Mechanism Works Across Provider Accounts

OmniRoute protects request throughput through three complementary layers: per-account rotation with configurable thresholds, connection-level exponential backoff, and provider-wide circuit breakers that automatically isolate failing credentials.

OmniRoute is an open-source AI gateway that intelligently routes requests across multiple LLM providers. Its key rotation and failover mechanism ensures high availability by automatically retiring compromised accounts, cooling down rate-limited keys, and blocking entire providers when systemic failures occur. This article explains how these systems work based on the source code in diegosouzapw/OmniRoute.

The Three-Layer Resilience Architecture

OmniRoute implements defense in depth through three coordinated mechanisms:

Mechanism	Purpose	Core Implementation
Per-account rotation (fallback)	Decides when to retire an account and try a fresh credential	open-sse/services/accountFallback.ts — functions gateFor, recordErrorAndCheckThreshold, evaluateRotationGate
Connection-level cooldown	Applies exponential backoff to the specific key that failed	accountFallback.ts — recordProviderFailure, calculateBackoffCooldown
Provider-wide circuit breaker	Blocks all keys for a provider when aggregated failures exceed thresholds	accountFallback.ts with src/shared/utils/circuitBreaker.ts

Runtime Rotation Configuration

All rotation behavior is controlled through rotationConfig.ts, which reads environment variables and supports per-connection overrides.

// open-sse/services/rotationConfig.ts
export function getGlobalRotationConfig(): RotationConfig { … }

Environment Variable Controls

Variable	Default	Behavior
OMNIROUTE_ROTATE_ON_429	true	Enable rotation on rate-limit (429) errors
OMNIROUTE_ROTATE_429_THRESHOLD	1	Errors required before rotation triggers
OMNIROUTE_ROTATE_429_WINDOW_SECONDS	120	Sliding window for error counting

Per-connection overrides via providerSpecificData.rotationOverrides allow fine-tuning without restarts. See resolveRotationConfig (lines 60-88) for the merge logic.

Decision Flow for Failed Requests

When a request fails, accountFallback.ts executes this sequence:

// open-sse/services/accountFallback.ts
export function gateFor(status: number, account?: unknown) {
  const { rotationOverrides, rotationKey } = extractRotationContext(account);
  return evaluateRotationGate(
    status,
    resolveRotationConfig(rotationOverrides),
    rotationKey,
  );
}

The evaluateRotationGate function applies three checks in order:

1. Master switch validation — isFallbackBlockedForStatus returns true if rotation is globally disabled for this error class
2. Threshold sliding window — recordErrorAndCheckThreshold counts errors per (key, status) pair; rotation triggers when threshold errors occur within windowMs
3. 400-force fallback — If rotateOn400 is enabled, bad requests can trigger rotation using rateLimitCooldownOverrideMs

If all checks pass without triggering rotation, OmniRoute falls back to built-in heuristics like parsing Retry-After headers.

Cooling Down Individual Keys

Rotated credentials enter a cooldown period computed by calculateBackoffCooldown in config/errorConfig.ts:

backoffMs = min(baseCooldownMs * 2^(failureCount - 1), maxBackoffMs)

The cooldown state is dual-tracked:

• In-memory: Via CircuitBreaker instance per provider
• Persistent: In providers.rateLimitedUntil database column

// open-sse/services/accountFallback.ts
export function recordProviderFailure(
  provider: string | null | undefined,
  log?,
  connectionId?,
  profile?,
  opts?,
) {
  const breaker = configureProviderBreaker(provider, profile);
  if (!breaker?.canExecute()) return;
  breaker._onFailure();  // Updates circuit-breaker state
}

Provider-Wide Circuit Breaker

Separate from per-key cooldowns, the circuit breaker aggregates failures across all keys for a provider. It deduplicates errors using:

// Constants in accountFallback.ts (lines 28-42)
const CONNECTION_FAILURE_DEDUP_MS = 5000;    // 5 seconds
const NETWORK_ERROR_DEDUP_MS = 10000;        // 10 seconds

Only distinct failures contribute to the threshold. When failureThreshold is exceeded, the breaker opens and triggers providerCooldownMs:

export function isProviderInCooldown(provider: string | null | undefined): boolean {
  const breaker = getProviderBreaker(provider);
  return breaker ? !breaker.canExecute() : false;
}

While OPEN, recordProviderSuccess does not reset the cooldown — the provider remains blocked until resetTimeoutMs expires.

Per-Model Lockouts for Multiplexed Providers

Providers like Gemini and Codex multiplex multiple models on single credentials. OmniRoute can lock individual models without disabling the entire connection:

• lockModelIfPerModelQuota — applies model-specific lockout
• isModelLocked — checks lockout status before routing
• recordModelLockoutFailure — triggers exponential backoff per model

The hasPerModelQuota function (lines 145-162) determines whether a provider uses this behavior. Lockout entries are evicted by lockoutEviction.ts to prevent unbounded memory growth.

Complete Rotation and Failover Pipeline

1. Request arrives → API route dispatches to accountFallback.ts
2. Error classification → classifyErrorText and RateLimitReason categorize the failure
3. Rotation gate → gateFor decides: rotate now, hold, or continue
4. If rotation required:
   • Mark credential cooldown via recordProviderFailure
   • Update provider circuit breaker if failure contributes to threshold
   • Optionally lock specific model via lockModelIfPerModelQuota
5. Router selection → open-sse/services/combo.ts skips cooled credentials and selects next viable key

Practical Code Examples

Force Rotation on 429 for a Specific Account

import { gateFor } from '@/open-sse/services/accountFallback';
import { getGlobalRotationConfig } from '@/open-sse/services/rotationConfig';

const account = {
  id: 'account-123',
  providerSpecificData: {
    rotationOverrides: {
      rotateOn429: true,
      error429Threshold: 2,
      error429WindowSeconds: 60,
    },
  },
};

const decision = gateFor(429, account);
if (decision?.shouldFallback) {
  console.log('Rotate this key, cooldown:', decision.cooldownMs);
}

Check Provider Status Before Dispatching

import { isProviderInCooldown, getProviderCooldownRemainingMs } from '@/open-sse/services/accountFallback';

const provider = 'openai';
if (isProviderInCooldown(provider)) {
  const remaining = getProviderCooldownRemainingMs(provider);
  console.warn(`Provider ${provider} is throttled for ${remaining} ms`);
} else {
  // Safe to send request
}

Key Source Files

File	Purpose
open-sse/services/accountFallback.ts	Core fallback engine, rotation gate, circuit-breaker integration, model lockouts
open-sse/services/rotationConfig.ts	Runtime configuration from env vars and per-connection overrides
src/shared/utils/circuitBreaker.ts	Shared circuit-breaker implementation
open-sse/services/accountFallback/lockoutEviction.ts	Stale model-lockout cleanup
open-sse/services/accountFallback/exactModelLock.ts	Model-specific lockout key utilities
src/lib/resilience/settings.ts	Default resilience parameters

Summary

• Per-account rotation uses configurable thresholds and sliding windows to retire failing credentials, implemented in accountFallback.ts with rules from rotationConfig.ts

• Connection-level cooldown applies exponential backoff to individual keys, storing state in both memory (CircuitBreaker) and database (providers.rateLimitedUntil)

• Provider-wide circuit breaker blocks all keys when aggregated failures exceed thresholds, with deduplication preventing false triggers

• Per-model lockouts enable granular control for multiplexed providers without disabling entire connections

• Runtime configurability via environment variables and rotationOverrides allows operators to tune behavior without code changes

Frequently Asked Questions

What triggers key rotation in OmniRoute?

Key rotation triggers when error counts exceed configured thresholds within a sliding time window. By default, a single 429 error rotates the key (OMNIROUTE_ROTATE_429_THRESHOLD=1), but this is configurable per status code. The gateFor function in accountFallback.ts evaluates these rules against each failed request.

How does OmniRoute handle rate limits differently from quota exhaustion?

Rate limits (typically 429) trigger rotation or cooldown based on configuration, while quota exhaustion may be detected through provider-specific error text classification. The classifyErrorText function in accountFallback.ts parses error messages to distinguish transient rate limits from permanent quota depletion, applying different rotation policies accordingly.

Can a single failing model disable an entire provider account?

No. For providers identified by hasPerModelQuota (Gemini, Codex), OmniRoute uses per-model lockouts via lockModelIfPerModelQuota. Only the specific model enters cooldown while other models on the same credential remain available. This is essential for providers that multiplex dozens of models through single API keys.

What happens when all accounts for a provider are in cooldown?

The provider-wide circuit breaker in isProviderInCooldown returns true, causing the router in combo.ts to skip that provider entirely. Requests then fail over to alternative providers in the configured pool. If no providers are available, the gateway returns an appropriate error to the client rather than queueing indefinitely.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}