How OmniRoute's Three-Layer Resilience System Works: Circuit Breakers, Cooldowns, and Model Lockouts

OmniRoute protects request throughput through a hierarchical three-layer resilience architecture that isolates failures at the provider, connection, and model levels to prevent cascading outages across AI service integrations.

OmniRoute (available at diegosouzapw/OmniRoute) is an open-source request router designed for AI model providers. Its three-layer resilience system ensures that transient upstream failures, rate limits, or quota exhaustion on specific credentials do not degrade the entire routing pipeline. The implementation spans src/shared/utils/circuitBreaker.ts and open-sse/services/accountFallback.ts, creating a defense-in-depth strategy against provider instability.

Provider-Level Circuit Breaker

The outermost layer resides in src/shared/utils/circuitBreaker.ts (lines 4-17) and implements a classic CLOSED → DEGRADED → OPEN → HALF_OPEN → CLOSED state machine. This protects all connections belonging to a provider (e.g., openai or anthropic) by stopping traffic to any provider experiencing repeated upstream failures.

Failure-type awareness drives the breaker logic. The PROVIDER_FAILURE_ERROR_CODES array defines which HTTP status codes increment the failure threshold, including 408, 429, 500, 502, 503, and 504 (lines 20-27). When failure counts exceed configured thresholds, the breaker transitions to OPEN, rejecting requests immediately until the reset timeout expires.

The implementation features adaptive back-off through the _effectiveResetTimeout mechanism (lines 55-64). After each OPEN → HALF_OPEN → OPEN cycle, the timeout escalates according to a configurable multiplier, preventing aggressive retry storms against still-unhealthy providers.

Per-failure-kind thresholds allow granular control via kindThresholds (lines 78-84). Different failure types—such as rate_limit or quota_exhausted—can trigger distinct thresholds and immediate-open behavior. State persistence ensures that breaker status survives process restarts through saveCircuitBreakerState and loadCircuitBreakerState (lines 19-24).

Helpers such as recordProviderFailure (lines 87-95) and recordProviderSuccess (lines 52-57) automatically deduplicate rapid-fire failures per connection and manage per-provider network error accounting.

Connection-Level Cooldown

While the provider breaker handles aggregate failures, the connection-level cooldown isolates individual credentials or API keys. Implemented in open-sse/services/accountFallback.ts (lines 20-27), this layer tracks a rateLimitedUntil timestamp on each connection record.

When a request fails for a specific key, setConnectionRateLimitUntil advances the timestamp (typically 5 seconds for OAuth connections or 3 seconds for API-key based connections). The router skips any connection where new Date(rateLimitedUntil).getTime() > Date.now(), creating a lazy cooling mechanism that requires no additional housekeeping—connections automatically become eligible once the timestamp passes.

Repeated failures trigger exponential back-off via calculateBackoffCooldown, respecting the maxCooldownMs and maxBackoffSteps values defined in PROVIDER_PROFILE configurations. This prevents hammering a temporarily rate-limited key while allowing faster recovery than provider-level outages.

Model-Level Lockout

The innermost layer addresses per-model quota limitations on specific connections. When a provider uses per-model quotas (e.g., Gemini, Codex), OmniRoute tracks failures per provider:connection:model tuple in open-sse/services/accountFallback.ts (lines 53-61).

The lockModel function creates a lock entry containing the reason, until timestamp, and failureCount. This prevents the router from using a specific model on a particular connection when only that model is unavailable or quota-limited, avoiding the unnecessary disablement of an entire connection for a single model failure.

Lock state is exposed through isModelLocked, getModelLockoutInfo, and clearModelLock for dashboard visibility. An automated cleanup timer (ensureCleanupTimer, lines 17-33) purges expired locks without manual intervention.

How the Three Layers Interact

During request processing, OmniRoute evaluates resilience layers in strict order before attempting a connection. The accountFallback service performs three checks for each routing target:

1. Provider breaker status – isProviderInCooldown (line 57) aborts the target if the provider-level circuit breaker is OPEN.
2. Connection cooldown – The router skips connections with a future rateLimitedUntil timestamp (line 10).
3. Model lockout – isModelLocked (line 36) prevents use of a specific model on that connection.

If a request fails, the system records the failure at the appropriate granularity. recordProviderFailure increments the circuit breaker, setConnectionRateLimitUntil updates the connection timestamp, and recordModelLockoutFailure (lines 19-28) creates or extends a model lock. The router then attempts the next candidate, automatically honoring new cooldowns and locks.

This hierarchical approach ensures that a quota limit on gpt-4 for one API key does not trigger a provider-wide outage, while a total provider failure does not permanently disable individual connections that might recover faster.

Code Examples

Checking Provider Circuit Breaker Status

Before routing, verify whether a provider is currently isolated:

import { isProviderInCooldown } from '@/lib/usage/accountFallback';

if (isProviderInCooldown('openai')) {
  // Skip all OpenAI targets for this request
  return [];
}

Recording Connection-Level Rate Limits

Handle 429 responses by cooling down the specific credential:

import { setConnectionRateLimitUntil } from '@/lib/db/providers';
import { getConnectionById } from '@/lib/db/connections';

async function onRateLimited(connectionId: string) {
  const conn = await getConnectionById(connectionId);
  // 5 seconds base cooldown for OAuth, 3 seconds for API-key
  const cooldownMs = conn.category === 'oauth' ? 5000 : 3000;
  setConnectionRateLimitUntil(connectionId, Date.now() + cooldownMs);
}

Locking Models After Quota Exhaustion

When a specific model hits its quota on a connection:

import { lockModel } from '@/lib/usage/accountFallback';

function handleQuotaExhausted(provider: string, connectionId: string, model: string) {
  // 1 hour lock for quota exhausted
  const ONE_HOUR = 60 * 60 * 1000;
  lockModel(provider, connectionId, model, 'quota_exhausted', ONE_HOUR);
}

Querying Active Model Lockouts

For dashboard monitoring or debugging:

import { getAllModelLockouts } from '@/lib/usage/accountFallback';

const lockouts = getAllModelLockouts();
lockouts.forEach(l => {
  console.log(`${l.provider}:${l.connectionId}:${l.model} locked for ${l.reason}, ${l.remainingMs}ms left`);
});

Summary

• Provider-level circuit breakers in src/shared/utils/circuitBreaker.ts use a finite state machine (CLOSED → DEGRADED → OPEN → HALF_OPEN → CLOSED) to halt traffic to failing providers, with adaptive back-off and persistent state across restarts.
• Connection-level cooldowns managed in open-sse/services/accountFallback.ts isolate individual API keys through lazy timestamp evaluation (rateLimitedUntil), preventing bad credentials from affecting healthy ones.
• Model-level lockouts provide granular protection for per-model quotas, tracking failures per provider:connection:model tuple without disabling entire connections.
• The layers interact hierarchically during request routing, with failure recording automatically escalating protection at the appropriate granularity and recovery occurring independently at each level.

Frequently Asked Questions

How does OmniRoute prevent a single bad API key from disabling an entire provider?

OmniRoute uses connection-level cooldowns to isolate individual credentials. When a specific key returns rate limits or errors, setConnectionRateLimitUntil sets a future timestamp on that connection only (open-sse/services/accountFallback.ts, lines 20-27). The provider-level circuit breaker tracks aggregate failures separately, ensuring that one problematic key does not trip the provider-wide breaker unless the failure pattern indicates a service-wide outage.

What happens when the circuit breaker enters the HALF_OPEN state?

In HALF_OPEN state (defined in src/shared/utils/circuitBreaker.ts, lines 4-17), the breaker allows a limited number of test requests to pass through to probe provider health. If these succeed, the state transitions back to CLOSED; if they fail, it returns to OPEN with an increased _effectiveResetTimeout (lines 55-64). This adaptive back-off prevents retry storms while gradually restoring traffic to recovered services.

Can different failure types trigger different circuit breaker thresholds?

Yes. The kindThresholds configuration (lines 78-84) allows distinct thresholds and immediate-open behavior for specific failure kinds such as rate_limit or quota_exhausted. This enables aggressive circuit breaking for permanent errors like authentication failures while tolerating transient 503 errors with higher thresholds.

How long do model-level lockouts persist?

Model lockouts persist for the duration specified when calling lockModel (lines 53-61), after which the ensureCleanupTimer (lines 17-33) automatically purges expired entries. There is no hardcoded default; the lockout duration is caller-defined (e.g., one hour for quota exhaustion), and the cleanup timer ensures memory does not accumulate stale lockouts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}