OmniRoute 3-Layer Resilience Model: How Circuit Breaker, Connection Cooldown, and Model Lockout Work

OmniRoute's three-layer resilience architecture isolates failures at the provider, connection, and model levels through hierarchical circuit breaking, exponential backoff cooldowns, and success-decay lockouts to ensure continuous LLM service availability.

The diegosouzapw/OmniRoute repository implements this sophisticated OmniRoute 3-layer resilience model to handle cascading failures in multi-provider LLM routing. By separating concerns across provider-wide outages, individual API credential issues, and per-model quotas, the system minimizes downtime and automates recovery without manual intervention.

Layer 1: Provider Circuit Breaker

The Provider Circuit Breaker operates at the entire provider scope (e.g., openai, anthropic), guarding against repeated upstream service errors such as 5xx responses and timeouts that affect all API keys for that provider.

According to the source code in src/shared/utils/circuitBreaker.ts, the breaker maintains four states: CLOSED, DEGRADED, OPEN, and HALF_OPEN. When the failure count crosses the configurable failureThreshold, the state transitions from CLOSED → DEGRADED → OPEN. While OPEN, the provider is completely excluded from combo routing.

Recovery happens lazily. When code calls getStatus(), canExecute(), or getRetryAfterMs() after the resetTimeout expires, the breaker automatically moves to HALF_OPEN and allows a probe request. A successful probe closes the breaker; another failure re-opens it.

Default thresholds vary by provider type:

• OAuth: Degraded at 5 failures, opens at 8 failures, resets after 60 seconds
• API-key: Degraded at 7 failures, opens at 12 failures, resets after 30 seconds
• Local: Opens at 2 failures, resets after 15 seconds

Layer 2: Connection Cooldown

The Connection Cooldown layer isolates transient errors specific to a single API key or account, such as rate limits, temporary authentication errors, or network hiccups. This prevents one bad credential from triggering a provider-wide circuit breaker.

When checkFallbackError() in open-sse/services/accountFallback.ts classifies an error as transient, the system calls markAccountUnavailable() to set a back-off timestamp (rateLimitedUntil). The cooldown duration grows exponentially using baseCooldownMs * 2ⁿ, starting at 5 seconds for OAuth keys and 3 seconds for API-key keys.

Recovery is lazy: once rateLimitedUntil passes, the connection becomes eligible again. A successful request clears all error fields via clearAccountError(). The helper function canExecute() in src/sse/services/auth.ts checks eligibility without requiring manual intervention.

Layer 3: Model Lockout

The Model Lockout mechanism targets failures isolated to a specific model within a provider and connection triple (provider + connection ID + model name). This handles per-model quotas, missing models, or mode-specific permission errors.

When recordModelLockoutFailure() detects a model-scoped error, it invokes lockModel() to create a temporary lockout entry. Unlike the circuit breaker, model lockouts implement success-decay recovery via decayModelFailureCount(), which halves the failure count on successful requests and eventually deletes the lockout entry entirely.

Model lockout settings are defined in src/lib/resilience/modelLockoutSettings.ts, with default configurations allowing operators to enable or disable the feature and define specific error codes that trigger lockouts.

How the Layers Interact During Routing

When OmniRoute processes a request, it evaluates resilience layers in strict hierarchical order:

1. Circuit Breaker Check: If the provider state is OPEN, the provider is omitted from the candidate list entirely.
2. Connection Cooldown Filter: For remaining providers, connections with rateLimitedUntil timestamps in the future are filtered out.
3. Model Lockout Exclusion: For surviving connections, models with active lockout entries are excluded.

Only if all connections for a provider are filtered out does the combo engine fall back to the next provider or invoke emergency fallback logic. This sequential evaluation ensures that granular issues (single model) don't trigger broader isolation (entire provider) unless necessary.

Inspecting and Managing Resilience State

Checking Provider Circuit Breaker Status

To programmatically inspect circuit breaker states:

import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';
import { Provider } from '@/shared/constants/providers';

async function logProviderStatus(provider: Provider) {
  const cb = await getCircuitBreaker(provider);
  console.log(`${provider} → state: ${cb.state}, failures: ${cb.failureCount}`);
}

// Example: check OpenAI status
await logProviderStatus('openai');

Resetting Breakers via Admin API

To manually reset a provider's circuit breaker:

curl -X POST \
  -H "Authorization: Bearer $ADMIN_JWT" \
  https://localhost:20128/api/resilience/reset \
  -d '{"provider":"anthropic"}' \
  -H "Content-Type: application/json"

This endpoint is implemented in src/sse/handlers/chatHelpers.ts via the resetCircuitBreaker() function.

Detecting Connection Cooldowns

Use the canExecute() helper to check connection availability:

import { canExecute } from '@/sse/services/auth';

async function executeIfAvailable(conn) {
  if (!canExecute(conn)) {
    console.warn(`Connection ${conn.id} is in cooldown until ${new Date(conn.rateLimitedUntil)}`);
    return; // skip or fallback
  }
  // normal executor logic …
}

Listing Active Model Lockouts

Query active model lockouts via the REST API:

curl -H "Authorization: Bearer $ADMIN_JWT" \
  https://localhost:20128/api/resilience/model-cooldowns

This endpoint is defined in src/app/api/v1/resilience/model-cooldowns/route.ts.

Clearing Specific Model Lockouts

To manually clear a model lockout:

curl -X DELETE \
  -H "Authorization: Bearer $ADMIN_JWT" \
  -H "Content-Type: application/json" \
  -d '{"provider":"openai","connectionId":"abc123","model":"gpt-4o"}' \
  https://localhost:20128/api/resilience/model-cooldowns

The underlying logic resides in open-sse/services/accountFallback.ts in the clearModelLock() function.

Summary

• Provider Circuit Breakers protect entire LLM providers from cascading failures using state machines (CLOSED → DEGRADED → OPEN → HALF_OPEN) with lazy recovery mechanisms implemented in src/shared/utils/circuitBreaker.ts.
• Connection Cooldowns isolate individual API keys through exponential back-off strategies, preventing single-credential issues from affecting provider-wide availability.
• Model Lockouts provide granular protection at the model level using success-decay algorithms that automatically heal after successful requests.
• The three layers operate sequentially during request routing, ensuring that isolation occurs at the most specific applicable scope.
• All layers support both automatic lazy recovery and manual administrative intervention via REST APIs and database queries.

Frequently Asked Questions

How do I determine which resilience layer is blocking my requests?

Check the scope of the failure. If all keys for a provider are skipped, inspect the circuit breaker state in domain_circuit_breakers. If only one API key fails, examine the connection's rateLimitedUntil field for a cooldown issue. If only a specific model fails, look for lockout entries in open-sse/services/accountFallback.ts. Reading raw database columns won't trigger lazy recovery; use helper functions like getStatus() or canExecute() instead.

Why does the circuit breaker use lazy recovery instead of scheduled jobs?

Lazy recovery reduces system complexity and resource usage. The breaker transitions from OPEN to HALF_OPEN only when getStatus(), canExecute(), or getRetryAfterMs() is called after the resetTimeout expires. This ensures that recovery checks happen precisely when needed—during actual routing decisions—rather than consuming resources with background polling.

Can I adjust the exponential backoff for connection cooldowns?

Yes. The base cooldown values are defined in src/lib/resilience/settings.ts—5 seconds for OAuth keys and 3 seconds for API-key keys. The backoff follows the formula baseCooldownMs * 2ⁿ where n is the retry count. Modify these base values or the classification logic in checkFallbackError() to customize cooldown behavior for specific error types.

What triggers a model lockout versus a connection cooldown?

Connection cooldowns handle transient, credential-specific errors like HTTP 429 rate limits or temporary network failures detected by checkFallbackError(). Model lockouts trigger on persistent model-specific errors such as quota exhaustion for individual models (recordModelLockoutFailure()), missing model IDs, or mode-specific permission denials that affect only the provider+connection+model triple.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}