OmniRoute 3-Layer Resilience System: How Circuit Breaker, Connection Cooldown, and Model Lockout Work

OmniRoute uses three independent resilience layers—Provider Circuit Breaker, Connection Cooldown, and Model Lockout—to isolate failures at the provider, credential, and model levels, ensuring automatic recovery without manual intervention.

The 3-layer resilience system in the diegosouzapw/OmniRoute repository protects request routing by operating at distinct scopes to handle different failure classes. Each layer maintains its own state machine and recovery logic, allowing the system to gracefully degrade when upstream AI providers experience outages, rate limits, or model-specific errors. Understanding how these mechanisms interact is essential for operating a production OmniRoute deployment.

Layer 1: Provider Circuit Breaker

The Provider Circuit Breaker guards against repeated upstream service errors that affect all API keys of a specific provider, such as openai or anthropic.

State Transitions and Failure Thresholds

Implemented in src/shared/utils/circuitBreaker.ts, the breaker tracks provider-level failures across all connections. When the failure count crosses the failureThreshold, the state transitions from CLOSED → DEGRADED → OPEN. While OPEN, the provider is completely excluded from combo routing.

Configuration defaults vary by provider type:

• OAuth: Degrades at 5 failures, opens at 8 failures, resets after 60 seconds
• API-key: Degrades at 7 failures, opens at 12 failures, resets after 30 seconds
• Local: Opens at 2 failures, resets after 15 seconds

Lazy Recovery Mechanism

Recovery occurs lazily when getStatus(), canExecute(), or getRetryAfterMs() is called after the resetTimeout expires. This automatically moves the breaker to HALF_OPEN, allowing a single probe request. A successful probe closes the breaker; another failure re-opens it.

import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';
import { Provider } from '@/shared/constants/providers';

async function logProviderStatus(provider: Provider) {
  const cb = await getCircuitBreaker(provider);
  console.log(`${provider} → state: ${cb.state}, failures: ${cb.failureCount}`);
}

// Example: check OpenAI status
await logProviderStatus('openai');

To manually reset a breaker via the admin API:

curl -X POST \
  -H "Authorization: Bearer $ADMIN_JWT" \
  https://localhost:20128/api/resilience/reset \
  -d '{"provider":"anthropic"}' \
  -H "Content-Type: application/json"

Implementation reference: src/sse/handlers/chatHelpers.ts → resetCircuitBreaker()

Layer 2: Connection Cooldown

The Connection Cooldown layer isolates transient errors specific to a single credential or API key, such as rate limits, temporary authentication failures, or network hiccups.

Exponential Back-off Strategy

When checkFallbackError() in open-sse/services/accountFallback.ts classifies an error as transient, the connection is marked unavailable via markAccountUnavailable(). The system sets a rateLimitedUntil timestamp calculated as baseCooldownMs * 2ⁿ, where base values are 5 seconds for OAuth keys and 3 seconds for API-key keys.

import { canExecute } from '@/sse/services/auth';

async function executeIfAvailable(conn) {
  if (!canExecute(conn)) {
    console.warn(`Connection ${conn.id} is in cooldown until ${new Date(conn.rateLimitedUntil)}`);
    return; // skip or fallback
  }
  // normal executor logic …
}

Helper defined in src/sse/services/auth.ts::canExecute

Integration with Request Routing

During the routing phase in src/sse/handlers/chat.ts, connections with a future rateLimitedUntil timestamp are filtered from the candidate pool. Recovery happens automatically once the timestamp passes, and a successful request clears all error fields via clearAccountError().

Layer 3: Model Lockout

The Model Lockout layer handles failures isolated to a specific provider-connection-model triple, such as per-model quota exhaustion or missing model permissions.

Per-Model Failure Isolation

When recordModelLockoutFailure() detects a model-scoped error, it invokes lockModel() in open-sse/services/accountFallback.ts to create a lockout entry. This prevents subsequent requests from attempting that specific model on that connection for a configurable duration.

Success-Decay Recovery

Model lockout offers two recovery paths:

1. Timer expiration: The lockout automatically expires after the cooldown period
2. Success-decay: A successful request triggers decayModelFailureCount(), which halves the failure count and eventually deletes the lockout entry entirely

List active lockouts via the REST endpoint:

curl -H "Authorization: Bearer $ADMIN_JWT" \
  https://localhost:20128/api/resilience/model-cooldowns

Endpoint implemented in src/app/api/v1/resilience/model-cooldowns/route.ts

Clear a specific lockout manually:

curl -X DELETE \
  -H "Authorization: Bearer $ADMIN_JWT" \
  -H "Content-Type: application/json" \
  -d '{"provider":"openai","connectionId":"abc123","model":"gpt-4o"}' \
  https://localhost:20128/api/resilience/model-cooldowns

Lockout removal logic lives in open-sse/services/accountFallback.ts::clearModelLock()

How the Layers Interact in the Routing Pipeline

When OmniRoute evaluates a request, it applies the three layers in strict sequence:

1. Circuit Breaker – If the provider state is OPEN, the provider is omitted from combo candidates entirely
2. Connection Cooldown – For remaining providers, connections with future rateLimitedUntil timestamps are filtered out
3. Model Lockout – For surviving connections, currently locked models are excluded

Only if all connections of a provider are filtered does the combo engine fall back to the next provider or invoke emergency fallback logic. This hierarchical filtering ensures that model-specific issues don't trigger unnecessary provider-wide circuit breaks.

Configuration Defaults and Tuning

Default settings are defined across several configuration files:

Layer	Configuration File	Key Parameters
Circuit Breaker	src/lib/resilience/settings.ts	Thresholds and timeouts per provider type
Connection Cooldown	src/lib/resilience/settings.ts	baseCooldownMs (5s OAuth, 3s API-key)
Model Lockout	src/lib/resilience/modelLockoutSettings.ts	Enabled flag, error codes, duration parameters

Model lockout defaults to enabled: false and must be explicitly activated in modelLockoutSettings.ts before the system creates per-model lockout entries.

Debugging and Operational Commands

When troubleshooting why a provider is being skipped:

• All keys for a provider are skipped – Check the circuit breaker state in domain_circuit_breakers and each connection's rateLimitedUntil/testStatus
• Only one key fails – Investigate connection cooldown in src/sse/services/auth.ts; this indicates a credential-specific issue rather than a provider-wide outage
• Only one model fails – Search for lockout entries in open-sse/services/accountFallback.ts using the provider-connection-model triple
• State should have recovered but didn't – Verify code is reading status via getStatus() or canExecute() helpers, which trigger lazy recovery. Reading raw database columns directly will not refresh expired entries.

Summary

• Provider Circuit Breaker operates at the provider level with states CLOSED, DEGRADED, OPEN, and HALF_OPEN, using lazy recovery after resetTimeout expires
• Connection Cooldown manages per-credential back-off with exponential growth (baseCooldownMs * 2ⁿ), filtering unavailable connections during routing
• Model Lockout isolates per-model failures with dual recovery paths: timer expiration and success-decay via decayModelFailureCount()
• The routing pipeline evaluates layers sequentially: breaker first, then cooldown, then lockout
• Configuration files in src/lib/resilience/ control thresholds, with different defaults for OAuth, API-key, and Local provider types

Frequently Asked Questions

How do I know which resilience layer is blocking my request?

Check the symptoms: if all keys for a provider are skipped, the Circuit Breaker is open; if only one key fails, it is a Connection Cooldown; if only one model fails on an otherwise working key, it is Model Lockout. Inspect domain_circuit_breakers for breaker states, rateLimitedUntil timestamps for cooldowns, and the model-cooldowns API for lockouts.

What triggers the transition from DEGRADED to OPEN in the circuit breaker?

The breaker moves from DEGRADED to OPEN when the failureCount reaches the provider-specific open threshold (8 for OAuth, 12 for API-key). While DEGRADED, the provider still receives traffic but is marked as unstable; once OPEN, it is excluded from routing entirely until the resetTimeout triggers lazy recovery.

Can I disable the model lockout layer?

Yes. Model lockout defaults to enabled: false in src/lib/resilience/modelLockoutSettings.ts. When disabled, model-specific errors fall through to the Connection Cooldown or Circuit Breaker layers instead of creating per-model lockout entries.

Why is my provider still marked as OPEN after the reset timeout passed?

OmniRoute uses lazy recovery: the breaker only checks the timeout when getStatus(), canExecute(), or getRetryAfterMs() is invoked. If you are checking raw database columns directly without calling these helper functions, the state machine will not transition to HALF_OPEN. Use the provided utility functions to ensure proper state evaluation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}