OmniRoute's Three-Layer Resilience System: Circuit Breaker, Fallback, and Emergency Explained

OmniRoute protects every request with a three-layer resilience architecture that throttles connection cooldowns, isolates failing providers via circuit breakers, and guarantees responses through an emergency fallback when all targets are exhausted.

The diegosouzapw/OmniRoute repository implements a fault-tolerant routing pipeline designed to prevent cascading failures, safeguard upstream credentials, and maintain availability even when AI providers exhaust quotas or return errors. By combining request-queue throttling, provider circuit breakers, and a credential-isolated emergency fallback, the system ensures that a single point of failure never degrades the entire user experience.

Layer 1: Connection Cooldown and Request Throttling

The first layer prevents a flood of retries from overwhelming a provider by implementing per-connection rate limiting. In src/lib/resilience/services.ts, a Bottleneck queue throttles calls for each connection based on the resilienceSettings.connectionCooldownMs value defined in src/lib/resilience/settings.ts.

When a provider returns a 429 (rate-limit) status, the connection enters a cooldown period. New requests targeting a cooled-down connection are immediately skipped, and the router attempts the next available provider. This protects upstream services from being hammered during transient overloads while preserving client resources.

Layer 2: Provider Circuit Breaker

The second layer stops repeated failing calls to a faulty provider from consuming resources and leaking credentials. Each provider profile in src/shared/constants/providerProfiles.ts defines two critical thresholds: circuitBreakerThreshold (maximum consecutive failures) and circuitBreakerReset (cool-down time in milliseconds).

The generic CircuitBreaker class in src/shared/utils/circuitBreaker.ts tracks failures and state transitions between CLOSED (normal operation) and OPEN (failing fast). When failures exceed the threshold, the circuit opens and refuses further calls until the reset timeout expires. The classifyFailKind function marks these failures as circuit-open, signaling the routing engine to bypass the provider instantly.

Layer 3: Emergency Fallback

The final layer guarantees a response when all targets are exhausted due to budget limits, quota restrictions, or open circuits. The EmergencyFallback service in open-sse/services/emergencyFallback.ts activates when the routing layer detects a budget-exhausted or circuit-open error.

Controlled by the EMERGENCY_FALLBACK_FLAG_KEY (exposed in src/lib/resilience/settings.ts), this layer routes the request to a free fallback model—by default openai/gpt-oss-120b. The runtimeOptions.emergencyFallbackTried flag ensures the fallback is invoked only once per request, and the implementation explicitly prevents credential leakage by never forwarding the original provider’s API keys or tokens.

How the Layers Interact

The resilience system operates as a cascading filter during the request lifecycle:

1. Incoming request enters through src/sse/handlers/chat.ts.
2. Rate-limit queue (withRateLimit) attempts the first target. If the provider is in connection cooldown, the queue skips it immediately.
3. If the provider returns a 429 or a circuit-open pattern, classifyFailKind in src/shared/utils/circuitBreaker.ts records the failure and triggers the circuit breaker if thresholds are met.
4. The routing engine (open-sse/services/combo.ts via handleComboChat) selects the next candidate provider.
5. When all targets report budget-exhausted or circuit-open, the EmergencyFallback service checks the feature flag and routes to the free model without exposing original credentials.
6. The response streams back to the client. If the emergency model also fails, the system returns the original error to ensure transparency.

Configuring Resilience Settings

Resetting Circuit Breakers via API

To manually reset all provider circuits after an outage, use the management API endpoint defined in src/app/api/resilience/reset/route.ts:

curl -X POST http://localhost:20128/api/resilience/reset \
     -H "Authorization: Bearer <MANAGEMENT_TOKEN>"

This invokes resetAllCircuitBreakers() from src/shared/utils/circuitBreaker.ts, clearing failure counts and returning all circuits to the CLOSED state.

Enabling Emergency Fallback at Runtime

Toggle the emergency fallback feature without restarting the service:

import { setFeatureFlag } from "@omniroute/open-sse/utils/featureFlags";

await setFeatureFlag("EMERGENCY_FALLBACK", true);

The open-sse/services/emergencyFallback.ts module caches this flag value in emergencyFallbackFlagCache and evaluates it on every request to determine if the free fallback model should be used.

Adjusting Circuit-Breaker Thresholds

Modify provider-specific resilience profiles via the PATCH endpoint in src/app/api/resilience/route.ts:

curl -X PATCH http://localhost:20128/api/resilience \
     -H "Authorization: Bearer <MANAGEMENT_TOKEN>" \
     -H "Content-Type: application/json" \
     -d '{
           "providerBreaker": {
             "oauth": { "failureThreshold": 10, "resetTimeoutMs": 120000 }
           }
         }'

This updates the persisted ResilienceSettings record, which the circuit breaker reads via PROVIDER_PROFILES in src/shared/constants/providerProfiles.ts.

Summary

• Connection Cooldown (src/lib/resilience/services.ts) prevents provider overload by throttling requests after 429 responses using resilienceSettings.connectionCooldownMs.
• Circuit Breaker (src/shared/utils/circuitBreaker.ts) isolates flaky providers after configurable failure thresholds, protecting credentials and resources.
• Emergency Fallback (open-sse/services/emergencyFallback.ts) guarantees service continuity with a free model when all providers are exhausted, ensuring emergencyFallbackTried prevents infinite loops.
• All layers are configurable via the Resilience API (src/app/api/resilience/route.ts) and respect the EMERGENCY_FALLBACK_FLAG_KEY feature flag.

Frequently Asked Questions

How does OmniRoute prevent credential leakage during provider failures?

The emergency fallback implementation in open-sse/services/emergencyFallback.ts explicitly strips original provider credentials before routing to the fallback model openai/gpt-oss-120b. As noted in the credential-leak guard comments within src/sse/handlers/chat.ts, the system never forwards API keys or tokens from exhausted providers to the emergency service, ensuring that authentication secrets remain isolated to their intended targets.

What triggers the circuit breaker to open?

The circuit breaker opens when consecutive failures exceed the circuitBreakerThreshold defined in src/shared/constants/providerProfiles.ts. Failures are classified by the classifyFailKind function in src/shared/utils/circuitBreaker.ts, which tracks 5xx errors, malformed responses, and authentication failures. Once the threshold is reached, the circuit transitions to OPEN and remains in that state for the duration specified by circuitBreakerReset (in milliseconds).

Can I customize the emergency fallback model?

Yes. While the default emergency model is openai/gpt-oss-120b, you can configure alternative endpoints by modifying the fallback resolver logic in open-sse/services/emergencyFallback.ts. Ensure that any custom model does not require the original provider's credentials, as the system explicitly disables credential forwarding to prevent leakage during fallback execution.

How do I reset a tripped circuit breaker programmatically?

Use the resetAllCircuitBreakers() utility exported from src/shared/utils/circuitBreaker.ts, which is exposed via the POST endpoint at src/app/api/resilience/reset/route.ts. This clears the failure counters for all providers, immediately transitioning their circuits from OPEN to CLOSED and restoring normal traffic flow to previously isolated targets.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}