Understanding OmniRoute’s Three Resilience Layers and Circuit Breaker Functionality

OmniRoute safeguards every request with a three-layer resilience architecture that isolates failures, protects upstream credentials, and maintains availability through connection-cooldown throttling, per-provider circuit breakers, and an emergency fallback system.

The diegosouzapw/OmniRoute repository implements a robust AI request proxy designed to handle provider instability without compromising user experience or security. Grasping the three resilience layers and circuit breaker functionality is critical for production deployments, as these mechanisms work sequentially to prevent cascading failures, stop credential leakage, and guarantee responses even when all primary providers are exhausted.

Layer 1: Request Queue and Connection Cooldown

The first line of defense prevents a single client from overwhelming a provider with retry storms. In src/lib/resilience/services.ts, OmniRoute utilizes a Bottleneck queue to throttle per-connection calls according to the limits defined in src/lib/resilience/settings.ts.

When a provider returns an HTTP 429 (rate-limit) status, the connection immediately enters a cooldown period specified by resilienceSettings.connectionCooldownMs. Subsequent requests targeting a cooled-down connection are automatically skipped, forcing the router to select the next available provider. This layer protects upstream services from duplicate traffic during transient outages without permanently disabling the provider.

Layer 2: Provider Circuit Breaker

While Layer 1 handles temporary rate limits, the Provider Circuit Breaker addresses sustained failures. Each provider profile defined in src/shared/constants/providerProfiles.ts configures circuitBreakerThreshold (maximum consecutive failures) and circuitBreakerReset (cool-down duration). The generic CircuitBreaker class in src/shared/utils/circuitBreaker.ts tracks these failures via an internal STATE machine (CLOSED vs. OPEN).

When failures exceed the threshold, the circuit transitions to OPEN, and the classifyFailKind utility marks subsequent attempts as circuit-open. In this state, the breaker instantly rejects calls to that provider, preventing wasted resources and credential reuse against a known-bad endpoint. The circuit automatically closes after the circuitBreakerReset timeout expires, allowing controlled retry.

Layer 3: Emergency Fallback

When every candidate provider is exhausted—whether through budget constraints, quota limits, or open circuits—the Emergency Fallback layer guarantees a response. Implemented in open-sse/services/emergencyFallback.ts, this layer checks the EMERGENCY_FALLBACK_FLAG_KEY (exposed in src/lib/resilience/settings.ts) to determine if the feature is enabled.

If activated, the system routes the request to a free fallback model (defaulting to openai/gpt-oss-120b) exactly once per request, tracked via runtimeOptions.emergencyFallbackTried. Crucially, as noted in the "credential-leak guard" comments within src/sse/handlers/chat.ts, this fallback operates without forwarding the original provider’s API keys or authentication tokens, ensuring security even during total provider failure.

How the Three Layers Interact

The resilience stack operates as a sequential pipeline inside the request handler (src/sse/handlers/chat.ts):

1. Rate-Limit Check: The withRateLimit wrapper consults the Bottleneck queue. If the provider is in connectionCooldownMs, the request skips to the next target.
2. Failure Classification: When a provider returns a 429 or persistent 5xx error, classifyFailKind in src/shared/utils/circuitBreaker.ts categorizes the failure.
3. Circuit Evaluation: The CircuitBreaker increments its failure counter. If the circuitBreakerThreshold is breached, the circuit opens and the provider is marked circuit-open.
4. Routing Decision: The combo handler (open-sse/services/combo.ts → handleComboChat) iterates to the next provider profile.
5. Exhaustion Handling: If all targets return budget-exhausted or circuit-open states, the handler invokes EmergencyFallback, which verifies the feature flag and routes to the free model without leaking credentials.
6. Client Response: The final stream is returned to the client, preserving the real error only if the emergency model also fails.

Configuring Resilience via the Management API

Operators can adjust thresholds and reset circuits at runtime without restarting the service.

Resetting All Circuit Breakers

To manually clear tripped circuits via the management endpoint:

curl -X POST http://localhost:20128/api/resilience/reset \
     -H "Authorization: Bearer <MANAGEMENT_TOKEN>"

This invokes resetAllCircuitBreakers() in src/shared/utils/circuitBreaker.ts via the route handler in src/app/api/resilience/reset/route.ts.

Enabling Emergency Fallback at Runtime

Toggle the fallback feature programmatically:

import { setFeatureFlag } from "@omniroute/open-sse/utils/featureFlags";

await setFeatureFlag("EMERGENCY_FALLBACK", true);

The open-sse/services/emergencyFallback.ts service reads this flag from emergencyFallbackFlagCache on each request.

Adjusting Circuit Breaker Thresholds

Modify provider-specific resilience profiles dynamically:

curl -X PATCH http://localhost:20128/api/resilience \
     -H "Authorization: Bearer <MANAGEMENT_TOKEN>" \
     -H "Content-Type: application/json" \
     -d '{
           "providerBreaker": {
             "oauth": { "failureThreshold": 10, "resetTimeoutMs": 120000 }
           }
         }'

The PATCH handler in src/app/api/resilience/route.ts persists these changes to the ResilienceSettings record, which the CircuitBreaker references through PROVIDER_PROFILES in src/shared/constants/providerProfiles.ts.

Summary

• Connection Cooldown (src/lib/resilience/settings.ts): Throttles 429 responses using a Bottleneck queue to prevent retry floods against rate-limited providers.
• Provider Circuit Breaker (src/shared/utils/circuitBreaker.ts): Tracks failures per provider profile and opens circuits when circuitBreakerThreshold is exceeded, stopping credential leakage to failing endpoints.
• Emergency Fallback (open-sse/services/emergencyFallback.ts): Provides a free model response when all providers are exhausted, triggered only once per request and isolated from original provider credentials.
• Interaction Flow: Handlers in src/sse/handlers/chat.ts orchestrate the layers, ensuring that cooldowns are respected before circuits are checked, and circuits are respected before the emergency fallback is invoked.

Frequently Asked Questions

How does the connection cooldown differ from the circuit breaker in OmniRoute?

The connection cooldown (Layer 1) is a short, automatic timeout triggered specifically by HTTP 429 responses to prevent immediate retries against a rate-limited endpoint, whereas the circuit breaker (Layer 2) tracks all failure types (5xx, auth errors, timeouts) across multiple requests and opens only after the configurable circuitBreakerThreshold is reached, typically indicating sustained provider unavailability.

What happens when all configured providers are exhausted?

When every provider returns either a budget-exhausted or circuit-open status, OmniRoute invokes the EmergencyFallback service. If the EMERGENCY_FALLBACK_FLAG_KEY is enabled, it routes the request to a free fallback model (default openai/gpt-oss-120b) exactly once per request without exposing the original providers' credentials, ensuring the end user receives a response rather than a hard failure.

Can I reset a tripped circuit breaker without restarting OmniRoute?

Yes. Send a POST request to /api/resilience/reset with a valid management token. This triggers resetAllCircuitBreakers() in src/shared/utils/circuitBreaker.ts, immediately transitioning all circuits from OPEN to CLOSED and allowing traffic to resume against previously failing providers.

Does the emergency fallback layer expose my provider API keys?

No. The emergency fallback explicitly operates as a credential-leak guard. According to the implementation in src/sse/handlers/chat.ts, when the fallback is invoked, the request is routed to the free model without attaching any authentication headers or tokens from the original provider profiles, ensuring complete isolation between failed premium providers and the emergency response path.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}