# Understanding OmniRoute’s Three Resilience Layers and Circuit Breaker Functionality

> Explore OmniRoute's three resilience layers and circuit breaker functionality. Safeguard requests, isolate failures, and maintain availability with this robust architecture.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: architecture
- Published: 2026-07-27

---

**OmniRoute safeguards every request with a three-layer resilience architecture that isolates failures, protects upstream credentials, and maintains availability through connection-cooldown throttling, per-provider circuit breakers, and an emergency fallback system.**

The diegosouzapw/OmniRoute repository implements a robust AI request proxy designed to handle provider instability without compromising user experience or security. Grasping the **three resilience layers and circuit breaker functionality** is critical for production deployments, as these mechanisms work sequentially to prevent cascading failures, stop credential leakage, and guarantee responses even when all primary providers are exhausted.

## Layer 1: Request Queue and Connection Cooldown

The first line of defense prevents a single client from overwhelming a provider with retry storms. In [`src/lib/resilience/services.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/services.ts), OmniRoute utilizes a **Bottleneck** queue to throttle per-connection calls according to the limits defined in [`src/lib/resilience/settings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/settings.ts).

When a provider returns an HTTP **429** (rate-limit) status, the connection immediately enters a cooldown period specified by `resilienceSettings.connectionCooldownMs`. Subsequent requests targeting a cooled-down connection are automatically skipped, forcing the router to select the next available provider. This layer protects upstream services from duplicate traffic during transient outages without permanently disabling the provider.

## Layer 2: Provider Circuit Breaker

While Layer 1 handles temporary rate limits, the **Provider Circuit Breaker** addresses sustained failures. Each provider profile defined in [`src/shared/constants/providerProfiles.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/constants/providerProfiles.ts) configures `circuitBreakerThreshold` (maximum consecutive failures) and `circuitBreakerReset` (cool-down duration). The generic **CircuitBreaker** class in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) tracks these failures via an internal `STATE` machine (CLOSED vs. OPEN).

When failures exceed the threshold, the circuit transitions to **OPEN**, and the `classifyFailKind` utility marks subsequent attempts as `circuit-open`. In this state, the breaker instantly rejects calls to that provider, preventing wasted resources and credential reuse against a known-bad endpoint. The circuit automatically closes after the `circuitBreakerReset` timeout expires, allowing controlled retry.

## Layer 3: Emergency Fallback

When every candidate provider is exhausted—whether through budget constraints, quota limits, or open circuits—the **Emergency Fallback** layer guarantees a response. Implemented in [`open-sse/services/emergencyFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/emergencyFallback.ts), this layer checks the `EMERGENCY_FALLBACK_FLAG_KEY` (exposed in [`src/lib/resilience/settings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/settings.ts)) to determine if the feature is enabled.

If activated, the system routes the request to a free fallback model (defaulting to `openai/gpt-oss-120b`) exactly **once per request**, tracked via `runtimeOptions.emergencyFallbackTried`. Crucially, as noted in the "credential-leak guard" comments within [`src/sse/handlers/chat.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/handlers/chat.ts), this fallback operates without forwarding the original provider’s API keys or authentication tokens, ensuring security even during total provider failure.

## How the Three Layers Interact

The resilience stack operates as a sequential pipeline inside the request handler ([`src/sse/handlers/chat.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/handlers/chat.ts)):

1. **Rate-Limit Check**: The `withRateLimit` wrapper consults the Bottleneck queue. If the provider is in `connectionCooldownMs`, the request skips to the next target.
2. **Failure Classification**: When a provider returns a 429 or persistent 5xx error, `classifyFailKind` in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) categorizes the failure.
3. **Circuit Evaluation**: The CircuitBreaker increments its failure counter. If the `circuitBreakerThreshold` is breached, the circuit opens and the provider is marked `circuit-open`.
4. **Routing Decision**: The combo handler ([`open-sse/services/combo.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/combo.ts) → `handleComboChat`) iterates to the next provider profile.
5. **Exhaustion Handling**: If all targets return *budget-exhausted* or *circuit-open* states, the handler invokes EmergencyFallback, which verifies the feature flag and routes to the free model without leaking credentials.
6. **Client Response**: The final stream is returned to the client, preserving the real error only if the emergency model also fails.

## Configuring Resilience via the Management API

Operators can adjust thresholds and reset circuits at runtime without restarting the service.

### Resetting All Circuit Breakers

To manually clear tripped circuits via the management endpoint:

```bash
curl -X POST http://localhost:20128/api/resilience/reset \
     -H "Authorization: Bearer <MANAGEMENT_TOKEN>"

```

This invokes `resetAllCircuitBreakers()` in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) via the route handler in [`src/app/api/resilience/reset/route.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/app/api/resilience/reset/route.ts).

### Enabling Emergency Fallback at Runtime

Toggle the fallback feature programmatically:

```typescript
import { setFeatureFlag } from "@omniroute/open-sse/utils/featureFlags";

await setFeatureFlag("EMERGENCY_FALLBACK", true);

```

The [`open-sse/services/emergencyFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/emergencyFallback.ts) service reads this flag from `emergencyFallbackFlagCache` on each request.

### Adjusting Circuit Breaker Thresholds

Modify provider-specific resilience profiles dynamically:

```bash
curl -X PATCH http://localhost:20128/api/resilience \
     -H "Authorization: Bearer <MANAGEMENT_TOKEN>" \
     -H "Content-Type: application/json" \
     -d '{
           "providerBreaker": {
             "oauth": { "failureThreshold": 10, "resetTimeoutMs": 120000 }
           }
         }'

```

The PATCH handler in [`src/app/api/resilience/route.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/app/api/resilience/route.ts) persists these changes to the `ResilienceSettings` record, which the CircuitBreaker references through `PROVIDER_PROFILES` in [`src/shared/constants/providerProfiles.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/constants/providerProfiles.ts).

## Summary

- **Connection Cooldown** ([`src/lib/resilience/settings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/settings.ts)): Throttles 429 responses using a Bottleneck queue to prevent retry floods against rate-limited providers.
- **Provider Circuit Breaker** ([`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts)): Tracks failures per provider profile and opens circuits when `circuitBreakerThreshold` is exceeded, stopping credential leakage to failing endpoints.
- **Emergency Fallback** ([`open-sse/services/emergencyFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/emergencyFallback.ts)): Provides a free model response when all providers are exhausted, triggered only once per request and isolated from original provider credentials.
- **Interaction Flow**: Handlers in [`src/sse/handlers/chat.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/handlers/chat.ts) orchestrate the layers, ensuring that cooldowns are respected before circuits are checked, and circuits are respected before the emergency fallback is invoked.

## Frequently Asked Questions

### How does the connection cooldown differ from the circuit breaker in OmniRoute?

The **connection cooldown** (Layer 1) is a short, automatic timeout triggered specifically by HTTP 429 responses to prevent immediate retries against a rate-limited endpoint, whereas the **circuit breaker** (Layer 2) tracks all failure types (5xx, auth errors, timeouts) across multiple requests and opens only after the configurable `circuitBreakerThreshold` is reached, typically indicating sustained provider unavailability.

### What happens when all configured providers are exhausted?

When every provider returns either a *budget-exhausted* or *circuit-open* status, OmniRoute invokes the **EmergencyFallback** service. If the `EMERGENCY_FALLBACK_FLAG_KEY` is enabled, it routes the request to a free fallback model (default `openai/gpt-oss-120b`) exactly once per request without exposing the original providers' credentials, ensuring the end user receives a response rather than a hard failure.

### Can I reset a tripped circuit breaker without restarting OmniRoute?

Yes. Send a POST request to `/api/resilience/reset` with a valid management token. This triggers `resetAllCircuitBreakers()` in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts), immediately transitioning all circuits from OPEN to CLOSED and allowing traffic to resume against previously failing providers.

### Does the emergency fallback layer expose my provider API keys?

No. The emergency fallback explicitly operates as a **credential-leak guard**. According to the implementation in [`src/sse/handlers/chat.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/handlers/chat.ts), when the fallback is invoked, the request is routed to the free model without attaching any authentication headers or tokens from the original provider profiles, ensuring complete isolation between failed premium providers and the emergency response path.