# How OmniRoute's Circuit Breaker Handles Provider Failures: State Machine & Recovery Strategy

> Learn how OmniRoute's circuit breaker manages provider failures using a persistent three-state machine. Discover its recovery strategy and how it ensures resilience.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: internals
- Published: 2026-08-15

---

**OmniRoute uses a persistent three-state circuit breaker in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) that tracks upstream failures per provider, opens the circuit when thresholds are exceeded, and lazily transitions to half-open after timeouts to probe for recovery.**

The `diegosouzapw/OmniRoute` repository implements a resilient routing layer that shields the platform from unreliable upstream AI providers. The circuit breaker monitors HTTP error patterns, maintains durable state across process restarts, and integrates directly with the provider selection logic to ensure only healthy endpoints receive traffic.

## Three-State Circuit Breaker Architecture

The core implementation follows the classic circuit breaker pattern with three distinct states defined in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts):

- **CLOSED**: Traffic flows normally to the provider. The breaker counts consecutive failures in this state.
- **OPEN**: All requests to the provider are blocked immediately. The breaker remains open until the configured reset timeout expires.
- **HALF_OPEN**: A probationary state allowing a single probe request to test if the provider has recovered.

State transitions occur automatically through lazy evaluation. When code calls `canExecute()` or `getStatus()`, the breaker checks the elapsed time since entering OPEN. If the timeout has expired, it atomically transitions to HALF_OPEN for the next request. A successful execution in HALF_OPEN resets the breaker to CLOSED, while a failure reverts it immediately to OPEN with a fresh timeout.

## Failure Detection and Classification

Not all errors count as circuit breaker failures. In [`open-sse/handlers/chatCore.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/handlers/chatCore.ts), the `isLocalStreamLifecycleError` function classifies errors to distinguish between transient upstream issues and authentication failures.

The circuit breaker increments its failure counter only for upstream service errors including HTTP **408** (Request Timeout), **500** (Internal Server Error), **502** (Bad Gateway), **503** (Service Unavailable), and **504** (Gateway Timeout). These are recorded via `recordFailure('upstream')`.

Authentication-related errors such as **401** (Unauthorized), **403** (Forbidden), and **429** (Rate Limited) are intentionally excluded from the circuit breaker logic. These trigger separate mechanisms like connection cooldowns or model lockouts managed by [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts), preventing the breaker from opening due to credential issues or quota violations.

## Provider-Specific Thresholds and Configuration

Thresholds and reset timeouts vary by provider authentication type and are defined in [`src/shared/constants/providers.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/constants/providers.ts):

- **OAuth providers**: Threshold of **3** failures, reset timeout of **60 seconds**
- **API-key providers**: Threshold of **5** failures, reset timeout of **30 seconds**
- **Local (self-hosted) providers**: Threshold of **2** failures, reset timeout of **15 seconds**

This differentiation recognizes that self-hosted infrastructure typically requires faster failure detection, while managed OAuth endpoints may tolerate brief transient issues.

## Persistence and State Management

The circuit breaker maintains durable state in the `domain_circuit_breakers` database table, ensuring failure counts and state transitions survive process restarts. The public API exposes three primary functions in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts):

- `getCircuitBreaker(name)`: Retrieves or initializes a breaker instance for a specific provider
- `canExecute()`: Returns boolean indicating whether the provider is currently reachable (CLOSED or HALF_OPEN)
- `getStatus()`: Returns the current state object including failure counts and timestamps

After a successful request, callers must invoke `recordSuccess()` to reset the failure accumulator to zero. Conversely, `recordFailure(kind)` increments counters and triggers the OPEN transition when thresholds are breached.

## Integration with the Routing Layer

The combo router ([`open-sse/services/combo/targetSorters.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/combo/targetSorters.ts) and [`quotaStrategies.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/quotaStrategies.ts)) consults the circuit breaker during provider selection. Before adding a provider to the candidate pool, the router checks `getCircuitBreaker(provider).getStatus()`. Providers in the OPEN state are automatically excluded from routing decisions.

The resilience explanation system ([`src/lib/usage/resilienceExplain.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/usage/resilienceExplain.ts)) also consumes breaker status to generate health reports, allowing dashboards to display why specific providers are temporarily unavailable.

## Token Refresh Isolation

Token refresh flows maintain a separate lightweight circuit breaker in [`open-sse/services/tokenRefresh/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/tokenRefresh/circuitBreaker.ts). This isolation prevents OAuth refresh failures from impacting the main provider circuit while still protecting the token endpoint from excessive retry traffic during outages.

## Summary

- **Three-state machine**: CLOSED tracks failures, OPEN blocks traffic, HALF_OPEN probes for recovery via lazy state transitions in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts)
- **Selective failure counting**: Only upstream HTTP errors (408, 500, 502, 503, 504) trigger the breaker; auth errors (401, 403, 429) are handled separately in [`open-sse/handlers/chatCore.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/handlers/chatCore.ts)
- **Configurable thresholds**: Provider types in [`src/shared/constants/providers.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/constants/providers.ts) define specific failure thresholds (2-5) and reset timeouts (15-60 seconds)
- **Durable state**: Breaker status persists to `domain_circuit_breakers` table, with `getCircuitBreaker()`, `canExecute()`, and `recordSuccess()` forming the primary API
- **Routing integration**: The combo router filters OPEN providers via [`targetSorters.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/targetSorters.ts), while a separate breaker in [`tokenRefresh/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/tokenRefresh/circuitBreaker.ts) handles OAuth refresh isolation

## Frequently Asked Questions

### What triggers the circuit breaker to open in OmniRoute?

The circuit breaker transitions from CLOSED to OPEN when `recordFailure()` is called for an upstream error (HTTP 408, 500, 502, 503, or 504) and the consecutive failure count reaches the provider-specific threshold defined in [`src/shared/constants/providers.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/constants/providers.ts). Authentication errors like 401 or 429 do not increment the failure counter.

### How does the circuit breaker recover from an OPEN state?

Recovery uses lazy state transition. When `canExecute()` or `getStatus()` is called after the reset timeout expires, the breaker automatically transitions to HALF_OPEN, allowing a single probe request. If that request succeeds and `recordSuccess()` is called, the breaker resets to CLOSED. If it fails, the breaker returns to OPEN with a fresh timeout.

### Are authentication errors like 401 or 403 counted by the circuit breaker?

No. The `isLocalStreamLifecycleError` classifier in [`open-sse/handlers/chatCore.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/handlers/chatCore.ts) explicitly excludes 401, 403, and 429 errors from circuit breaker consideration. These trigger connection cooldown mechanisms instead, preventing the breaker from opening due to credential or quota issues rather than service unavailability.

### Where is the circuit breaker state stored in OmniRoute?

Provider circuit breaker state persists in the `domain_circuit_breakers` database table. The `getCircuitBreaker(name)` function in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) handles read/write operations, ensuring that failure counts and OPEN/HALF_OPEN states survive application restarts or horizontal scaling events.