# How the OmniRoute Provider Circuit Breaker Handles Failures: State Machine & Resilience Patterns

> Learn how the OmniRoute Provider Circuit Breaker handles failures. Discover its state machine, resilience patterns, and automatic recovery for robust upstream provider monitoring.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: internals
- Published: 2026-08-08

---

**The OmniRoute Provider Circuit Breaker monitors upstream providers using a three-state machine (CLOSED → OPEN → HALF_OPEN) that blocks traffic after configurable failure thresholds, persists state to a database table, and automatically recovers after reset timeouts while excluding authentication errors from failure counts.**

The OmniRoute repository implements a sophisticated resilience layer to shield the request pipeline from unreliable upstream providers. At the heart of this system sits a **provider-level circuit breaker** implemented in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) that tracks transient failures and automatically manages traffic flow based on provider health.

## Circuit Breaker State Machine: CLOSED, OPEN, and HALF_OPEN

The breaker operates through three distinct states defined in the `STATE` enum within [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts):

- **CLOSED**: The default state where traffic flows normally to the provider. The breaker counts consecutive failures but allows requests to proceed.
- **OPEN**: Triggered when the failure count reaches the configured threshold. In this state, all requests to the provider are blocked immediately, failing fast to prevent cascade failures.
- **HALF_OPEN**: After the reset timeout expires, the breaker transitions lazily to HALF_OPEN when `canExecute()` or `getStatus()` is called. This allows a single probe request to test provider recovery.

If the probe succeeds, the breaker resets to **CLOSED** and clears the failure count. If it fails, the breaker returns immediately to **OPEN** with a fresh timeout period.

## Failure Detection and Error Classification

The breaker only increments its failure counter for specific transient upstream errors. The `recordFailure(kind)` method processes errors classified by `isLocalStreamLifecycleError`, which is utilized in [`open-sse/handlers/chatCore.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/handlers/chatCore.ts).

**Counted as upstream failures:**
- HTTP 408 (Request Timeout)
- HTTP 500 (Internal Server Error)
- HTTP 502 (Bad Gateway)
- HTTP 503 (Service Unavailable)
- HTTP 504 (Gateway Timeout)

**Excluded from circuit breaker counting:**
- HTTP 401/403 (Authentication/Authorization errors)
- HTTP 429 (Rate limiting)

These excluded errors are handled separately by connection-cooldown or model-lockout mechanisms in the account fallback layer.

## Provider-Specific Thresholds and Timeouts

Configuration values are defined in [`src/shared/constants/providers.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/constants/providers.ts) and vary by provider type:

- **OAuth providers**: Threshold of **3** failures, reset timeout of **60 seconds**
- **API-key providers**: Threshold of **5** failures, reset timeout of **30 seconds**
- **Local (self-hosted) providers**: Threshold of **2** failures, reset timeout of **15 seconds**

These differentiated thresholds allow stricter monitoring of less reliable local endpoints while giving more leeway to established OAuth services.

## Persisting State to the Database

Circuit breaker state survives process restarts through persistence in the `domain_circuit_breakers` table. The implementation exposes three key functions:

- `getCircuitBreaker(name)`: Retrieves or creates a breaker instance for a specific provider
- `getStatus(name)`: Returns the current state and metadata
- `canExecute(name)`: Checks if requests are currently allowed

These functions read from and write to the database layer, ensuring that state transitions persist across deployments and server restarts.

## Integration with the Routing Layer

The combo routing engine in `open-sse/services/combo/` integrates the breaker into provider selection:

In [`targetSorters.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/targetSorters.ts) and [`quotaStrategies.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/quotaStrategies.ts), the router calls `getCircuitBreaker(provider)` for each candidate. Providers whose breaker is **OPEN** are excluded from the candidate pool entirely. The breaker state also feeds into health explanations generated by [`src/lib/usage/resilienceExplain.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/usage/resilienceExplain.ts), allowing dashboards to display why specific providers are unavailable.

## Automatic Recovery and Lazy State Transitions

The OmniRoute circuit breaker employs a **"lazy" recovery** pattern rather than background timers. When the reset timeout has elapsed, the next call to `canExecute()` or `getStatus()` automatically transitions the state from **OPEN** to **HALF_OPEN**. This approach eliminates the need for background threads or interval checks while ensuring that dashboards and routing logic do not permanently blacklist providers that have recovered.

## Separate Token Refresh Circuit Breaker

Token refresh flows use a dedicated lightweight breaker located at [`open-sse/services/tokenRefresh/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/tokenRefresh/circuitBreaker.ts). This separate implementation tracks refresh failures per provider and blocks token refresh attempts for a short cooldown period, preventing noisy authentication loops without affecting the main request circuit breaker.

## Code Examples

**Basic circuit breaker usage:**

```typescript
import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';

// Obtain or create breaker for the "openai" provider
const cb = getCircuitBreaker('openai');

// Check if provider is healthy before sending request
if (cb.canExecute()) {
  // ... execute request
} else {
  // Provider is OPEN - implement fallback logic
  console.log('Provider circuit is open, using fallback');
}

// After request completion, record outcome
if (response.ok) {
  cb.recordSuccess(); // Resets failure count to zero
} else if (isLocalStreamLifecycleError(error)) {
  cb.recordFailure('upstream'); // Increments failure counter
}

```

**Filtering providers in the combo router:**

```typescript
import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';
import type { ProviderCandidate } from '@/open-sse/services/combo/types';

function filterHealthyProviders(candidates: ProviderCandidate[]): ProviderCandidate[] {
  return candidates.filter(candidate => {
    const breaker = getCircuitBreaker(candidate.provider);
    const status = breaker.getStatus();
    
    // Exclude providers with OPEN circuit
    return status.state !== 'OPEN';
  });
}

```

## Summary

- The **three-state machine** (CLOSED → OPEN → HALF_OPEN) automatically blocks failing providers while allowing gradual recovery through probe requests.
- **Error classification** via `isLocalStreamLifecycleError` ensures only transient upstream failures (HTTP 408/5xx) trigger the breaker, excluding auth and rate-limit errors.
- **Provider-specific thresholds** defined in [`src/shared/constants/providers.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/constants/providers.ts) apply different sensitivity levels to OAuth, API-key, and local providers.
- **Database persistence** via the `domain_circuit_breakers` table and utility functions like `getCircuitBreaker()` maintains state across process restarts.
- **Lazy recovery** transitions OPEN → HALF_OPEN on the next status check after timeout, eliminating the need for background timers.
- The **combo router** automatically excludes OPEN providers from candidate selection in [`targetSorters.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/targetSorters.ts).

## Frequently Asked Questions

### What triggers the circuit breaker to open?

The circuit breaker transitions from **CLOSED** to **OPEN** when the failure count reaches the configured threshold for that provider type (2-5 failures depending on provider category). Each call to `recordFailure()` increments this counter when the error is classified as an upstream service failure (HTTP 408, 500, 502, 503, or 504).

### Does the circuit breaker track authentication errors?

No. HTTP 401, 403, and 429 responses are explicitly excluded from circuit breaker counting. These errors are handled by separate connection-cooldown mechanisms in [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts). The `isLocalStreamLifecycleError` function used in [`open-sse/handlers/chatCore.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/handlers/chatCore.ts) filters these out before calling `recordFailure()`.

### How does OmniRoute recover a provider after the circuit opens?

After the configured reset timeout expires (15-60 seconds depending on provider type), the breaker enters **HALF_OPEN** state lazily when `canExecute()` or `getStatus()` is queried. This allows exactly one probe request. If successful, the circuit closes immediately; if it fails, the circuit reopens with a fresh timeout period.

### Where is the circuit breaker state stored?

State persists in the `domain_circuit_breakers` database table. The `getCircuitBreaker(name)` function in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) handles retrieval and updates, ensuring that provider status survives application restarts and is shared across all running instances of the OmniRoute service.