How the OmniRoute Provider Circuit Breaker Handles Failures: State Machine & Resilience Patterns

The OmniRoute Provider Circuit Breaker monitors upstream providers using a three-state machine (CLOSED → OPEN → HALF_OPEN) that blocks traffic after configurable failure thresholds, persists state to a database table, and automatically recovers after reset timeouts while excluding authentication errors from failure counts.

The OmniRoute repository implements a sophisticated resilience layer to shield the request pipeline from unreliable upstream providers. At the heart of this system sits a provider-level circuit breaker implemented in src/shared/utils/circuitBreaker.ts that tracks transient failures and automatically manages traffic flow based on provider health.

Circuit Breaker State Machine: CLOSED, OPEN, and HALF_OPEN

The breaker operates through three distinct states defined in the STATE enum within src/shared/utils/circuitBreaker.ts:

• CLOSED: The default state where traffic flows normally to the provider. The breaker counts consecutive failures but allows requests to proceed.
• OPEN: Triggered when the failure count reaches the configured threshold. In this state, all requests to the provider are blocked immediately, failing fast to prevent cascade failures.
• HALF_OPEN: After the reset timeout expires, the breaker transitions lazily to HALF_OPEN when canExecute() or getStatus() is called. This allows a single probe request to test provider recovery.

If the probe succeeds, the breaker resets to CLOSED and clears the failure count. If it fails, the breaker returns immediately to OPEN with a fresh timeout period.

Failure Detection and Error Classification

The breaker only increments its failure counter for specific transient upstream errors. The recordFailure(kind) method processes errors classified by isLocalStreamLifecycleError, which is utilized in open-sse/handlers/chatCore.ts.

Counted as upstream failures:

• HTTP 408 (Request Timeout)
• HTTP 500 (Internal Server Error)
• HTTP 502 (Bad Gateway)
• HTTP 503 (Service Unavailable)
• HTTP 504 (Gateway Timeout)

Excluded from circuit breaker counting:

• HTTP 401/403 (Authentication/Authorization errors)
• HTTP 429 (Rate limiting)

These excluded errors are handled separately by connection-cooldown or model-lockout mechanisms in the account fallback layer.

Provider-Specific Thresholds and Timeouts

Configuration values are defined in src/shared/constants/providers.ts and vary by provider type:

• OAuth providers: Threshold of 3 failures, reset timeout of 60 seconds
• API-key providers: Threshold of 5 failures, reset timeout of 30 seconds
• Local (self-hosted) providers: Threshold of 2 failures, reset timeout of 15 seconds

These differentiated thresholds allow stricter monitoring of less reliable local endpoints while giving more leeway to established OAuth services.

Persisting State to the Database

Circuit breaker state survives process restarts through persistence in the domain_circuit_breakers table. The implementation exposes three key functions:

• getCircuitBreaker(name): Retrieves or creates a breaker instance for a specific provider
• getStatus(name): Returns the current state and metadata
• canExecute(name): Checks if requests are currently allowed

These functions read from and write to the database layer, ensuring that state transitions persist across deployments and server restarts.

Integration with the Routing Layer

The combo routing engine in open-sse/services/combo/ integrates the breaker into provider selection:

In targetSorters.ts and quotaStrategies.ts, the router calls getCircuitBreaker(provider) for each candidate. Providers whose breaker is OPEN are excluded from the candidate pool entirely. The breaker state also feeds into health explanations generated by src/lib/usage/resilienceExplain.ts, allowing dashboards to display why specific providers are unavailable.

Automatic Recovery and Lazy State Transitions

The OmniRoute circuit breaker employs a "lazy" recovery pattern rather than background timers. When the reset timeout has elapsed, the next call to canExecute() or getStatus() automatically transitions the state from OPEN to HALF_OPEN. This approach eliminates the need for background threads or interval checks while ensuring that dashboards and routing logic do not permanently blacklist providers that have recovered.

Separate Token Refresh Circuit Breaker

Token refresh flows use a dedicated lightweight breaker located at open-sse/services/tokenRefresh/circuitBreaker.ts. This separate implementation tracks refresh failures per provider and blocks token refresh attempts for a short cooldown period, preventing noisy authentication loops without affecting the main request circuit breaker.

Code Examples

Basic circuit breaker usage:

import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';

// Obtain or create breaker for the "openai" provider
const cb = getCircuitBreaker('openai');

// Check if provider is healthy before sending request
if (cb.canExecute()) {
  // ... execute request
} else {
  // Provider is OPEN - implement fallback logic
  console.log('Provider circuit is open, using fallback');
}

// After request completion, record outcome
if (response.ok) {
  cb.recordSuccess(); // Resets failure count to zero
} else if (isLocalStreamLifecycleError(error)) {
  cb.recordFailure('upstream'); // Increments failure counter
}

Filtering providers in the combo router:

import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';
import type { ProviderCandidate } from '@/open-sse/services/combo/types';

function filterHealthyProviders(candidates: ProviderCandidate[]): ProviderCandidate[] {
  return candidates.filter(candidate => {
    const breaker = getCircuitBreaker(candidate.provider);
    const status = breaker.getStatus();
    
    // Exclude providers with OPEN circuit
    return status.state !== 'OPEN';
  });
}

Summary

• The three-state machine (CLOSED → OPEN → HALF_OPEN) automatically blocks failing providers while allowing gradual recovery through probe requests.
• Error classification via isLocalStreamLifecycleError ensures only transient upstream failures (HTTP 408/5xx) trigger the breaker, excluding auth and rate-limit errors.
• Provider-specific thresholds defined in src/shared/constants/providers.ts apply different sensitivity levels to OAuth, API-key, and local providers.
• Database persistence via the domain_circuit_breakers table and utility functions like getCircuitBreaker() maintains state across process restarts.
• Lazy recovery transitions OPEN → HALF_OPEN on the next status check after timeout, eliminating the need for background timers.
• The combo router automatically excludes OPEN providers from candidate selection in targetSorters.ts.

Frequently Asked Questions

What triggers the circuit breaker to open?

The circuit breaker transitions from CLOSED to OPEN when the failure count reaches the configured threshold for that provider type (2-5 failures depending on provider category). Each call to recordFailure() increments this counter when the error is classified as an upstream service failure (HTTP 408, 500, 502, 503, or 504).

Does the circuit breaker track authentication errors?

No. HTTP 401, 403, and 429 responses are explicitly excluded from circuit breaker counting. These errors are handled by separate connection-cooldown mechanisms in open-sse/services/accountFallback.ts. The isLocalStreamLifecycleError function used in open-sse/handlers/chatCore.ts filters these out before calling recordFailure().

How does OmniRoute recover a provider after the circuit opens?

After the configured reset timeout expires (15-60 seconds depending on provider type), the breaker enters HALF_OPEN state lazily when canExecute() or getStatus() is queried. This allows exactly one probe request. If successful, the circuit closes immediately; if it fails, the circuit reopens with a fresh timeout period.

Where is the circuit breaker state stored?

State persists in the domain_circuit_breakers database table. The getCircuitBreaker(name) function in src/shared/utils/circuitBreaker.ts handles retrieval and updates, ensuring that provider status survives application restarts and is shared across all running instances of the OmniRoute service.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}