# How OmniRoute's Provider Circuit Breaker Works: Architecture and Implementation

> Discover how OmniRoute's provider circuit breaker prevents cascading failures. Explore its four-state architecture, adaptive back-off, and state recovery for robust LLM integrations.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: architecture
- Published: 2026-08-21

---

**OmniRoute prevents cascading failures by wrapping every LLM provider request in a four-state circuit breaker with failure-kind awareness, adaptive back-off, and persistent state recovery.**

OmniRoute is an open-source LLM routing platform that implements enterprise-grade fault tolerance through its provider circuit breaker system. The implementation in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) extends the classic circuit breaker pattern with OmniRoute-specific enhancements like granular error classification and state persistence across restarts. This ensures that a single misbehaving provider cannot degrade the entire routing infrastructure.

## Four-State Circuit Breaker Architecture

Unlike standard circuit breakers that use three states, OmniRoute implements a four-state machine: `CLOSED`, `DEGRADED`, `OPEN`, and `HALF_OPEN`. These states are defined at lines 12–16 of [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts).

- **CLOSED** – The circuit operates normally and all requests pass through to the provider.
- **DEGRADED** – The failure rate has exceeded 60% of the configured threshold (the `degradationThreshold`), but traffic still flows while the system logs warnings.
- **OPEN** – The failure threshold has been breached; requests are short-circuited and immediately fail without reaching the provider.
- **HALF_OPEN** – A probe window allowing a limited number of requests (default 1) to test whether the provider has recovered.

### State Transition Logic

When a failure occurs, the breaker increments both a global failure counter and a per-kind counter (lines 79–104). If the total count exceeds the `degradationThreshold`, the state shifts to **DEGRADED**. Upon reaching the full `failureThreshold`, the state jumps to **OPEN**. For error kinds flagged with `immediateOpen`, the circuit bypasses **DEGRADED** and opens immediately.

On success while in **HALF_OPEN**, the breaker resets to **CLOSED**. Otherwise, successful requests gradually decrement the failure count.

## Failure Classification and Threshold Management

OmniRoute's circuit breaker implements **failure-kind awareness**, allowing different error types to trigger distinct thresholds and cooldown behaviors. The `FailureKind` enumeration (defined in [`src/shared/utils/classify429.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/classify429.ts)) includes categories like `rate_limit`, `quota_exhausted`, and `transient`.

The `CircuitBreakerOptions` interface (lines 77–84) permits configuring `kindThresholds`, where each failure type can specify:
- Custom failure thresholds
- Unique cooldown durations
- An `immediateOpen` flag to bypass the degraded state

This granularity prevents transient network blips from triggering the same penalties as quota exhaustion.

## Adaptive Back-off and Lazy Recovery

To prevent rapid state flapping on flaky providers, OmniRoute implements an **adaptive back-off** mechanism. Every time the circuit transitions from **OPEN → HALF_OPEN → OPEN**, the reset timeout doubles, up to a `maxBackoffMultiplier` limit (lines 55–63).

The system uses **lazy recovery** rather than background timers. When a request checks `canExecute()`, the code invokes `_refreshOpenState()` (lines 63–66), which transitions the breaker from **OPEN** to **HALF_OPEN** only after the cooldown period has elapsed. This eliminates the overhead of active polling.

## Persistence and Memory Management

Circuit breaker state survives process restarts through the `domainState` table defined in [`src/lib/db/domainState.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/db/domainState.ts). On instantiation, each breaker calls `loadCircuitBreakerState()` (lines 97–112) to restore previous counters and state, ensuring that a restart does not reset protection against recently failed providers.

To prevent memory leaks in long-running processes, breakers are stored in a global `Map` registry with a maximum size of 500 instances (`MAX_REGISTRY_SIZE`). A periodic sweep removes idle, **CLOSED** breakers that have exceeded their idle timeout (lines 18–33).

## Implementation Reference

The following pattern demonstrates how to wrap provider calls using the breaker factory:

```typescript
// Import the breaker factory
import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';

// Create (or retrieve) a breaker for the OpenAI provider
const openaiBreaker = getCircuitBreaker('openai', {
  failureThreshold: 5,          // trip after 5 failures
  resetTimeout: 30_000,         // 30 s back-off
  halfOpenRequests: 1,          // one probe request
  // optional: classify 5xx errors as transient failures
  classifyError: (err) => (err?.statusCode >= 500 ? 'transient' : undefined),
});

// Wrap a provider call in the circuit-breaker
async function fetchChatCompletion(payload: any) {
  return openaiBreaker.execute(async () => {
    // This is the actual upstream request (fetch, axios, etc.)
    const resp = await fetch('https://api.openai.com/v1/chat/completions', {
      method: 'POST',
      body: JSON.stringify(payload),
      headers: { Authorization: `Bearer ${process.env.OPENAI_KEY}` },
    });
    return resp.json();
  });
}

// Check breaker status (useful for monitoring dashboards)
const status = openaiBreaker.getStatus();
console.log(`OpenAI breaker state: ${status.state}, retry after ${status.retryAfterMs} ms`);

```

The `execute()` method handles state checks internally, while `canExecute()` (used in [`src/sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/accountFallback.ts)) allows pre-flight validation before attempting expensive operations.

## Summary

- OmniRoute implements a **four-state circuit breaker** (`CLOSED`, `DEGRADED`, `OPEN`, `HALF_OPEN`) with a degradation threshold at 60% of the failure limit.
- **Failure-kind awareness** allows distinct handling of `rate_limit`, `quota_exhausted`, and `transient` errors via the `kindThresholds` configuration.
- **Adaptive back-off** doubles the reset timeout after each failed recovery attempt to prevent flapping.
- **Lazy state recovery** uses `_refreshOpenState()` triggered by request checks rather than background timers.
- State persists across restarts via the `domainState` table in [`src/lib/db/domainState.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/db/domainState.ts).
- A bounded registry with periodic sweep prevents memory leaks (`MAX_REGISTRY_SIZE = 500`).

## Frequently Asked Questions

### What are the four states of OmniRoute's provider circuit breaker?

The four states are `CLOSED` (normal operation), `DEGRADED` (high failure rate but traffic continues with warnings), `OPEN` (requests blocked), and `HALF_OPEN` (probe window for recovery testing). The **DEGRADED** state is unique to OmniRoute and triggers when failures exceed 60% of the configured threshold but have not yet reached the full trip limit.

### How does the circuit breaker handle different types of failures?

OmniRoute classifies errors into kinds like `rate_limit`, `quota_exhausted`, and `transient` using the `classifyError` option. Each kind can define its own `kindThresholds` with specific failure limits, cooldown periods, and an `immediateOpen` flag that bypasses the degraded state entirely. This prevents temporary network errors from counting against the same threshold as authentication failures.

### Where is circuit breaker state stored in OmniRoute?

State is persisted in the `domainState` table managed by [`src/lib/db/domainState.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/db/domainState.ts). The breaker serializes its current state, counters, and configuration via `saveCircuitBreakerState()` and restores them on process startup through `loadCircuitBreakerState()` (lines 97–112), ensuring continuity of protection across deployments.

### How does the adaptive back-off mechanism prevent flapping?

Each time the circuit transitions from **OPEN** to **HALF_OPEN** and back to **OPEN** due to failed probes, the `resetTimeout` doubles up to the `maxBackoffMultiplier` limit. This exponential back-off prevents rapid cycling between open and closed states when a provider is experiencing intermittent issues, reducing load on both the provider and the routing system.