# How OmniRoute's Provider Circuit Breaker Protects LLM Routing

> OmniRoute's provider circuit breaker uses a four phase state machine to protect LLM routing by preventing cascade failures and automatically recovering misbehaving providers.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: internals
- Published: 2026-08-23

---

**OmniRoute's provider circuit breaker wraps every LLM provider request in a state machine with four phases—CLOSED, DEGRADED, OPEN, and HALF_OPEN—to prevent cascade failures by short-circuiting requests to misbehaving providers while automatically recovering when health improves.**

OmniRoute is an open-source LLM routing framework that isolates unstable providers using a sophisticated circuit breaker implementation. Located in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts), the breaker extends the classic pattern with failure-kind awareness, adaptive exponential back-off, and persistent state storage to ensure system resilience across process restarts.

## The Four-State Circuit Breaker State Machine

Unlike standard three-state breakers, OmniRoute implements a four-state machine defined at lines 12–16 of the core utility file:

- **CLOSED**: Traffic passes normally to the provider. This is the healthy state.
- **DEGRADED**: A warning state activated when the failure count exceeds 60% of the configured threshold. Requests continue to pass, but the system logs warnings that the provider is becoming unstable.
- **OPEN**: The circuit has tripped. All requests are short-circuited and immediately fail with a circuit-breaker error, forcing the combo router to select an alternative provider.
- **HALF_OPEN**: A probing state entered after the reset timeout expires. Only a limited number of probe requests (default 1) are permitted to test whether the provider has recovered.

## Configuring Thresholds and Timeouts

The `CircuitBreakerOptions` interface allows per-provider customization of resilience parameters. According to the constructor defaults at lines 69–73, the breaker initializes with:

- **`failureThreshold`**: 5 failures before tripping to OPEN.
- **`resetTimeout`**: 30,000 milliseconds (30 seconds) before attempting recovery.
- **`halfOpenRequests`**: 1 probe request allowed during the HALF_OPEN phase.

These defaults can be overridden when instantiating a breaker via the `getCircuitBreaker(name, options)` factory function.

## Failure Classification and Kind-Specific Handling

OmniRoute enhances the classic pattern by classifying errors into distinct `FailureKind` types (defined in [`src/shared/utils/classify429.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/classify429.ts)), such as `rate_limit`, `quota_exhausted`, and `transient`. The `kindThresholds` interface (lines 77–84) allows each failure type to have its own threshold, cooldown period, or an `immediateOpen` flag that trips the circuit instantly for critical errors.

The `classifyError` callback function analyzes HTTP responses or exceptions to categorize failures, enabling fine-grained control. For example, a 429 rate-limit error might trigger a shorter cooldown than a 500 server error.

## State Transitions and Adaptive Back-Off

State transitions follow strict rules based on the `degradationThreshold`, which defaults to 60% of the `failureThreshold`:

- **On Failure**: The breaker increments both a global counter and a per-kind counter. If the total reaches the degradation threshold, the state shifts to **DEGRADED**. If it hits the full `failureThreshold`, the state jumps to **OPEN** (or immediately to **OPEN** if the failure kind has `immediateOpen` set).
- **On Success**: While in **HALF_OPEN**, a successful probe resets the breaker to **CLOSED** and clears all counters. In other states, successful requests gradually decrement the failure count.

To prevent flapping on intermittent failures, the implementation uses adaptive back-off (lines 55–63). Every time the circuit transitions **OPEN → HALF_OPEN → OPEN**, the effective reset timeout doubles, up to a configurable `maxBackoffMultiplier`. This exponential back-off ensures that genuinely unhealthy providers are isolated for progressively longer periods.

## Lazy Recovery and Persistence

The breaker does not rely on background timers for state transitions. Instead, it uses **lazy recovery**: the `_refreshOpenState()` method (lines 63–66) is invoked only when `canExecute()` checks a request while the circuit is **OPEN**. If the `resetTimeout` has elapsed, the breaker atomically transitions to **HALF_OPEN**.

For durability across process restarts, the breaker persists its current state, counters, and configuration to the `domainState` database table (managed in [`src/lib/db/domainState.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/db/domainState.ts)). The methods `saveCircuitBreakerState` and `loadCircuitBreakerState` (lines 97–112) serialize and hydrate the breaker, ensuring that a restart does not reset protection counters or accidentally close an open circuit.

## Registry Management and Memory Optimization

To prevent memory leaks in long-running processes, breakers are stored in a global `Map` registry. A periodic sweep removes idle, **CLOSED** breakers to bound memory usage, respecting a `MAX_REGISTRY_SIZE` limit of 500 instances. This cleanup logic appears at lines 18–33 of the utility file.

## Implementing Provider Protection in Practice

In production, each provider (e.g., *openai*, *anthropic*, *groq*) receives its own named breaker via the factory function. The request-handling code in [`src/sse/handlers/chatHelpers.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/handlers/chatHelpers.ts) consults the breaker before issuing upstream HTTP calls.

```typescript
import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';

// Create or retrieve a breaker for the OpenAI provider
const openaiBreaker = getCircuitBreaker('openai', {
  failureThreshold: 5,
  resetTimeout: 30_000,
  halfOpenRequests: 1,
  classifyError: (err) => (err?.statusCode >= 500 ? 'transient' : undefined),
});

// Wrap a provider call in the circuit breaker
async function fetchChatCompletion(payload: any) {
  return openaiBreaker.execute(async () => {
    const resp = await fetch('https://api.openai.com/v1/chat/completions', {
      method: 'POST',
      body: JSON.stringify(payload),
      headers: { Authorization: `Bearer ${process.env.OPENAI_KEY}` },
    });
    return resp.json();
  });
}

// Check breaker status for monitoring dashboards
const status = openaiBreaker.getStatus();
console.log(`State: ${status.state}, Retry after: ${status.retryAfterMs}ms`);

```

When the breaker is **OPEN**, the `execute()` method immediately rejects, triggering the fallback logic in [`src/sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/accountFallback.ts) to reroute the request. If **HALF_OPEN**, only the configured number of probe requests are permitted through.

## Summary

- **Four-state protection**: CLOSED, DEGRADED (early warning), OPEN (blocked), and HALF_OPEN (probing) states provide granular control over provider health.
- **Intelligent classification**: Per-failure-kind thresholds via `classifyError` allow different handling for rate limits versus server errors.
- **Adaptive back-off**: Exponential timeout doubling prevents rapid circuit flapping on intermittent failures.
- **Zero-timer recovery**: Lazy state refresh via `_refreshOpenState()` eliminates background process overhead.
- **Durable state**: Persistence to the `domainState` table ensures protection survives application restarts.
- **Memory-safe**: The global registry with `MAX_REGISTRY_SIZE` prevents unbounded growth in multi-tenant deployments.

## Frequently Asked Questions

### What is the difference between the DEGRADED and OPEN states in OmniRoute's circuit breaker?

**DEGRADED** acts as an early warning system triggered when failures reach 60% of the configured threshold; traffic still flows but warnings are logged to indicate declining provider health. **OPEN** means the circuit has fully tripped after reaching the `failureThreshold`, causing all new requests to fail immediately without hitting the upstream provider, forcing the router to select an alternative.

### How does OmniRoute prevent rapid circuit flapping on intermittent provider failures?

The implementation employs **adaptive back-off**: each time the circuit cycles from **OPEN** to **HALF_OPEN** and back to **OPEN**, the `resetTimeout` doubles (up to `maxBackoffMultiplier`). This exponential delay ensures that providers exhibiting inconsistent behavior are isolated for progressively longer periods, rather than being probed at fixed intervals.

### Does the circuit breaker state persist across application restarts?

Yes. According to the source code in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) (lines 97–112), the breaker serializes its current state, failure counters, and configuration to the `domainState` database table using `saveCircuitBreakerState`. On initialization, `loadCircuitBreakerState` restores the previous condition, preventing a restart from resetting an open circuit or losing failure history.

### What triggers the transition from OPEN to HALF_OPEN state?

Rather than using background timers, OmniRoute uses **lazy recovery**: the `_refreshOpenState()` method checks whether the `resetTimeout` has elapsed only when `canExecute()` is invoked by an incoming request. If the cooldown period has passed, the breaker atomically transitions to **HALF_OPEN** and permits the probe request to proceed.