How OmniRoute's Provider Circuit Breaker Protects LLM Routing

OmniRoute's provider circuit breaker wraps every LLM provider request in a state machine with four phases—CLOSED, DEGRADED, OPEN, and HALF_OPEN—to prevent cascade failures by short-circuiting requests to misbehaving providers while automatically recovering when health improves.

OmniRoute is an open-source LLM routing framework that isolates unstable providers using a sophisticated circuit breaker implementation. Located in src/shared/utils/circuitBreaker.ts, the breaker extends the classic pattern with failure-kind awareness, adaptive exponential back-off, and persistent state storage to ensure system resilience across process restarts.

The Four-State Circuit Breaker State Machine

Unlike standard three-state breakers, OmniRoute implements a four-state machine defined at lines 12–16 of the core utility file:

• CLOSED: Traffic passes normally to the provider. This is the healthy state.
• DEGRADED: A warning state activated when the failure count exceeds 60% of the configured threshold. Requests continue to pass, but the system logs warnings that the provider is becoming unstable.
• OPEN: The circuit has tripped. All requests are short-circuited and immediately fail with a circuit-breaker error, forcing the combo router to select an alternative provider.
• HALF_OPEN: A probing state entered after the reset timeout expires. Only a limited number of probe requests (default 1) are permitted to test whether the provider has recovered.

Configuring Thresholds and Timeouts

The CircuitBreakerOptions interface allows per-provider customization of resilience parameters. According to the constructor defaults at lines 69–73, the breaker initializes with:

• failureThreshold: 5 failures before tripping to OPEN.
• resetTimeout: 30,000 milliseconds (30 seconds) before attempting recovery.
• halfOpenRequests: 1 probe request allowed during the HALF_OPEN phase.

These defaults can be overridden when instantiating a breaker via the getCircuitBreaker(name, options) factory function.

Failure Classification and Kind-Specific Handling

OmniRoute enhances the classic pattern by classifying errors into distinct FailureKind types (defined in src/shared/utils/classify429.ts), such as rate_limit, quota_exhausted, and transient. The kindThresholds interface (lines 77–84) allows each failure type to have its own threshold, cooldown period, or an immediateOpen flag that trips the circuit instantly for critical errors.

The classifyError callback function analyzes HTTP responses or exceptions to categorize failures, enabling fine-grained control. For example, a 429 rate-limit error might trigger a shorter cooldown than a 500 server error.

State Transitions and Adaptive Back-Off

State transitions follow strict rules based on the degradationThreshold, which defaults to 60% of the failureThreshold:

• On Failure: The breaker increments both a global counter and a per-kind counter. If the total reaches the degradation threshold, the state shifts to DEGRADED. If it hits the full failureThreshold, the state jumps to OPEN (or immediately to OPEN if the failure kind has immediateOpen set).
• On Success: While in HALF_OPEN, a successful probe resets the breaker to CLOSED and clears all counters. In other states, successful requests gradually decrement the failure count.

To prevent flapping on intermittent failures, the implementation uses adaptive back-off (lines 55–63). Every time the circuit transitions OPEN → HALF_OPEN → OPEN, the effective reset timeout doubles, up to a configurable maxBackoffMultiplier. This exponential back-off ensures that genuinely unhealthy providers are isolated for progressively longer periods.

Lazy Recovery and Persistence

The breaker does not rely on background timers for state transitions. Instead, it uses lazy recovery: the _refreshOpenState() method (lines 63–66) is invoked only when canExecute() checks a request while the circuit is OPEN. If the resetTimeout has elapsed, the breaker atomically transitions to HALF_OPEN.

For durability across process restarts, the breaker persists its current state, counters, and configuration to the domainState database table (managed in src/lib/db/domainState.ts). The methods saveCircuitBreakerState and loadCircuitBreakerState (lines 97–112) serialize and hydrate the breaker, ensuring that a restart does not reset protection counters or accidentally close an open circuit.

Registry Management and Memory Optimization

To prevent memory leaks in long-running processes, breakers are stored in a global Map registry. A periodic sweep removes idle, CLOSED breakers to bound memory usage, respecting a MAX_REGISTRY_SIZE limit of 500 instances. This cleanup logic appears at lines 18–33 of the utility file.

Implementing Provider Protection in Practice

In production, each provider (e.g., openai, anthropic, groq) receives its own named breaker via the factory function. The request-handling code in src/sse/handlers/chatHelpers.ts consults the breaker before issuing upstream HTTP calls.

import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';

// Create or retrieve a breaker for the OpenAI provider
const openaiBreaker = getCircuitBreaker('openai', {
  failureThreshold: 5,
  resetTimeout: 30_000,
  halfOpenRequests: 1,
  classifyError: (err) => (err?.statusCode >= 500 ? 'transient' : undefined),
});

// Wrap a provider call in the circuit breaker
async function fetchChatCompletion(payload: any) {
  return openaiBreaker.execute(async () => {
    const resp = await fetch('https://api.openai.com/v1/chat/completions', {
      method: 'POST',
      body: JSON.stringify(payload),
      headers: { Authorization: `Bearer ${process.env.OPENAI_KEY}` },
    });
    return resp.json();
  });
}

// Check breaker status for monitoring dashboards
const status = openaiBreaker.getStatus();
console.log(`State: ${status.state}, Retry after: ${status.retryAfterMs}ms`);

When the breaker is OPEN, the execute() method immediately rejects, triggering the fallback logic in src/sse/services/accountFallback.ts to reroute the request. If HALF_OPEN, only the configured number of probe requests are permitted through.

Summary

• Four-state protection: CLOSED, DEGRADED (early warning), OPEN (blocked), and HALF_OPEN (probing) states provide granular control over provider health.
• Intelligent classification: Per-failure-kind thresholds via classifyError allow different handling for rate limits versus server errors.
• Adaptive back-off: Exponential timeout doubling prevents rapid circuit flapping on intermittent failures.
• Zero-timer recovery: Lazy state refresh via _refreshOpenState() eliminates background process overhead.
• Durable state: Persistence to the domainState table ensures protection survives application restarts.
• Memory-safe: The global registry with MAX_REGISTRY_SIZE prevents unbounded growth in multi-tenant deployments.

Frequently Asked Questions

What is the difference between the DEGRADED and OPEN states in OmniRoute's circuit breaker?

DEGRADED acts as an early warning system triggered when failures reach 60% of the configured threshold; traffic still flows but warnings are logged to indicate declining provider health. OPEN means the circuit has fully tripped after reaching the failureThreshold, causing all new requests to fail immediately without hitting the upstream provider, forcing the router to select an alternative.

How does OmniRoute prevent rapid circuit flapping on intermittent provider failures?

The implementation employs adaptive back-off: each time the circuit cycles from OPEN to HALF_OPEN and back to OPEN, the resetTimeout doubles (up to maxBackoffMultiplier). This exponential delay ensures that providers exhibiting inconsistent behavior are isolated for progressively longer periods, rather than being probed at fixed intervals.

Does the circuit breaker state persist across application restarts?

Yes. According to the source code in src/shared/utils/circuitBreaker.ts (lines 97–112), the breaker serializes its current state, failure counters, and configuration to the domainState database table using saveCircuitBreakerState. On initialization, loadCircuitBreakerState restores the previous condition, preventing a restart from resetting an open circuit or losing failure history.

What triggers the transition from OPEN to HALF_OPEN state?

Rather than using background timers, OmniRoute uses lazy recovery: the _refreshOpenState() method checks whether the resetTimeout has elapsed only when canExecute() is invoked by an incoming request. If the cooldown period has passed, the breaker atomically transitions to HALF_OPEN and permits the probe request to proceed.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}