How OmniRoute's Provider Circuit Breaker Works: Architecture and Implementation

OmniRoute prevents cascading failures by wrapping every LLM provider request in a four-state circuit breaker with failure-kind awareness, adaptive back-off, and persistent state recovery.

OmniRoute is an open-source LLM routing platform that implements enterprise-grade fault tolerance through its provider circuit breaker system. The implementation in src/shared/utils/circuitBreaker.ts extends the classic circuit breaker pattern with OmniRoute-specific enhancements like granular error classification and state persistence across restarts. This ensures that a single misbehaving provider cannot degrade the entire routing infrastructure.

Four-State Circuit Breaker Architecture

Unlike standard circuit breakers that use three states, OmniRoute implements a four-state machine: CLOSED, DEGRADED, OPEN, and HALF_OPEN. These states are defined at lines 12–16 of src/shared/utils/circuitBreaker.ts.

• CLOSED – The circuit operates normally and all requests pass through to the provider.
• DEGRADED – The failure rate has exceeded 60% of the configured threshold (the degradationThreshold), but traffic still flows while the system logs warnings.
• OPEN – The failure threshold has been breached; requests are short-circuited and immediately fail without reaching the provider.
• HALF_OPEN – A probe window allowing a limited number of requests (default 1) to test whether the provider has recovered.

State Transition Logic

When a failure occurs, the breaker increments both a global failure counter and a per-kind counter (lines 79–104). If the total count exceeds the degradationThreshold, the state shifts to DEGRADED. Upon reaching the full failureThreshold, the state jumps to OPEN. For error kinds flagged with immediateOpen, the circuit bypasses DEGRADED and opens immediately.

On success while in HALF_OPEN, the breaker resets to CLOSED. Otherwise, successful requests gradually decrement the failure count.

Failure Classification and Threshold Management

OmniRoute's circuit breaker implements failure-kind awareness, allowing different error types to trigger distinct thresholds and cooldown behaviors. The FailureKind enumeration (defined in src/shared/utils/classify429.ts) includes categories like rate_limit, quota_exhausted, and transient.

The CircuitBreakerOptions interface (lines 77–84) permits configuring kindThresholds, where each failure type can specify:

• Custom failure thresholds
• Unique cooldown durations
• An immediateOpen flag to bypass the degraded state

This granularity prevents transient network blips from triggering the same penalties as quota exhaustion.

Adaptive Back-off and Lazy Recovery

To prevent rapid state flapping on flaky providers, OmniRoute implements an adaptive back-off mechanism. Every time the circuit transitions from OPEN → HALF_OPEN → OPEN, the reset timeout doubles, up to a maxBackoffMultiplier limit (lines 55–63).

The system uses lazy recovery rather than background timers. When a request checks canExecute(), the code invokes _refreshOpenState() (lines 63–66), which transitions the breaker from OPEN to HALF_OPEN only after the cooldown period has elapsed. This eliminates the overhead of active polling.

Persistence and Memory Management

Circuit breaker state survives process restarts through the domainState table defined in src/lib/db/domainState.ts. On instantiation, each breaker calls loadCircuitBreakerState() (lines 97–112) to restore previous counters and state, ensuring that a restart does not reset protection against recently failed providers.

To prevent memory leaks in long-running processes, breakers are stored in a global Map registry with a maximum size of 500 instances (MAX_REGISTRY_SIZE). A periodic sweep removes idle, CLOSED breakers that have exceeded their idle timeout (lines 18–33).

Implementation Reference

The following pattern demonstrates how to wrap provider calls using the breaker factory:

// Import the breaker factory
import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';

// Create (or retrieve) a breaker for the OpenAI provider
const openaiBreaker = getCircuitBreaker('openai', {
  failureThreshold: 5,          // trip after 5 failures
  resetTimeout: 30_000,         // 30 s back-off
  halfOpenRequests: 1,          // one probe request
  // optional: classify 5xx errors as transient failures
  classifyError: (err) => (err?.statusCode >= 500 ? 'transient' : undefined),
});

// Wrap a provider call in the circuit-breaker
async function fetchChatCompletion(payload: any) {
  return openaiBreaker.execute(async () => {
    // This is the actual upstream request (fetch, axios, etc.)
    const resp = await fetch('https://api.openai.com/v1/chat/completions', {
      method: 'POST',
      body: JSON.stringify(payload),
      headers: { Authorization: `Bearer ${process.env.OPENAI_KEY}` },
    });
    return resp.json();
  });
}

// Check breaker status (useful for monitoring dashboards)
const status = openaiBreaker.getStatus();
console.log(`OpenAI breaker state: ${status.state}, retry after ${status.retryAfterMs} ms`);

The execute() method handles state checks internally, while canExecute() (used in src/sse/services/accountFallback.ts) allows pre-flight validation before attempting expensive operations.

Summary

• OmniRoute implements a four-state circuit breaker (CLOSED, DEGRADED, OPEN, HALF_OPEN) with a degradation threshold at 60% of the failure limit.
• Failure-kind awareness allows distinct handling of rate_limit, quota_exhausted, and transient errors via the kindThresholds configuration.
• Adaptive back-off doubles the reset timeout after each failed recovery attempt to prevent flapping.
• Lazy state recovery uses _refreshOpenState() triggered by request checks rather than background timers.
• State persists across restarts via the domainState table in src/lib/db/domainState.ts.
• A bounded registry with periodic sweep prevents memory leaks (MAX_REGISTRY_SIZE = 500).

Frequently Asked Questions

What are the four states of OmniRoute's provider circuit breaker?

The four states are CLOSED (normal operation), DEGRADED (high failure rate but traffic continues with warnings), OPEN (requests blocked), and HALF_OPEN (probe window for recovery testing). The DEGRADED state is unique to OmniRoute and triggers when failures exceed 60% of the configured threshold but have not yet reached the full trip limit.

How does the circuit breaker handle different types of failures?

OmniRoute classifies errors into kinds like rate_limit, quota_exhausted, and transient using the classifyError option. Each kind can define its own kindThresholds with specific failure limits, cooldown periods, and an immediateOpen flag that bypasses the degraded state entirely. This prevents temporary network errors from counting against the same threshold as authentication failures.

Where is circuit breaker state stored in OmniRoute?

State is persisted in the domainState table managed by src/lib/db/domainState.ts. The breaker serializes its current state, counters, and configuration via saveCircuitBreakerState() and restores them on process startup through loadCircuitBreakerState() (lines 97–112), ensuring continuity of protection across deployments.

How does the adaptive back-off mechanism prevent flapping?

Each time the circuit transitions from OPEN to HALF_OPEN and back to OPEN due to failed probes, the resetTimeout doubles up to the maxBackoffMultiplier limit. This exponential back-off prevents rapid cycling between open and closed states when a provider is experiencing intermittent issues, reducing load on both the provider and the routing system.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}