How OmniRoute's Provider Circuit Breaker Prevents Cascading Failures in AI Services

OmniRoute isolates misbehaving LLM providers using a state machine that tracks failure rates and short-circuits traffic when thresholds are exceeded, ensuring that a single provider outage cannot cascade into a total service failure.

The diegosouzapw/OmniRoute repository implements a robust provider-level circuit breaker that protects AI routing infrastructure from cascading failures. When upstream LLM APIs experience outages or degraded performance, this mechanism automatically blocks traffic to failing providers while maintaining service availability through healthy alternatives. Understanding how this Circuit Breaker architecture classifies errors and manages state transitions is essential for operating resilient AI service aggregators.

Circuit Breaker State Machine Architecture

In src/shared/utils/circuitBreaker.ts, the core implementation follows a classic state machine pattern with four distinct states that govern provider availability.

The Four States of Provider Isolation

The breaker cycles through these states based on real-time failure metrics:

• CLOSED: Normal operation where requests flow to the provider without interference.
• DEGRADED: A transitional warning state where failure rates rise but traffic continues; the system logs warnings to alert operators before full isolation (lines 24-31).
• OPEN: The provider is temporarily blocked; the execute() method (lines 269-275) throws a CircuitBreakerOpenError, instantly aborting requests and preventing additional load.
• HALF_OPEN: After a cooldown period, a limited number of probe requests are permitted to test provider recovery before fully reinstating traffic.

Failure Detection and Classification Mechanisms

Precise error classification prevents the breaker from reacting to transient or irrelevant faults while aggressively isolating genuine upstream failures.

Granular Error Classification

The classifyError and isFailure utilities distinguish between fatal upstream errors and harmless local conditions. The helper isLocalStreamLifecycleError (lines 46-65) specifically filters out local stream-lifecycle errors, ensuring that only upstream 5xx responses and quota violations increment failure counters. This granularity prevents false positives from client-side disconnections or network blips.

Adaptive Exponential Back-off

When a provider repeatedly fails recovery probes, the _effectiveResetTimeout logic (lines 55-63) implements exponential back-off. Each OPEN → HALF_OPEN → OPEN cycle increases the cooldown duration, ensuring that flapping providers remain isolated from the routing pool for progressively longer periods. This adaptive mechanism automatically adjusts to persistent instability without manual configuration changes.

State Persistence Across Restarts

The breaker persists its state to the domainState database table through _persistToDb and _restoreFromDb methods (lines 31-35). This ensures that protection survives server restarts—if a provider was blocked before deployment, it remains blocked afterward, preventing immediate re-flooding of a still-recovering upstream service.

Memory Management and Registry Operations

The src/lib/warmupScheduler/circuitBreakerStore.ts maintains an in-memory registry of active breakers created by src/lib/warmupScheduler/circuitBreakerFactory.ts. A background sweep process (lines 18-33 in circuitBreaker.ts) evicts idle, healthy breakers after 30 minutes of inactivity. This keeps the runtime memory footprint low while preserving active protection for currently utilized providers.

Integration with the Routing Layer

When the circuit breaker enters the OPEN state, the combo routing logic in open-sse/services/combo.ts automatically excludes the blocked provider from candidate selection. The router continues distributing traffic across remaining healthy providers, guaranteeing that a single point of failure cannot propagate through the AI service infrastructure.

Implementation Example

The following pattern demonstrates how to instantiate and utilize a provider-specific circuit breaker:

// Create a circuit breaker for the OpenAI provider
import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';

// Provider-specific failure detection (counts only 5xx upstream errors)
const openAiBreaker = getCircuitBreaker('openai', {
  failureThreshold: 10,
  resetTimeout: 60_000, // 1 min initial cooldown
  isFailure: (err) => err?.statusCode >= 500,
});

// Wrap an HTTP call to the provider
async function fetchChatCompletion(payload: any) {
  return openAiBreaker.execute(async () => {
    // … actual fetch to OpenAI…
    const resp = await fetch('https://api.openai.com/v1/chat/completions', {
      method: 'POST',
      headers: { Authorization: `Bearer ${process.env.OPENAI_KEY}` },
      body: JSON.stringify(payload),
    });
    if (!resp.ok) throw new Error(`HTTP ${resp.status}`);
    return resp.json();
  });
}

// Using the breaker in a route handler
export async function handleChatCore(req) {
  try {
    const result = await fetchChatCompletion(req.body);
    return { status: 200, data: result };
  } catch (e) {
    if (e instanceof CircuitBreakerOpenError) {
      // Provider is temporarily blocked – fallback to another model
      return fallbackToOtherProvider(req);
    }
    throw e; // Let higher-level error handling deal with other errors
  }
}

Summary

• OmniRoute's provider Circuit Breaker lives in src/shared/utils/circuitBreaker.ts and implements a four-state machine (CLOSED, DEGRADED, OPEN, HALF_OPEN) to isolate failing LLM providers.
• Granular error classification via isLocalStreamLifecycleError ensures only genuine upstream failures trigger state transitions, while local stream errors are ignored.
• Adaptive back-off increases isolation periods for repeatedly failing providers, preventing flapping services from re-entering the pool prematurely.
• State persistence to the domainState table maintains protection across server restarts, eliminating vulnerability windows during deployments.
• Automatic routing failover in open-sse/services/combo.ts skips OPEN providers, ensuring cascading failures cannot propagate through the AI service infrastructure.

Frequently Asked Questions

What triggers a circuit breaker to open in OmniRoute?

When the total failure count exceeds the configured failureThreshold or a per-kind threshold for specific error types (such as rate_limit or quota_exhausted), the _openCircuit method transitions the state to OPEN. This typically occurs after 10 or more upstream 5xx errors within the observation window, depending on provider-specific configuration.

How does the circuit breaker handle different types of errors?

The implementation uses classifyError and isFailure predicates to categorize errors. Local stream-lifecycle errors identified by isLocalStreamLifecycleError (lines 46-65) are excluded from failure counts, while upstream quota exhaustion, rate limits, and 5xx responses increment the counters. This distinction prevents client-side issues from unnecessarily blocking providers.

What happens to requests when a provider's circuit is open?

While the circuit is OPEN, the execute() method (lines 269-275) immediately throws a CircuitBreakerOpenError without attempting the upstream request. The routing layer catches this exception and redirects traffic to alternative providers, effectively short-circuiting the failing path and preventing additional load on the distressed service.

How does OmniRoute ensure circuit breaker state survives server restarts?

The breaker persists state to a database table named domainState through _persistToDb and restores it via _restoreFromDb (lines 31-35). This persistence ensures that if a provider was blocked before a deployment or crash, it remains blocked upon restart, protecting the recovering upstream service from immediate traffic floods.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}