How OmniRoute's Circuit Breaker Prevents Cascading Provider Failures

OmniRoute isolates failing upstream providers by wrapping every API call in a per-provider circuit breaker that tracks error counts and opens when configurable thresholds are exceeded, immediately returning a CircuitBreakerOpenError instead of hitting the upstream to stop cascading failures through the combo routing pipeline.

OmniRoute, an open-source multi-provider routing engine, implements a defensive resilience layer to prevent a single misbehaving AI provider from triggering retry storms and resource exhaustion across the entire request pipeline. The circuit breaker pattern, implemented in src/shared/utils/circuitBreaker.ts, maintains independent state machines for each provider endpoint to ensure localized failure containment. This architecture guarantees that HTTP errors like 502, 429, or 5xx responses from one provider do not propagate through the combo routing logic and degrade the entire system.

Per-Provider Isolation Architecture

OmniRoute’s circuit breaker guarantees that only the failing provider is blocked, not the entire routing combo or other healthy upstreams.

The In-Memory Registry (getCircuitBreaker)

Each provider receives its own CircuitBreaker instance via the getCircuitBreaker(name) function, which stores instances in an in-memory registry defined in src/shared/utils/circuitBreaker.ts. This registry maps provider identifiers to specific breaker objects, ensuring complete isolation between different API endpoints. When the combo router iterates through multiple targets, it retrieves a unique breaker for each provider using getCircuitBreaker(provider:${providerId}), preventing cross-provider state contamination.

Persistent State Across Restarts (saveCircuitBreakerState)

To prevent a failing provider from being silently retried after a server crash or deployment, the breaker persists its state to SQLite using saveCircuitBreakerState and loadCircuitBreakerState. The internal methods _persistToDb() and _restoreFromDb() serialize the current state (OPEN, CLOSED, etc.) and failure counts to the database layer in src/lib/db/domainState.ts. This persistence ensures that an open circuit remains open across process restarts, maintaining protection continuity without manual operator intervention.

State Machine and Failure Detection

The circuit breaker operates as a deterministic state machine transitioning between CLOSED, DEGRADED, OPEN, HALF_OPEN, and back to CLOSED, with specific triggers for each transition.

CLOSED to DEGRADED to OPEN Transitions

In the CLOSED state, the breaker forwards all requests normally while tracking failures. When the failure count reaches 60% of the configured threshold, the breaker transitions to DEGRADED—still forwarding requests but emitting warning logs for early visibility. Once errors exceed the full failureThreshold, the breaker OPENs, and all subsequent calls immediately short-circuit with a CircuitBreakerOpenError without ever reaching the network layer. This transition logic is handled in the _onFailure() method within the core implementation file.

Failure-Kind Aware Thresholds (kindThresholds)

Not all HTTP errors are treated equally. The implementation supports granular failure classification via kindThresholds and cooldownByKind maps. For example, rate-limit responses (429) can trigger an immediate open with immediateOpen: true, while transient 500 errors might tolerate higher retry counts. This discrimination prevents overly aggressive breaker opening on recoverable glitches while reacting instantly to quota exhaustion or authentication failures.

Recovery and Back-Off Strategies

Automatic recovery mechanisms ensure that providers are not blacklisted indefinitely and that flaky endpoints do not waste resources with rapid retry attempts.

Exponential Back-Off (_effectiveResetTimeout)

After each failed recovery attempt (OPEN → HALF_OPEN → OPEN), the reset timeout multiplies exponentially up to a maxBackoffMultiplier. The _effectiveResetTimeout() method calculates this dynamic delay, ensuring that repeatedly failing providers face progressively longer cooldowns (e.g., 30s → 60s → 120s) rather than fixed intervals. This back-off strategy protects the routing pipeline from wasting cycles on unstable upstreams.

Half-Open Probing (halfOpenRequests)

When the reset timeout expires, the breaker enters HALF_OPEN state and allows a limited number of probe requests—configured via halfOpenRequests—to test provider recovery. The execute() method permits these probes while _refreshOpenState() monitors their success rate. If probes succeed, the breaker closes; if they fail, it reopens immediately. This controlled probing enables fast recovery detection without flooding the recovering provider with full traffic volume.

Registry Eviction for Memory Management

To prevent memory bloat in long-running instances, the registry sweeps idle breakers every 30 minutes via evictColdBreakersIfNeeded(). Breakers in CLOSED state with no recent activity are removed from the in-memory map, keeping the registry size bounded while preserving active or open circuits. This cleanup runs automatically and requires no manual configuration.

Integration with the Combo Router

The combo routing engine in open-sse/services/combo.ts leverages the breaker to implement the short-circuit pattern. Before attempting any provider call, the router checks breaker.canExecute():

import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';

async function routeCombo(request: Request, combo: ComboConfig) {
  for (const target of combo.targets) {
    const breaker = getCircuitBreaker(`provider:${target.providerId}`);
    
    if (!breaker.canExecute()) {
      continue; // Skip failing provider immediately
    }

    try {
      return await breaker.execute(async () => {
        const resp = await fetch(target.url, { body: request.body });
        if (!resp.ok) throw new Error(`HTTP ${resp.status}`);
        return resp.json();
      });
    } catch (err) {
      // Error already recorded by breaker; try next target
    }
  }
  throw new Error('All combo targets failed');
}

When canExecute() returns false, the router skips to the next provider in the combo strategy, effectively isolating the failure. The execute() wrapper automatically records success or failure, updating the state machine without manual instrumentation.

Configuration and Usage Examples

Creating a breaker with custom thresholds for OAuth-sensitive providers:

import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';

const breaker = getCircuitBreaker(`provider:${providerId}`, {
  failureThreshold: 5,
  resetTimeout: 30_000,
  halfOpenRequests: 1,
  kindThresholds: {
    rate_limit: { threshold: 1, immediateOpen: true },
    quota_exhausted: { threshold: 1, immediateOpen: true }
  },
  maxBackoffMultiplier: 5
});

Checking breaker status before manual calls:

if (!breaker.canExecute()) {
  throw new CircuitBreakerOpenError(
    `Provider ${providerId} is circuit-open`,
    providerId,
    breaker.getRetryAfterMs()
  );
}

Administrative reset for testing or manual recovery:

import { resetAllCircuitBreakers } from '@/shared/utils/circuitBreaker';

// Clears all open states and DB entries
resetAllCircuitBreakers();

Summary

• Per-provider isolation via getCircuitBreaker() ensures failures stay contained to individual upstreams.
• State persistence through _persistToDb() and _restoreFromDb() maintains protection across server restarts.
• Graduated state machine (CLOSED → DEGRADED → OPEN) provides early warnings before full isolation.
• Kind-aware thresholds allow aggressive opening on critical errors like rate limits while tolerating transient failures.
• Exponential back-off in _effectiveResetTimeout() prevents resource waste on repeatedly flaky providers.
• Half-open probing via halfOpenRequests enables safe recovery detection without traffic flooding.
• Automatic eviction via evictColdBreakersIfNeeded() prevents memory leaks in the breaker registry.

Frequently Asked Questions

What is a circuit breaker in OmniRoute?

A circuit breaker in OmniRoute is a per-provider state machine that monitors failure rates and opens when error thresholds are exceeded, immediately returning errors instead of attempting upstream requests. This pattern prevents cascading failures by isolating unhealthy providers from the routing combo, protecting system resources and maintaining overall request latency.

How does OmniRoute handle circuit breaker state after a server restart?

OmniRoute persists circuit breaker states to SQLite using saveCircuitBreakerState and loadCircuitBreakerState functions, ensuring that an open circuit remains open after a crash or deployment. The internal _persistToDb() method serializes state changes to the database, while _restoreFromDb() hydrates the breaker registry on startup, maintaining continuity of protection without manual reset.

What happens when a provider circuit breaker opens?

When a breaker opens, the canExecute() method returns false, causing the combo router to skip that provider entirely and attempt the next target in the sequence. Any direct calls to execute() while open immediately throw a CircuitBreakerOpenError without making an HTTP request, eliminating network latency and preventing load on the failing upstream.

How does the combo router handle multiple provider failures?

The combo router iterates through its target list, checking breaker.canExecute() for each provider before attempting the request. If a provider is open, the router short-circuits that iteration and moves to the next healthy target. This allows the request to fail over through multiple providers until finding a healthy one, or exhausting all options, ensuring maximum availability despite individual upstream failures.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}