How OmniRoute's Provider Circuit Breaker Works: A Deep Dive into the Implementation

OmniRoute protects your LLM routing system from provider failures by wrapping every upstream request in a stateful CircuitBreaker instance that automatically degrades, opens, and recovers based on configurable failure thresholds.

The circuit breaker lives in src/shared/utils/circuitBreaker.ts and extends the classic pattern with failure-kind awareness, adaptive back-off, and persistent state recovery. This guide examines the four-state implementation, configuration options, and how the breaker integrates with OmniRoute's multi-provider routing logic.

---

Four-State Circuit Breaker Design

Unlike traditional three-state breakers, OmniRoute uses four distinct states to provide finer-grained control over provider health:

State	Behavior
CLOSED	Traffic passes normally; success decrements failure counters.
DEGRADED	Failure rate elevated but below threshold; requests proceed with logged warnings.
OPEN	All requests short-circuit immediately; no upstream calls made.
HALF_OPEN	Probe window allows limited requests to test provider recovery.

The STATE enum is defined at src/shared/utils/circuitBreaker.ts#L12-L16. This four-state design lets operators monitor when a provider is "limping" before it fully fails.

---

Failure-Kind Awareness and Classification

OmniRoute's breaker goes beyond simple success/failure counting. It classifies errors into per-kind categories with independent thresholds:

// From src/shared/utils/circuitBreaker.ts#L77-84
interface KindThresholds {
  [kind: string]: {
    threshold: number;      // failures before this kind trips the breaker
    resetTimeout?: number;  // optional override for this kind's cooldown
    immediateOpen?: boolean; // true = bypass DEGRADED, jump straight to OPEN
  };
}

Common failure kinds include:

• rate_limit (HTTP 429 with retry-after headers)
• quota_exhausted (hard limit reached)
• transient (5xx server errors, timeouts)

This classification lets you treat a flaky 5xx differently from a quota exhaustion. For example, immediateOpen: true on quota_exhausted disables a provider instantly rather than waiting for five failures.

---

State Transitions and Thresholds

The breaker uses two thresholds to drive state changes:

1. degradationThreshold — defaults to 60% of failureThreshold
2. failureThreshold — absolute limit before circuit opens

When a failure occurs, the logic at src/shared/utils/circuitBreaker.ts#L79-L104:

1. Increments global and per-kind failure counters
2. Compares against degradationThreshold → move to DEGRADED
3. Compares against failureThreshold → move to OPEN (or check immediateOpen flag)

On success in HALF_OPEN, the breaker resets to CLOSED and clears all counters. Success in other states gradually decrements failure counts via proportional reduction.

---

Adaptive Back-Off and Lazy Recovery

To prevent flapping (rapid OPEN/CLOSED oscillation), OmniRoute implements exponential back-off:

// From src/shared/utils/circuitBreaker.ts#L55-63
private getEffectiveResetTimeout(): number {
  const multiplier = Math.min(
    this.backoffMultiplier,
    this.options.maxBackoffMultiplier ?? 5
  );
  return this.options.resetTimeout * Math.pow(2, multiplier - 1);
}

Each full cycle of OPEN → HALF_OPEN → OPEN doubles the reset timeout up to the configured maximum.

The breaker uses lazy recovery — no background timers run. Instead, canExecute() checks _refreshOpenState() at src/shared/utils/circuitBreaker.ts#L63-L66, comparing Date.now() against the stored open timestamp. This eliminates timer management overhead and works correctly across process restarts.

---

Persistence Across Restarts

Circuit breaker state survives process restarts through the domainState database table (src/lib/db/domainState.ts):

Stored Data	Purpose
state	Current breaker state (serialized enum)
counters	Global and per-kind failure counts
options	Configuration snapshot
openedAt / halfOpenedAt	Timestamps for cooldown calculation
backoffMultiplier	Current back-off level

On construction, each breaker loads prior state via loadCircuitBreakerState() and restores its position in the state machine. This prevents a restart from "forgiving" a misbehaving provider.

---

Registry Management and Memory Boundaries

Breakers are cached in a global Map keyed by provider name. To prevent unbounded growth:

• Maximum registry size: 500 active breakers (MAX_REGISTRY_SIZE)
• Periodic sweep: Idle breakers in CLOSED state are evicted
• Factory pattern: getCircuitBreaker(name, options) retrieves existing or creates new

The registry sweep runs at src/shared/utils/circuitBreaker.ts#L18-L33.

---

Practical Usage Example

Here's how to integrate the breaker into a provider handler:

import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';

// Configure provider-specific thresholds
const anthropicBreaker = getCircuitBreaker('anthropic', {
  failureThreshold: 5,
  resetTimeout: 30_000,
  halfOpenRequests: 1,
  classifyError: (err) => {
    if (err.status === 529) return 'rate_limit';
    if (err.status >= 500) return 'transient';
    return undefined;
  },
});

// Wrap upstream call
async function callAnthropic(messages: Message[]) {
  return anthropicBreaker.execute(async () => {
    const response = await fetch('https://api.anthropic.com/v1/messages', {
      method: 'POST',
      headers: { 'x-api-key': process.env.ANTHROPIC_KEY },
      body: JSON.stringify({ messages, model: 'claude-3-opus-20240229' }),
    });
    if (!response.ok) throw await response.json();
    return response.json();
  });
}

// Pre-flight check for routing decisions
function canRouteToAnthropic(): boolean {
  return anthropicBreaker.canExecute();
}

The execute() method handles all state machine logic, error classification, and persistence automatically.

---

Integration with OmniRoute's Fallback System

In src/sse/services/accountFallback.ts, the breaker drives provider selection:

// Simplified excerpt from account fallback logic
if (!breaker.canExecute()) {
  // Skip this provider key, try next in rotation
  return tryNextProvider();
}

When a breaker is OPEN, the combo router immediately excludes that provider from consideration. When HALF_OPEN, only one concurrent probe request is permitted — subsequent calls fail fast until the probe completes.

---

Summary

• Four states (CLOSED, DEGRADED, OPEN, HALF_OPEN) provide granular health visibility
• Per-failure-kind thresholds let you customize behavior for rate limits, quotas, and transient errors
• Adaptive back-off with exponential increase prevents flapping on flaky providers
• Lazy recovery eliminates timer overhead while maintaining accurate cooldowns
• Database persistence ensures protection survives process restarts
• Registry bounds keep memory usage predictable with automatic cleanup

---

Frequently Asked Questions

How does OmniRoute's circuit breaker differ from standard implementations?

Standard breakers use three states (CLOSED, OPEN, HALF_OPEN). OmniRoute adds DEGRADED as an intermediate warning state, plus failure-kind classification that lets rate limits, quota errors, and transient failures have independent thresholds and behaviors. The adaptive back-off and database persistence are also atypical enhancements.

What happens when a provider circuit opens mid-request?

The breaker checks canExecute() before any upstream call. If already OPEN, the request aborts immediately with a standardized error that the routing layer interprets as "skip this provider." No network I/O occurs, and the combo router selects an alternative provider from its pool.

Can I disable circuit breaking for specific providers?

Yes — pass failureThreshold: Infinity or a very high number in CircuitBreakerOptions. However, this removes protection against cascading failures. A better practice is setting immediateOpen: false for specific failure kinds while keeping the overall breaker active.

How do I monitor circuit breaker health?

Call breaker.getStatus() to retrieve:

{
  state: 'OPEN' | 'CLOSED' | 'DEGRADED' | 'HALF_OPEN',
  failureCount: number,
  retryAfterMs: number,  // 0 if canExecute() would return true
  lastFailureKind?: string
}

Integrate these metrics into your observability stack to detect provider degradation before users experience errors.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}