How OmniRoute's Circuit Breaker Resilience Layer Handles Provider Failures

TLDR: OmniRoute's circuit breaker resilience layer isolates failing downstream providers using a state machine that transitions from Closed to Degraded to Open based on configurable failure thresholds, rejects requests with CircuitBreakerOpenError when open, and recovers through Half-Open probes, with all states persisted to SQLite and managed through a global registry with automatic cleanup.

The OmniRoute platform safeguards AI provider integrations through a sophisticated circuit breaker resilience layer that prevents cascading failures across distributed chat services. Located in src/shared/utils/circuitBreaker.ts, this implementation wraps provider requests with automatic failure detection, granular threshold configuration, and durable state persistence to maintain system stability during outages.

Core State Machine and Architecture

In src/shared/utils/circuitBreaker.ts, each provider receives a dedicated circuit breaker instance through the getCircuitBreaker() registry function. The implementation follows the classic Closed → Degraded → Open → Half-Open → Closed state machine to manage provider health.

Breakers initialize with configurable options including failureThreshold, degradationThreshold, resetTimeout, and halfOpenRequests. The constructor applies these settings while attempting to restore previous state from the SQLite domainState table via _restoreFromDb().

Execution Wrapper and Failure Detection

The public execute(fn) method serves as the primary interface for wrapping provider calls. This method first refreshes cooldown timers, then evaluates whether to proceed:

• Immediate Rejection: If the breaker is OPEN or HALF_OPEN with no remaining probes, execute() throws CircuitBreakerOpenError without invoking the function.
• Execution: Otherwise, it runs the supplied async function and routes results to _onSuccess() or _onFailure() based on the isFailure predicate.

When a call succeeds, _onSuccess() immediately closes an OPEN or HALF_OPEN circuit, or gradually decays the failure counter in CLOSED/DEGRADED states. On failure, _onFailure(kind) increments global and per-kind counters, records timestamps, and evaluates threshold breaches.

Configurable Thresholds and Back-off Escalation

The circuit breaker resilience layer supports two tiers of failure detection:

Kind-Specific Thresholds: Configured via kindThresholds, specific failure types like rate_limit or quota_exhausted can trigger immediate opening (immediateOpen: true) or custom thresholds distinct from the global counter.

Global Thresholds: When aggregate failures reach degradationThreshold, the breaker transitions to DEGRADED. Crossing failureThreshold forces the circuit OPEN, blocking all subsequent requests until the reset timeout elapses.

The _effectiveResetTimeout() method calculates cooldown periods that escalate after repeated open-close cycles using backoffEscalationCount and maxBackoffMultiplier. Additionally, cooldownByKind allows per-failure-type timeout overrides for fine-tuned recovery semantics.

State Persistence and Registry Management

To survive process restarts, the breaker writes state changes to SQLite through saveCircuitBreakerState() in the domainState table. The _restoreFromDb() method hydrates breaker status on construction, ensuring continuity across deployments.

All breakers reside in a global Map managed by the registry. A periodic _registrySweep evicts CLOSED breakers idle for 30 minutes, enforcing a MAX_REGISTRY_SIZE of 500 entries to prevent memory leaks. Consumers like src/sse/handlers/chat.ts import getCircuitBreaker() and pass provider-specific options to protect request pipelines.

Implementation Example

The following pattern demonstrates protecting an OpenAI provider with custom thresholds:

import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';

const breaker = getCircuitBreaker('openai-gpt-4', {
  failureThreshold: 10,
  resetTimeout: 60_000,
  halfOpenRequests: 2,
  kindThresholds: {
    rate_limit: { threshold: 5, immediateOpen: true },
    quota_exhausted: { threshold: 3 },
  },
});

async function fetchChat(payload: any) {
  return breaker.execute(() => fetchOpenAIChat(payload));
}

For health monitoring, inspect breaker status across all providers:

import { getAllCircuitBreakerStatuses } from '@/shared/utils/circuitBreaker';

export async function getProviderHealth() {
  const statuses = getAllCircuitBreakerStatuses();
  return statuses.map(s => ({
    name: s.name,
    state: s.state,
    failures: s.failureCount,
    retryAfterMs: s.retryAfterMs,
  }));
}

Summary

• OmniRoute's circuit breaker resilience layer in src/shared/utils/circuitBreaker.ts implements a Closed → Degraded → Open → Half-Open state machine to isolate provider failures.
• The execute() method wraps async functions, throwing CircuitBreakerOpenError when circuits are open and routing outcomes to _onSuccess() or _onFailure() handlers.
• Kind-specific thresholds allow immediate tripping for critical failures like rate limits, while global thresholds trigger gradual degradation.
• Exponential back-off escalates reset timeouts through _effectiveResetTimeout() to prevent thundering herds during recovery.
• SQLite persistence via domainState ensures breaker states survive restarts, while the registry sweep caps memory usage at 500 breakers.

Frequently Asked Questions

How does OmniRoute's circuit breaker resilience layer differentiate between temporary and critical failures?

The implementation classifies failures through the kindThresholds configuration option. Critical failures like rate_limit can set immediateOpen: true to instantly open the circuit, while temporary errors respect the standard failureThreshold accumulation. This granular control prevents unnecessary isolation for transient network blips while immediately protecting against quota exhaustion.

What happens to requests when a provider's circuit breaker is in the Open state?

When the breaker is OPEN, the execute() method immediately rejects all incoming requests by throwing CircuitBreakerOpenError without invoking the wrapped function. This fast-fail behavior prevents resource exhaustion and gives the provider time to recover before the breaker transitions to HALF_OPEN and allows probe requests through.

How does the circuit breaker resilience layer maintain state across application restarts?

The breaker persists its current state to the SQLite domainState table via saveCircuitBreakerState() after every state transition. During construction, _restoreFromDb() attempts to load the previous state, ensuring that open circuits remain open and failure counts persist even through process restarts or deployments.

Why does OmniRoute use a global registry with automatic cleanup for circuit breakers?

The global Map registry in src/shared/utils/circuitBreaker.ts ensures that multiple call sites share the same breaker instance for a given provider name, maintaining consistent failure counting. The _registrySweep mechanism removes idle CLOSED breakers after 30 minutes of inactivity and enforces a MAX_REGISTRY_SIZE of 500, preventing memory leaks in long-running services that might dynamically create thousands of provider configurations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}