How Circuit Breakers Work in OmniRoute's Resilience Model: A Deep Dive into the State Machine Implementation

OmniRoute implements a four-state adaptive circuit breaker with exponential back-off escalation and per-failure-kind thresholds to isolate unhealthy providers and prevent cascading failures across its routing pipeline.

The circuit breaker is the cornerstone of OmniRoute's three-layer resilience strategy, living in src/shared/utils/circuitBreaker.ts and consumed by the provider-level resilience layer in open-sse/services/accountFallback.ts. This generic implementation gives fine-grained control over failure detection, adaptive cooldown, and degradation while persisting state across restarts.

The Four-State Circuit Breaker State Machine

The CircuitBreaker class defines four distinct states that govern traffic flow to a provider:

• CLOSED: Normal operation—requests pass through; failures are counted but traffic continues
• DEGRADED: Warning state—traffic still flows but is monitored more closely; serves as a buffer before full isolation
• OPEN: Traffic is blocked entirely; all requests are short-circuited immediately
• HALF_OPEN: Probe state—a limited number of trial requests are allowed to test recovery

State transitions are driven by two core methods: _onFailure and _onSuccess in src/shared/utils/circuitBreaker.ts#L53-L132.

How Transitions Occur

When _onFailure is invoked, it performs several operations:

1. Increments per-kind and global failure counters
2. Checks thresholds against kindThresholds configuration
3. Decides whether to move to DEGRADED or OPEN based on severity
4. Records the transition with timestamp for persistence

Conversely, _onSuccess can:

• Close an OPEN breaker on a successful probe in HALF_OPEN state
• Gradually decay failure counters while in CLOSED or DEGRADED to recover from transient issues

Adaptive Back-Off Escalation

A critical feature of OmniRoute's circuit breaker is the _effectiveResetTimeout method (src/shared/utils/circuitBreaker.ts#L55-L63), which implements exponential back-off for repeatedly failing providers.

// Simplified illustration of the back-off logic
const baseResetTimeout = 30_000; // 30 seconds
const multiplier = Math.min(2 ** tripCount, maxBackoffMultiplier); // capped at 16x
const effectiveTimeout = baseResetTimeout * multiplier; // 30s → 1min → 2min → ... → 8min max

This prevents flapping—where a provider repeatedly opens and closes the circuit—by giving longer recovery windows after each failed recovery attempt. The maxBackoffMultiplier defaults to 16×, yielding a maximum cooldown of 8 minutes.

Per-Failure-Kind Thresholds and Cooldowns

OmniRoute's breaker is failure-kind aware through two configuration maps:

• kindThresholds: Defines how many failures of a specific type trigger a state change
• cooldownByKind: Assigns distinct cooldown durations per error category

Key failure kinds include:

Kind	Typical Trigger	Behavior
rate_limit	HTTP 429 without retry-after	Standard threshold apply
quota_exhausted	HTTP 403 or 429 with exhaustion signal	May trigger immediateOpen
timeout	Request timeout	Higher threshold, longer back-off
network_error	DNS failure, proxy unreachable	Shorter threshold, quick isolation

Certain critical kinds can bypass the degraded state entirely via immediateOpen, instantly isolating a provider when specific errors occur (src/shared/utils/circuitBreaker.ts#L77-L99).

Persistence and Registry Management

The circuit breaker implements survive-restart persistence:

• Each breaker instance is stored in a global registry with a 500-entry maximum
• State changes are persisted via saveCircuitBreakerState to the domainState table
• A sweep worker evicts idle CLOSED breakers after 30 minutes
• Active OPEN and HALF_OPEN instances are never evicted

The registry is accessed through getCircuitBreaker(name, options) in src/shared/utils/circuitBreaker.ts#L66-L71, which returns existing instances on subsequent calls for the same provider name.

Provider-Level Integration

While the generic breaker lives in src/shared/utils/circuitBreaker.ts, OmniRoute's routing layer consumes it through open-sse/services/accountFallback.ts. The configureProviderBreaker function (accountFallback.ts#L984-L1016) builds provider-specific configurations from ProviderProfile settings.

Three key helpers form the public API for the routing layer:

• isProviderInCooldown(providerId) (accountFallback.ts#L1021-L1024): Checks breaker state before sending traffic
• recordProviderSuccess(providerId, connectionId) (accountFallback.ts#L1116-L1144): Signals successful completion
• recordProviderFailure(providerId, ...) (accountFallback.ts#L1051-L1099): Records failures with deduplication and classification

Deduplication Logic

Both recordProviderFailure and recordProviderSuccess implement connection-level deduplication—rapid retries on the same connection ID do not double-count failures. This prevents a single network partition from artificially inflating failure counts.

Complete Request Flow Example

// src/open-sse/services/accountFallback.ts — simplified pipeline
import {
  recordProviderFailure,
  recordProviderSuccess,
  isProviderInCooldown,
} from '@/open-sse/services/accountFallback';

async function executeProvider(providerId: string, requestBody: any) {
  // 1. Circuit breaker check
  if (isProviderInCooldown(providerId)) {
    throw new Error('Provider circuit breaker OPEN — routing to alternative');
  }

  try {
    const result = await fetchProviderApi(providerId, requestBody);
    
    // 2. Success path: decay counters or close circuit
    recordProviderSuccess(providerId);
    return result;
    
  } catch (err) {
    // 3. Failure path: classify and record
    const failureKind = classify429FromError(err); // rate_limit, quota_exhausted, etc.
    
    recordProviderFailure(providerId, logger, undefined, undefined, {
      isNetworkError: err.code === 'ECONNREFUSED',
      retryAfter: extractRetryAfter(err.response),
    });
    
    throw err;
  }
}

// Direct generic breaker usage for custom integrations
import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';

const anthropicBreaker = getCircuitBreaker('anthropic-claude', {
  failureThreshold: 5,
  resetTimeout: 30_000,
  halfOpenRequests: 1,
  maxBackoffMultiplier: 16,
  kindThresholds: {
    rate_limit: { threshold: 10, windowMs: 60_000 },
    quota_exhausted: { threshold: 1, immediateOpen: true },
  },
});

// Execute guarded request
const response = await anthropicBreaker.execute(async () => {
  return fetch('https://api.anthropic.com/v1/messages', {
    method: 'POST',
    headers: { 'x-api-key': process.env.ANTHROPIC_KEY },
    body: JSON.stringify({ model: 'claude-3-opus-20240229', messages: prompts }),
  });
});

Monitoring and Observability

Administrators can inspect circuit breaker states through:

• getAllCircuitBreakerStatuses(): Returns complete registry snapshot for dashboards
• Per-breaker metadata: current state, failure counts per kind, next retry timestamp, transition history

These endpoints power OmniRoute's operational visibility, allowing operators to identify provider health trends and adjust thresholds without code changes.

Configuration Sources

File	Purpose
src/lib/resilience/settings.ts	Default thresholds, timeouts, and degradation settings
ProviderProfile (runtime)	Per-provider overrides from configuration store
accountFallback.ts#L984-L1016	Profile-to-breaker conversion logic

Summary

• OmniRoute's circuit breaker implements a four-state machine (CLOSED, DEGRADED, OPEN, HALF_OPEN) in src/shared/utils/circuitBreaker.ts
• Adaptive back-off through _effectiveResetTimeout escalates cooldown exponentially to prevent flapping
• Failure-kind awareness via kindThresholds and cooldownByKind enables differentiated handling of error categories
• Persistence to domainState table ensures breaker state survives process restarts
• Provider-level integration through open-sse/services/accountFallback.ts provides deduplication, classification, and routing integration
• Registry management bounds memory usage to 500 entries with 30-minute idle eviction for closed breakers

Frequently Asked Questions

What triggers a circuit breaker to open in OmniRoute?

A circuit breaker opens when failure counts exceed configured thresholds within a time window, or immediately for certain critical failure kinds marked with immediateOpen. The _onFailure method in src/shared/utils/circuitBreaker.ts#L79-L132 evaluates per-kind counters against kindThresholds and transitions to DEGRADED or OPEN accordingly. Repeated quota exhaustion (quota_exhausted) typically opens instantly, while transient rate limits require multiple occurrences.

How does OmniRoute prevent a recovered provider from being immediately overwhelmed?

The HALF_OPEN state allows only halfOpenRequests (default 1) probe requests to test recovery. Successful probes transition the breaker to CLOSED; failures reopen with escalated back-off. This gradual re-admission pattern prevents thundering herds against recovering endpoints.

Can circuit breaker state survive an OmniRoute process restart?

Yes. OmniRoute persists breaker state to the domainState database table via saveCircuitBreakerState on every transition. On startup, the registry reloads active states, ensuring that an OPEN breaker remains open and its back-off timer is respected. Idle CLOSED breakers are not persisted to reduce storage churn.

How do I configure different thresholds for different error types?

Pass a kindThresholds map to getCircuitBreaker or define it in the ProviderProfile consumed by configureProviderBreaker. Each kind specifies its own threshold, windowMs, and optional immediateOpen flag. The cooldownByKind map separately configures recovery timeouts per error category.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}