# How Circuit Breakers Work in OmniRoute's Resilience Model: A Deep Dive into the State Machine Implementation

> Discover how OmniRoute's resilience model uses circuit breakers to prevent cascading failures. Learn about its adaptive state machine and exponential back-off for robust routing.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: deep-dive
- Published: 2026-08-26

---

**OmniRoute implements a four-state adaptive circuit breaker with exponential back-off escalation and per-failure-kind thresholds to isolate unhealthy providers and prevent cascading failures across its routing pipeline.**

The circuit breaker is the cornerstone of OmniRoute's three-layer resilience strategy, living in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) and consumed by the provider-level resilience layer in [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts). This generic implementation gives fine-grained control over failure detection, adaptive cooldown, and degradation while persisting state across restarts.

## The Four-State Circuit Breaker State Machine

The `CircuitBreaker` class defines four distinct states that govern traffic flow to a provider:

- **CLOSED**: Normal operation—requests pass through; failures are counted but traffic continues
- **DEGRADED**: Warning state—traffic still flows but is monitored more closely; serves as a buffer before full isolation
- **OPEN**: Traffic is blocked entirely; all requests are short-circuited immediately
- **HALF_OPEN**: Probe state—a limited number of trial requests are allowed to test recovery

State transitions are driven by two core methods: [`_onFailure`](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.51/src/shared/utils/circuitBreaker.ts#L79-L132) and [`_onSuccess`](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.51/src/shared/utils/circuitBreaker.ts#L53-L76) in `src/shared/utils/circuitBreaker.ts#L53-L132`.

### How Transitions Occur

When `_onFailure` is invoked, it performs several operations:

1. Increments per-kind and global failure counters
2. Checks thresholds against `kindThresholds` configuration
3. Decides whether to move to `DEGRADED` or `OPEN` based on severity
4. Records the transition with timestamp for persistence

Conversely, `_onSuccess` can:
- Close an `OPEN` breaker on a successful probe in `HALF_OPEN` state
- Gradually decay failure counters while in `CLOSED` or `DEGRADED` to recover from transient issues

## Adaptive Back-Off Escalation

A critical feature of OmniRoute's circuit breaker is the `_effectiveResetTimeout` method (`src/shared/utils/circuitBreaker.ts#L55-L63`), which implements **exponential back-off** for repeatedly failing providers.

```typescript
// Simplified illustration of the back-off logic
const baseResetTimeout = 30_000; // 30 seconds
const multiplier = Math.min(2 ** tripCount, maxBackoffMultiplier); // capped at 16x
const effectiveTimeout = baseResetTimeout * multiplier; // 30s → 1min → 2min → ... → 8min max

```

This prevents **flapping**—where a provider repeatedly opens and closes the circuit—by giving longer recovery windows after each failed recovery attempt. The `maxBackoffMultiplier` defaults to 16×, yielding a maximum cooldown of 8 minutes.

## Per-Failure-Kind Thresholds and Cooldowns

OmniRoute's breaker is **failure-kind aware** through two configuration maps:

- `kindThresholds`: Defines how many failures of a specific type trigger a state change
- `cooldownByKind`: Assigns distinct cooldown durations per error category

Key failure kinds include:

| Kind | Typical Trigger | Behavior |
|------|---------------|----------|
| `rate_limit` | HTTP 429 without retry-after | Standard threshold apply |
| `quota_exhausted` | HTTP 403 or 429 with exhaustion signal | May trigger `immediateOpen` |
| `timeout` | Request timeout | Higher threshold, longer back-off |
| `network_error` | DNS failure, proxy unreachable | Shorter threshold, quick isolation |

Certain critical kinds can bypass the degraded state entirely via `immediateOpen`, instantly isolating a provider when specific errors occur (`src/shared/utils/circuitBreaker.ts#L77-L99`).

## Persistence and Registry Management

The circuit breaker implements **survive-restart persistence**:

- Each breaker instance is stored in a global registry with a 500-entry maximum
- State changes are persisted via `saveCircuitBreakerState` to the `domainState` table
- A sweep worker evicts idle **CLOSED** breakers after 30 minutes
- Active `OPEN` and `HALF_OPEN` instances are never evicted

The registry is accessed through `getCircuitBreaker(name, options)` in `src/shared/utils/circuitBreaker.ts#L66-L71`, which returns existing instances on subsequent calls for the same provider name.

## Provider-Level Integration

While the generic breaker lives in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts), OmniRoute's routing layer consumes it through [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts). The `configureProviderBreaker` function (`accountFallback.ts#L984-L1016`) builds provider-specific configurations from `ProviderProfile` settings.

Three key helpers form the public API for the routing layer:

- **`isProviderInCooldown(providerId)`** (`accountFallback.ts#L1021-L1024`): Checks breaker state before sending traffic
- **`recordProviderSuccess(providerId, connectionId)`** (`accountFallback.ts#L1116-L1144`): Signals successful completion
- **`recordProviderFailure(providerId, ...)`** (`accountFallback.ts#L1051-L1099`): Records failures with deduplication and classification

### Deduplication Logic

Both `recordProviderFailure` and `recordProviderSuccess` implement **connection-level deduplication**—rapid retries on the same connection ID do not double-count failures. This prevents a single network partition from artificially inflating failure counts.

## Complete Request Flow Example

```typescript
// src/open-sse/services/accountFallback.ts — simplified pipeline
import {
  recordProviderFailure,
  recordProviderSuccess,
  isProviderInCooldown,
} from '@/open-sse/services/accountFallback';

async function executeProvider(providerId: string, requestBody: any) {
  // 1. Circuit breaker check
  if (isProviderInCooldown(providerId)) {
    throw new Error('Provider circuit breaker OPEN — routing to alternative');
  }

  try {
    const result = await fetchProviderApi(providerId, requestBody);
    
    // 2. Success path: decay counters or close circuit
    recordProviderSuccess(providerId);
    return result;
    
  } catch (err) {
    // 3. Failure path: classify and record
    const failureKind = classify429FromError(err); // rate_limit, quota_exhausted, etc.
    
    recordProviderFailure(providerId, logger, undefined, undefined, {
      isNetworkError: err.code === 'ECONNREFUSED',
      retryAfter: extractRetryAfter(err.response),
    });
    
    throw err;
  }
}

```

```typescript
// Direct generic breaker usage for custom integrations
import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';

const anthropicBreaker = getCircuitBreaker('anthropic-claude', {
  failureThreshold: 5,
  resetTimeout: 30_000,
  halfOpenRequests: 1,
  maxBackoffMultiplier: 16,
  kindThresholds: {
    rate_limit: { threshold: 10, windowMs: 60_000 },
    quota_exhausted: { threshold: 1, immediateOpen: true },
  },
});

// Execute guarded request
const response = await anthropicBreaker.execute(async () => {
  return fetch('https://api.anthropic.com/v1/messages', {
    method: 'POST',
    headers: { 'x-api-key': process.env.ANTHROPIC_KEY },
    body: JSON.stringify({ model: 'claude-3-opus-20240229', messages: prompts }),
  });
});

```

## Monitoring and Observability

Administrators can inspect circuit breaker states through:

- **`getAllCircuitBreakerStatuses()`**: Returns complete registry snapshot for dashboards
- Per-breaker metadata: current state, failure counts per kind, next retry timestamp, transition history

These endpoints power OmniRoute's operational visibility, allowing operators to identify provider health trends and adjust thresholds without code changes.

## Configuration Sources

| File | Purpose |
|------|---------|
| [`src/lib/resilience/settings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/settings.ts) | Default thresholds, timeouts, and degradation settings |
| `ProviderProfile` (runtime) | Per-provider overrides from configuration store |
| `accountFallback.ts#L984-L1016` | Profile-to-breaker conversion logic |

## Summary

- OmniRoute's **circuit breaker** implements a four-state machine (`CLOSED`, `DEGRADED`, `OPEN`, `HALF_OPEN`) in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts)
- **Adaptive back-off** through `_effectiveResetTimeout` escalates cooldown exponentially to prevent flapping
- **Failure-kind awareness** via `kindThresholds` and `cooldownByKind` enables differentiated handling of error categories
- **Persistence** to `domainState` table ensures breaker state survives process restarts
- **Provider-level integration** through [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts) provides deduplication, classification, and routing integration
- **Registry management** bounds memory usage to 500 entries with 30-minute idle eviction for closed breakers

## Frequently Asked Questions

### What triggers a circuit breaker to open in OmniRoute?

A circuit breaker opens when failure counts exceed configured thresholds within a time window, or immediately for certain critical failure kinds marked with `immediateOpen`. The `_onFailure` method in `src/shared/utils/circuitBreaker.ts#L79-L132` evaluates per-kind counters against `kindThresholds` and transitions to `DEGRADED` or `OPEN` accordingly. Repeated quota exhaustion (`quota_exhausted`) typically opens instantly, while transient rate limits require multiple occurrences.

### How does OmniRoute prevent a recovered provider from being immediately overwhelmed?

The `HALF_OPEN` state allows only `halfOpenRequests` (default 1) probe requests to test recovery. Successful probes transition the breaker to `CLOSED`; failures reopen with escalated back-off. This **gradual re-admission** pattern prevents thundering herds against recovering endpoints.

### Can circuit breaker state survive an OmniRoute process restart?

Yes. OmniRoute persists breaker state to the `domainState` database table via `saveCircuitBreakerState` on every transition. On startup, the registry reloads active states, ensuring that an `OPEN` breaker remains open and its back-off timer is respected. Idle `CLOSED` breakers are not persisted to reduce storage churn.

### How do I configure different thresholds for different error types?

Pass a `kindThresholds` map to `getCircuitBreaker` or define it in the `ProviderProfile` consumed by `configureProviderBreaker`. Each kind specifies its own `threshold`, `windowMs`, and optional `immediateOpen` flag. The `cooldownByKind` map separately configures recovery timeouts per error category.