# How to Troubleshoot OmniRoute Circuit Breaker OPEN State: A Complete Guide

> Troubleshoot OmniRoute circuit breaker OPEN state by checking health API failure counts, verifying 5xx errors, and resetting the breaker programmatically or via API. Resolve routing issues fast.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: how-to-guide
- Published: 2026-08-22

---

**When an OmniRoute provider displays `OPEN`, the circuit breaker has short-circuited routing due to accumulated upstream failures; resolve it by checking the health API for failure counts, verifying that only provider-level 5xx errors trigger the breaker, and resetting via `POST /api/resilience/reset` or the programmatic `reset()` method.**

When routing providers in the **diegosouzapw/OmniRoute** repository begin failing, the **Provider Circuit Breaker** automatically isolates them by transitioning to an `OPEN` state. This prevents cascading failures across your routing infrastructure, but requires immediate diagnosis to restore service. This guide explains the internal mechanics of the breaker, why providers get stuck in OPEN, and the exact steps to validate and recover from the state.

## Where the Circuit Breaker Lives in OmniRoute

The circuit breaker implementation resides in **[[`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts)](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.50/src/shared/utils/circuitBreaker.ts)**. This file defines the `CircuitBreaker` class, the state machine transitions, and the public `getCircuitBreaker()` factory function used throughout the routing layer.

Persistence is handled through **[[`src/lib/db/domainState.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/db/domainState.ts)](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.50/src/lib/db/domainState.ts)**, which saves state to the **`domain_circuit_breakers`** database table via `saveCircuitBreakerState()` and `loadCircuitBreakerState()`. The high-level architecture and default thresholds are documented in **[[`docs/architecture/RESILIENCE_GUIDE.md`](https://github.com/diegosouzapw/OmniRoute/blob/main/docs/architecture/RESILIENCE_GUIDE.md)](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.50/docs/architecture/RESILIENCE_GUIDE.md)**.

## Why Providers Enter the OPEN State

The breaker transitions to `OPEN` when specific failure thresholds are breached. Understanding these triggers is essential for root-cause analysis.

### Provider-Level Failure Accumulation

The breaker monitors for upstream 5xx errors (**`408, 500, 502, 503, 504`**) and increments an internal `failureCount`. When this count reaches the configured `failureThreshold` (default varies by provider type), the breaker transitions to `OPEN`.

### Degradation Threshold

If the failure count exceeds `degradationThreshold` (approximately 60% of `failureThreshold`), the provider enters a `DEGRADED` state first. This logs warnings but still allows traffic. Only after crossing the full `failureThreshold` does it open completely.

### Immediate-Open for Critical Failures

Certain `FailureKind` classifications (e.g., `rate_limit`) are configured with `immediateOpen: true`. These skip the `DEGRADED` phase and transition directly to `OPEN`.

### Half-Open Probe Failures

After the `resetTimeout` expires, the breaker enters `HALF_OPEN` and allows probe requests. If these probes fail, the `openCycleCount` increments and the effective timeout increases via exponential backoff, extending the `OPEN` duration.

## Step-by-Step Troubleshooting Guide

Follow this diagnostic sequence when a provider shows `OPEN` in OmniRoute.

### 1. Check Breaker Status via Health API

Query the monitoring endpoint to retrieve current state and metrics:

```http
GET /api/monitoring/health

```

The response contains an array of breaker objects with these critical fields:
- `state` – Current state (`CLOSED`, `DEGRADED`, `HALF_OPEN`, or `OPEN`)
- `failureCount` – Current count toward the threshold
- `lastFailureTime` – Timestamp of the most recent error
- `retryAfterMs` – Time remaining until automatic reset attempt
- `openCycleCount` – Number of consecutive open/close cycles indicating chronic issues

### 2. Inspect Transition History

Use the programmatic API to examine recent state changes:

```typescript
import { getAllCircuitBreakerStatuses } from '@/shared/utils/circuitBreaker';

const statuses = await getAllCircuitBreakerStatuses();
const brokenProvider = statuses.find(s => s.state === 'OPEN');

console.log(brokenProvider.transitionHistory);

```

Look for the most recent transition to `OPEN` and examine the `reason` field to identify the specific `FailureKind` that triggered the change.

### 3. Verify Error Classification Filters

Ensure only provider-level errors increment the failure count. The breaker accepts an `isFailure` option that defaults to `() => true`, but production routes in **[[`open-sse/handlers/chatHelpers.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/handlers/chatHelpers.ts)](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.50/open-sse/handlers/chatHelpers.ts)** and **[[`open-sse/handlers/chat.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/handlers/chat.ts)](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.50/open-sse/handlers/chat.ts)** wrap the breaker with custom logic.

Confirm that your implementation uses **`isLocalStreamLifecycleError()`** (defined in **[[`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts)](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.50/open-sse/services/accountFallback.ts)**) to exclude client-side aborts and "controller is already closed" errors from the failure count. Otherwise, transient client disconnects can falsely trip the breaker.

### 4. Confirm Threshold Configurations

Default thresholds are defined in **[[`open-sse/config/constants.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/config/constants.ts)](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.50/open-sse/config/constants.ts)** and vary by authentication type:
- **OAuth providers**: `failureThreshold: 8`, `resetTimeout: 60s`
- **API-key providers**: `failureThreshold: 12`, `resetTimeout: 30s`

Access these via Dashboard → Settings → Resilience to verify they match your operational requirements.

### 5. Audit Error Classification Logic

Review the **`classifyError()`** function in **[[`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts)](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.50/open-sse/services/accountFallback.ts)**. Misclassification can cause the breaker to ignore critical 5xx errors (failing to open when it should) or treat benign errors as critical (opening prematurely). Check any custom classification logic added for new providers.

### 6. Distinguish From Connection Cooldowns

Individual connections maintain separate `rateLimitedUntil` timestamps in the `provider_connections` table. If the circuit breaker reports `OPEN` but connections show future rate-limit dates, the breaker state may be stale. Check the **Connection Cooldown** layer documented in the Resilience Guide to differentiate between provider-level and connection-level blocks.

### 7. Exclude Model-Level Lockouts

Ensure the failure originated from the provider rather than a specific model. Model-scoped quota errors should trigger the **Model Lockout** mechanism (detailed in the Resilience Guide) rather than the provider circuit breaker. Opening the entire provider for a single model's quota exhaustion reduces routing capacity unnecessarily.

### 8. Manually Reset the Breaker

If the breaker is stuck or you need immediate recovery, use the reset API:

```http
POST /api/resilience/reset

```

Alternatively, programmatically reset via Node REPL or maintenance scripts:

```typescript
import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';

const breaker = getCircuitBreaker('openai');
breaker.reset(); // Clears failureCount, openCycleCount, and DB persistence

```

### 9. Validate Recovery

After resetting, send controlled test requests to the provider. Monitor the health API to confirm the state transitions back to `CLOSED` via the internal `_onSuccess` handler (lines 53-71 in the source). Verify that `failureCount` remains at zero and no new transitions to `OPEN` occur.

## Programmatic Diagnostics and Recovery

### Query All Breaker Statuses

```typescript
import { getAllCircuitBreakerStatuses } from '@/shared/utils/circuitBreaker';

const statuses = await getAllCircuitBreakerStatuses();

statuses.forEach((s) => {
  console.log(
    `${s.name}: ${s.state} (failures=${s.failureCount}, retryIn=${s.retryAfterMs}ms)`
  );
});

```

### Custom Failure Filtering

Prevent client aborts from counting against the breaker:

```typescript
import { getCircuitBreaker, isLocalStreamLifecycleError } from '@/shared/utils/circuitBreaker';

const breaker = getCircuitBreaker('anthropic', {
  isFailure: (err) => {
    // Only count actual provider failures, not local stream interruptions
    return !isLocalStreamLifecycleError(err);
  },
});

```

### Force Reset a Specific Provider

```typescript
import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';

// Immediately clear OPEN state for a specific provider
const breaker = getCircuitBreaker('openai');
breaker.reset();
console.log('Breaker reset to CLOSED');

```

## Summary

- **Location**: The circuit breaker logic lives in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts), with persistence handled in [`src/lib/db/domainState.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/db/domainState.ts).
- **Triggers**: Providers enter `OPEN` after upstream 5xx errors (`408, 500, 502, 503, 504`) exceed `failureThreshold`, or immediately for critical `FailureKind` types like `rate_limit`.
- **Diagnosis**: Check `GET /api/monitoring/health` for `failureCount` and `openCycleCount`; inspect `transitionHistory` via `getAllCircuitBreakerStatuses()`.
- **False Positives**: Verify that `isFailure` filters exclude client aborts using `isLocalStreamLifecycleError()` from [`accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/accountFallback.ts).
- **Recovery**: Reset via `POST /api/resilience/reset` or programmatic `getCircuitBreaker(name).reset()`, then validate success counts reset to zero.

## Frequently Asked Questions

### How can I check which specific error caused the circuit breaker to open?

Query the breaker status using `getAllCircuitBreakerStatuses()` from [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) and examine the `transitionHistory` array. The most recent entry contains a `reason` field indicating the `FailureKind` (e.g., `rate_limit`, `timeout`) that triggered the transition to `OPEN`.

### Why does my provider stay OPEN even after the upstream service recovers?

The breaker enters `HALF_OPEN` after `resetTimeout` expires, but if probe requests fail during this phase, it reopens immediately and increments `openCycleCount`, which extends the timeout via exponential backoff. Use the health API to check `openCycleCount` and manually reset the breaker if the upstream recovery is confirmed but the probes continue failing due to stale state.

### What is the difference between DEGRADED and OPEN states?

`DEGRADED` occurs when `failureCount` exceeds approximately 60% of `failureThreshold` (the degradation threshold); traffic still flows but is logged for monitoring. `OPEN` occurs when `failureThreshold` is fully reached, at which point the routing engine short-circuits all requests to that provider, causing combo strategies to skip it entirely.

### Can client-side request cancellations trigger the circuit breaker?

Only if the `isFailure` callback is misconfigured. The default `() => true` counts all errors, but production implementations in [`open-sse/handlers/chatHelpers.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/handlers/chatHelpers.ts) use `isLocalStreamLifecycleError()` to filter out client aborts and "controller is already closed" errors. Ensure your route handlers wrap the breaker with this filter to prevent false opens from client disconnects.