How to Troubleshoot OmniRoute Circuit Breaker OPEN State: A Complete Guide

When an OmniRoute provider displays OPEN, the circuit breaker has short-circuited routing due to accumulated upstream failures; resolve it by checking the health API for failure counts, verifying that only provider-level 5xx errors trigger the breaker, and resetting via POST /api/resilience/reset or the programmatic reset() method.

When routing providers in the diegosouzapw/OmniRoute repository begin failing, the Provider Circuit Breaker automatically isolates them by transitioning to an OPEN state. This prevents cascading failures across your routing infrastructure, but requires immediate diagnosis to restore service. This guide explains the internal mechanics of the breaker, why providers get stuck in OPEN, and the exact steps to validate and recover from the state.

Where the Circuit Breaker Lives in OmniRoute

The circuit breaker implementation resides in [src/shared/utils/circuitBreaker.ts](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.50/src/shared/utils/circuitBreaker.ts). This file defines the CircuitBreaker class, the state machine transitions, and the public getCircuitBreaker() factory function used throughout the routing layer.

Persistence is handled through [src/lib/db/domainState.ts](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.50/src/lib/db/domainState.ts), which saves state to the domain_circuit_breakers database table via saveCircuitBreakerState() and loadCircuitBreakerState(). The high-level architecture and default thresholds are documented in [docs/architecture/RESILIENCE_GUIDE.md](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.50/docs/architecture/RESILIENCE_GUIDE.md).

Why Providers Enter the OPEN State

The breaker transitions to OPEN when specific failure thresholds are breached. Understanding these triggers is essential for root-cause analysis.

Provider-Level Failure Accumulation

The breaker monitors for upstream 5xx errors (408, 500, 502, 503, 504) and increments an internal failureCount. When this count reaches the configured failureThreshold (default varies by provider type), the breaker transitions to OPEN.

Degradation Threshold

If the failure count exceeds degradationThreshold (approximately 60% of failureThreshold), the provider enters a DEGRADED state first. This logs warnings but still allows traffic. Only after crossing the full failureThreshold does it open completely.

Immediate-Open for Critical Failures

Certain FailureKind classifications (e.g., rate_limit) are configured with immediateOpen: true. These skip the DEGRADED phase and transition directly to OPEN.

Half-Open Probe Failures

After the resetTimeout expires, the breaker enters HALF_OPEN and allows probe requests. If these probes fail, the openCycleCount increments and the effective timeout increases via exponential backoff, extending the OPEN duration.

Step-by-Step Troubleshooting Guide

Follow this diagnostic sequence when a provider shows OPEN in OmniRoute.

1. Check Breaker Status via Health API

Query the monitoring endpoint to retrieve current state and metrics:

GET /api/monitoring/health

The response contains an array of breaker objects with these critical fields:

• state – Current state (CLOSED, DEGRADED, HALF_OPEN, or OPEN)
• failureCount – Current count toward the threshold
• lastFailureTime – Timestamp of the most recent error
• retryAfterMs – Time remaining until automatic reset attempt
• openCycleCount – Number of consecutive open/close cycles indicating chronic issues

2. Inspect Transition History

Use the programmatic API to examine recent state changes:

import { getAllCircuitBreakerStatuses } from '@/shared/utils/circuitBreaker';

const statuses = await getAllCircuitBreakerStatuses();
const brokenProvider = statuses.find(s => s.state === 'OPEN');

console.log(brokenProvider.transitionHistory);

Look for the most recent transition to OPEN and examine the reason field to identify the specific FailureKind that triggered the change.

3. Verify Error Classification Filters

Ensure only provider-level errors increment the failure count. The breaker accepts an isFailure option that defaults to () => true, but production routes in [open-sse/handlers/chatHelpers.ts](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.50/open-sse/handlers/chatHelpers.ts) and [open-sse/handlers/chat.ts](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.50/open-sse/handlers/chat.ts) wrap the breaker with custom logic.

Confirm that your implementation uses isLocalStreamLifecycleError() (defined in [open-sse/services/accountFallback.ts](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.50/open-sse/services/accountFallback.ts)) to exclude client-side aborts and "controller is already closed" errors from the failure count. Otherwise, transient client disconnects can falsely trip the breaker.

4. Confirm Threshold Configurations

Default thresholds are defined in [open-sse/config/constants.ts](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.50/open-sse/config/constants.ts) and vary by authentication type:

• OAuth providers: failureThreshold: 8, resetTimeout: 60s
• API-key providers: failureThreshold: 12, resetTimeout: 30s

Access these via Dashboard → Settings → Resilience to verify they match your operational requirements.

5. Audit Error Classification Logic

Review the classifyError() function in [open-sse/services/accountFallback.ts](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.50/open-sse/services/accountFallback.ts). Misclassification can cause the breaker to ignore critical 5xx errors (failing to open when it should) or treat benign errors as critical (opening prematurely). Check any custom classification logic added for new providers.

6. Distinguish From Connection Cooldowns

Individual connections maintain separate rateLimitedUntil timestamps in the provider_connections table. If the circuit breaker reports OPEN but connections show future rate-limit dates, the breaker state may be stale. Check the Connection Cooldown layer documented in the Resilience Guide to differentiate between provider-level and connection-level blocks.

7. Exclude Model-Level Lockouts

Ensure the failure originated from the provider rather than a specific model. Model-scoped quota errors should trigger the Model Lockout mechanism (detailed in the Resilience Guide) rather than the provider circuit breaker. Opening the entire provider for a single model's quota exhaustion reduces routing capacity unnecessarily.

8. Manually Reset the Breaker

If the breaker is stuck or you need immediate recovery, use the reset API:

POST /api/resilience/reset

Alternatively, programmatically reset via Node REPL or maintenance scripts:

import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';

const breaker = getCircuitBreaker('openai');
breaker.reset(); // Clears failureCount, openCycleCount, and DB persistence

9. Validate Recovery

After resetting, send controlled test requests to the provider. Monitor the health API to confirm the state transitions back to CLOSED via the internal _onSuccess handler (lines 53-71 in the source). Verify that failureCount remains at zero and no new transitions to OPEN occur.

Programmatic Diagnostics and Recovery

Query All Breaker Statuses

import { getAllCircuitBreakerStatuses } from '@/shared/utils/circuitBreaker';

const statuses = await getAllCircuitBreakerStatuses();

statuses.forEach((s) => {
  console.log(
    `${s.name}: ${s.state} (failures=${s.failureCount}, retryIn=${s.retryAfterMs}ms)`
  );
});

Custom Failure Filtering

Prevent client aborts from counting against the breaker:

import { getCircuitBreaker, isLocalStreamLifecycleError } from '@/shared/utils/circuitBreaker';

const breaker = getCircuitBreaker('anthropic', {
  isFailure: (err) => {
    // Only count actual provider failures, not local stream interruptions
    return !isLocalStreamLifecycleError(err);
  },
});

Force Reset a Specific Provider

import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';

// Immediately clear OPEN state for a specific provider
const breaker = getCircuitBreaker('openai');
breaker.reset();
console.log('Breaker reset to CLOSED');

Summary

• Location: The circuit breaker logic lives in src/shared/utils/circuitBreaker.ts, with persistence handled in src/lib/db/domainState.ts.
• Triggers: Providers enter OPEN after upstream 5xx errors (408, 500, 502, 503, 504) exceed failureThreshold, or immediately for critical FailureKind types like rate_limit.
• Diagnosis: Check GET /api/monitoring/health for failureCount and openCycleCount; inspect transitionHistory via getAllCircuitBreakerStatuses().
• False Positives: Verify that isFailure filters exclude client aborts using isLocalStreamLifecycleError() from accountFallback.ts.
• Recovery: Reset via POST /api/resilience/reset or programmatic getCircuitBreaker(name).reset(), then validate success counts reset to zero.

Frequently Asked Questions

How can I check which specific error caused the circuit breaker to open?

Query the breaker status using getAllCircuitBreakerStatuses() from src/shared/utils/circuitBreaker.ts and examine the transitionHistory array. The most recent entry contains a reason field indicating the FailureKind (e.g., rate_limit, timeout) that triggered the transition to OPEN.

Why does my provider stay OPEN even after the upstream service recovers?

The breaker enters HALF_OPEN after resetTimeout expires, but if probe requests fail during this phase, it reopens immediately and increments openCycleCount, which extends the timeout via exponential backoff. Use the health API to check openCycleCount and manually reset the breaker if the upstream recovery is confirmed but the probes continue failing due to stale state.

What is the difference between DEGRADED and OPEN states?

DEGRADED occurs when failureCount exceeds approximately 60% of failureThreshold (the degradation threshold); traffic still flows but is logged for monitoring. OPEN occurs when failureThreshold is fully reached, at which point the routing engine short-circuits all requests to that provider, causing combo strategies to skip it entirely.

Can client-side request cancellations trigger the circuit breaker?

Only if the isFailure callback is misconfigured. The default () => true counts all errors, but production implementations in open-sse/handlers/chatHelpers.ts use isLocalStreamLifecycleError() to filter out client aborts and "controller is already closed" errors. Ensure your route handlers wrap the breaker with this filter to prevent false opens from client disconnects.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}