# How to Debug OmniRoute Circuit Breaker Issues: A Complete Troubleshooting Guide

> Debug OmniRoute circuit breaker issues effectively. Use the health dashboard API endpoints and structured logs to identify failures and reset provider states.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: how-to-guide
- Published: 2026-08-14

---

**Use the health dashboard, API endpoints, and structured logs in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) to identify open breakers, inspect failure codes, and reset provider states when upstream LLM services recover.**

OmniRoute's **provider circuit breaker** sits at the top of its three-layer resilience system, blocking traffic to entire LLM providers when repeated upstream failures occur. Understanding how this state machine operates—where it lives, how it transitions, and where it guards the request pipeline—lets you quickly diagnose why providers appear offline and restore normal routing.

## Circuit Breaker Architecture in OmniRoute

The circuit breaker implementation spans five key components across the codebase:

| Component | Role | Implementation File |
|-----------|------|---------------------|
| **CircuitBreaker class** | Core state machine tracking failures, timestamps, and states (`CLOSED`, `OPEN`, `HALF_OPEN`) | [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) |
| **Provider-level registry** | Isolates one misbehaving provider from others via per-provider instances | [`src/lib/warmupScheduler/circuitBreakerFactory.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/warmupScheduler/circuitBreakerFactory.ts) |
| **AccountFallback logic** | Guards routing by checking `breaker.canExecute()` before request dispatch | [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts) |
| **Chat-handler guard** | Prevents SSE streams from inadvertently tripping the breaker | [`src/sse/handlers/chatHelpers.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/handlers/chatHelpers.ts) |
| **Health & reset API** | Exposes breaker status and allows manual recovery | `/api/resilience/reset` |

The `CircuitBreaker` class in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) implements a standard state machine with three states:

- **`CLOSED`**: Normal operation, requests pass through
- **`OPEN`**: Blocking all traffic, storing an expiration timestamp in `resetAfterMs`
- **`HALF_OPEN`**: Single probe request allowed to test recovery

Each provider receives its own isolated breaker instance through the factory in [`src/lib/warmupScheduler/circuitBreakerFactory.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/warmupScheduler/circuitBreakerFactory.ts), ensuring one failing LLM service doesn't cascade to others.

## How Provider Circuit Breaker Failures Flow

Understanding the lifecycle helps you identify where to intervene:

1. **Upstream error detection**: When a provider returns failure codes (`408`, `500`, `502`, `503`, `504`), `CircuitBreaker.recordFailure()` increments the failure counter.

2. **State transition**: After the configured threshold (default **3 for OAuth providers**, **5 for API-key providers**), the breaker moves `CLOSED` → `OPEN` and records `resetAfterMs`.

3. **Lazy recovery**: The next `breaker.canExecute()` call checks the timeout. If elapsed, it flips to `HALF_OPEN` and allows one probe request.

4. **Resolution**: Success closes the breaker; failure resets the timeout and returns to `OPEN`.

**Critical distinction**: Only **provider-level** failures trigger this flow. Connection-level rate-limits (`429`) and model-specific quota errors are handled by the *connection cooldown* and *model lockout* layers—**not** by the provider breaker. This separation explains why a provider might appear "offline" while individual connections still work.

## Step-by-Step Debugging Process

### 1. Check the Health Dashboard

The UI (`/dashboard/health`) and CLI provide immediate visibility into breaker states:

```bash
omniroute health components

```

Look for the **circuit breaker** card showing each provider's current state (`CLOSED`, `HALF_OPEN`, `OPEN`).

### 2. Query the Health API Directly

For programmatic access to raw metrics:

```http
GET /api/monitoring/health
Accept: application/json

```

The response contains a `providers` map with `circuitBreaker` fields:

```json
{
  "providers": {
    "openai": {
      "circuitBreaker": {
        "state": "OPEN",
        "consecutiveFailures": 5,
        "nextRetryMs": 30000
      }
    }
  }
}

```

### 3. Inspect Structured Logs

[`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) logs every transition via `pino`. Search logs for `"circuitBreaker"` to find:

- When the breaker opened
- Which error code caused the transition
- The `resetAfterMs` value set

### 4. Validate Failure Classification

Ensure the triggering error is truly a *provider* failure. The classification predicate lives in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) as `PROVIDER_FAILURE_ERROR_CODES`.

**Known issue**: If `429` appears in this list, the breaker trips on normal rate-limit responses. This regression was fixed in [#1767](https://github.com/diegosouzapw/OmniRoute/pull/1767).

### 5. Manually Reset When Safe

If the provider has recovered, bypass the timeout:

```http
POST /api/resilience/reset
Content-Type: application/json

{ "provider": "openai" }

```

This calls `resetProviderCircuitBreaker(provider)`, clearing `consecutiveFailures` and forcing state to `CLOSED`.

### 6. Reproduce in Isolation

Test the breaker logic directly using the exported `CircuitBreaker` class:

```ts
import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';

async function logBreakerState(provider: string) {
  const breaker = getCircuitBreaker(provider);
  console.log(`[${provider}] state = ${breaker.getStatus()}`);
  console.log(`failures = ${breaker.consecutiveFailures}`);
  console.log(`next retry in ${breaker.getRetryAfterMs()} ms`);
}

await logBreakerState('openai');

```

The `getStatus()` and `getRetryAfterMs()` helpers expose internal state for debugging.

## Common Circuit Breaker Issues and Fixes

| Symptom | Root Cause | Solution |
|---------|-----------|----------|
| Provider stays **OPEN** despite healthy upstream | Clock skew or overridden `resetAfterMs` | Verify system clock; use reset API |
| All models blocked, other providers work | Provider breaker opened on valid failure | Check error code isn't `429`; reference [#1767](https://github.com/diegosouzapw/OmniRoute/pull/1767) |
| Single account blocks entire provider | Account cooldown propagating incorrectly | Review [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts) guard order |
| Never recovers after transient outage | `HALF_OPEN` probe keeps failing | Increase reset timeout via environment variables |
| Frequent trips for stable provider | Overly aggressive threshold | Tune in [`src/shared/constants/resilience.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/constants/resilience.ts) |

## Code Examples for Debugging OmniRoute Circuit Breakers

### CLI Script to Reset a Provider

```ts
import fetch from 'node-fetch';

async function resetBreaker(provider: string) {
  const res = await fetch('http://localhost:20128/api/resilience/reset', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ provider })
  });
  const json = await res.json();
  console.log('Reset response:', json);
}

resetBreaker('anthropic');

```

### Bash One-Liner for Health Checks

```bash
curl -s http://localhost:20128/api/monitoring/health \
  | jq '.providers.openai.circuitBreaker'

```

Sample output:

```json
{
  "state": "OPEN",
  "consecutiveFailures": 5,
  "nextRetryMs": 30000
}

```

### Unit Test Pattern for Recovery Validation

```ts
import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';
import assert from 'assert';

describe('Provider breaker half-open recovery', () => {
  const provider = 'gemini';
  const breaker = getCircuitBreaker(provider);

  it('opens after threshold', () => {
    for (let i = 0; i < 5; i++) breaker.recordFailure(new Error('500'));
    assert.equal(breaker.getStatus(), 'OPEN');
  });

  it('recovers after timeout', async () => {
    breaker.forceOpenUntil(Date.now() - 1);
    assert.equal(breaker.canExecute(), true);
    assert.equal(breaker.getStatus(), 'CLOSED');
  });
});

```

Reference: [`tests/unit/provider-breaker-halfopen-recovery.test.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/tests/unit/provider-breaker-halfopen-recovery.test.ts)

## Key Source Files for Circuit Breaker Debugging

| File | Purpose |
|------|---------|
| [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) | Core `CircuitBreaker` class with state machine and public helpers |
| [`src/lib/warmupScheduler/circuitBreakerFactory.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/warmupScheduler/circuitBreakerFactory.ts) | Factory for per-provider breaker instances |
| [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts) | Routing guard calling `breaker.canExecute()` |
| [`src/sse/handlers/chatHelpers.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/handlers/chatHelpers.ts) | SSE stream guard respecting breaker state |
| [`docs/architecture/RESILIENCE_GUIDE.md`](https://github.com/diegosouzapw/OmniRoute/blob/main/docs/architecture/RESILIENCE_GUIDE.md) | Three-layer resilience design overview |
| [`docs/reference/API_REFERENCE.md`](https://github.com/diegosouzapw/OmniRoute/blob/main/docs/reference/API_REFERENCE.md) | Health and reset endpoint documentation |
| [`src/shared/constants/resilience.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/constants/resilience.ts) | Default thresholds (OAuth: 3, API-key: 5) |

## Summary

- OmniRoute's **provider circuit breaker** in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) uses a three-state machine (`CLOSED` → `OPEN` → `HALF_OPEN`) to protect against upstream LLM failures
- **Debug via**: health dashboard (`/dashboard/health`), CLI (`omniroute health`), API (`/api/monitoring/health`), and structured logs
- **Reset manually** through `/api/resilience/reset` when providers recover before timeout
- **Distinguish provider failures** (`408`, `500`, `502`, `503`, `504`) from connection rate-limits (`429`) to avoid misconfiguration
- **Tune thresholds** in [`src/shared/constants/resilience.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/constants/resilience.ts) if defaults prove too aggressive for your providers

## Frequently Asked Questions

### Why is my provider stuck in OPEN state even though the service is healthy?

The breaker may have a stale `resetAfterMs` timestamp due to clock skew, or the timeout was manually extended. Check system time synchronization, then use the reset API to force `CLOSED` state. Inspect logs for the original error code that triggered the open—if it was `429`, see [#1767](https://github.com/diegosouzapw/OmniRoute/pull/1767) for the fix.

### How do I distinguish a provider circuit breaker from connection cooldown?

Provider breakers live in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) and track `consecutiveFailures` with `recordFailure()`. Connection cooldown is a separate layer managing individual account rate-limits. The health API shows both: `circuitBreaker.state` for the provider, `connectionStatus.cooldownUntil` for accounts. Provider breakers only react to `PROVIDER_FAILURE_ERROR_CODES`, not `429` responses.

### Can I disable the circuit breaker for a specific provider?

No direct disable exists, but you can set an effectively infinite threshold in [`src/shared/constants/resilience.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/constants/resilience.ts) or environment configuration. The breaker still initializes but won't open. For production, prefer tuning `resetAfterMs` rather than disabling—this preserves the safety net while reducing recovery time.

### Where does the breaker guard actually block requests?

Two critical checkpoints enforce `breaker.canExecute()`: [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts) for standard requests and [`src/sse/handlers/chatHelpers.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/handlers/chatHelpers.ts) for streaming responses. Both call `getCircuitBreaker(provider)` from the factory and reject routing when `OPEN`. If requests bypass these guards, they won't respect breaker state.