# How to Debug OmniRoute Circuit Breaker Failures and Understand Provider Lockout

> Debug OmniRoute circuit breaker failures and understand provider lockout. Learn to manage CLOSED, DEGRADED, OPEN, and HALF_OPEN states for robust protection.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: how-to-guide
- Published: 2026-07-05

---

**OmniRoute protects downstream providers using a robust circuit breaker in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) that tracks four distinct states (CLOSED, DEGRADED, OPEN, HALF_OPEN), while a separate lightweight breaker in [`open-sse/services/tokenRefresh.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/tokenRefresh.ts) handles authentication-specific lockouts via `isProviderBlocked()` checks.**

OmniRoute implements a sophisticated resilience pattern to prevent cascading failures when calling external AI providers. The system maintains two distinct circuit breaker implementations: a generic request-level breaker for routing failures and a specialized token-refresh blocker for authentication issues. Both mechanisms persist their state to a SQLite-backed `domainState` table, enabling diagnostics across application restarts.

## Understanding the Circuit Breaker State Machine

The core circuit breaker implementation in **[`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts)** manages provider health through a finite state machine driven by failure counts and per-failure-kind thresholds.

### The Four States of Resilience

Each provider (or named resource) receives its own `CircuitBreaker` instance via the registry function `getCircuitBreaker()`. The breaker transitions between these states based on accumulated failures:

- **CLOSED**: Normal operation where all requests pass through to the provider.
- **DEGRADED**: Failure rate is elevated but below the critical threshold; requests continue while warnings are logged.
- **OPEN**: The provider is fully locked out; all requests are short-circuited immediately to prevent overload.
- **HALF_OPEN**: Recovery probing state where a limited number of test requests are allowed to check if the provider has recovered.

State transitions occur when `failureCount` reaches configured thresholds. When the count hits `failureThreshold`, the breaker moves to **OPEN** (or **DEGRADED** first if only the degradation threshold is met).

### Key Implementation Methods

The breaker logic centers on three critical methods in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts):

- **`execute(fn)`**: Wraps async calls, applying state checks and transition logic before invoking the wrapped function.
- **`_onFailure(kind?)`**: Tallies failures, applies per-kind thresholds from `kindThresholds`, and decides whether to open the circuit based on `immediateOpen` flags.
- **`_refreshOpenState()`**: Monitors the back-off timeout when in **OPEN** state; moves to **HALF_OPEN** once the reset period elapses.

## Token Refresh and Provider Lockout Mechanisms

Authentication failures trigger a separate lightweight circuit breaker located in **[`open-sse/services/tokenRefresh.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/tokenRefresh.ts)**. This mechanism specifically handles token refresh failures through a simple counter-based approach.

When `recordFailure()` detects consecutive failures for a provider, it increments a counter stored in `_circuitBreaker[provider].failures`. Once this exceeds `CIRCUIT_BREAKER_THRESHOLD`, the provider is marked as blocked:

```typescript
// open-sse/services/tokenRefresh.ts
if (_circuitBreaker[provider].failures >= CIRCUIT_BREAKER_THRESHOLD) {
  _circuitBreaker[provider].blockedUntil = Date.now() + CIRCUIT_BREAKER_COOLDOWN;
}

```

The helper **`isProviderBlocked(provider)`** checks this timestamp, returning `true` while the lockout persists:

```typescript
// open-sse/services/tokenRefresh.ts
if (!state) return false;
if (!state.blockedUntil) return false;
if (state.blockedUntil > Date.now()) return true;

```

## Debugging Circuit Breaker Failures Step-by-Step

### Inspect Global Breaker Status

To diagnose which providers are currently affected, use the global registry helper that returns a JSON array containing `state`, `failureCount`, `lastFailureTime`, `retryAfterMs`, and `openCycleCount`:

```typescript
import { getAllCircuitBreakerStatuses } from '@/shared/utils/circuitBreaker';

function dumpStatuses() {
  const list = getAllCircuitBreakerStatuses();
  list.forEach((s) => {
    console.log(
      `${s.name}: ${s.state} – failures:${s.failureCount} – retryAfter:${s.retryAfterMs}ms`
    );
  });
}
dumpStatuses();

```

This queries the SQLite persistence layer and reflects the current in-memory state as implemented in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) (lines 885-898).

### Check Specific Provider Lockout

For authentication-specific blocks, import the token refresh utilities to verify lockout status:

```typescript
import { getCircuitBreakerStatus, isProviderBlocked } from '@/open-sse/services/tokenRefresh';

console.log(getCircuitBreakerStatus()['openai']);
console.log(isProviderBlocked('openai')); // Returns true if blockedUntil > Date.now()

```

### Analyze Transition History

Each `CircuitBreaker` instance records the last 20 state transitions (configurable via `maxTransitionHistory`). Access this history to understand when and why a provider entered OPEN state:

```typescript
import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';

function showHistory(id: string) {
  const breaker = getCircuitBreaker(id);
  console.table(breaker.getStatus().transitionHistory);
}
showHistory('openai');

```

### Force Reset After Recovery

When you confirm an upstream service has recovered, manually reset the breaker to clear failure counts and move to CLOSED:

```typescript
import { resetAllCircuitBreakers } from '@/shared/utils/circuitBreaker';

resetAllCircuitBreakers(); // Clears all breakers and DB rows

```

For a single provider:

```typescript
import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';

function resetProviderBreaker(id: string) {
  const breaker = getCircuitBreaker(id);
  breaker.reset(); // Clears counts, moves to CLOSED
  console.info(`Breaker for ${id} has been reset`);
}

```

### Tune Per-Failure-Kind Thresholds

Configure different thresholds for specific error types using `kindThresholds`. This allows rate-limit errors (429s) to trigger degradation faster than transient 5xx errors:

```typescript
import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';

const breaker = getCircuitBreaker('myProvider', {
  failureThreshold: 10,
  kindThresholds: {
    rate_limit: { threshold: 3, immediateOpen: false },
    transient:  { threshold: 5, immediateOpen: true },
  },
});

```

The `immediateOpen` flag determines whether reaching that specific threshold forces an immediate transition to OPEN state.

### Monitor Back-off Escalation

After repeated OPEN→HALF_OPEN→OPEN cycles, the reset timeout grows exponentially via `_effectiveResetTimeout`. Check the current effective timeout:

```typescript
const breaker = getCircuitBreaker('myProvider');
console.log(breaker.getStatus().effectiveResetTimeout);

```

This value increases based on `maxBackoffMultiplier` and `backoffEscalationCount` settings as implemented in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) (lines 223-232).

## Common Circuit Breaker Pitfalls

| Symptom | Likely Cause | Verification Method |
|---------|--------------|---------------------|
| Provider remains blocked after a single 502 | The token-refresh breaker treats consecutive failures as immediate blocks; if the counter was left at `>= CIRCUIT_BREAKER_THRESHOLD`, the provider stays locked. | Check `recordFailure` counters via `getCircuitBreakerStatus()`. |
| Requests receive 429s despite breaker showing CLOSED | The breaker may be in **DEGRADED** state (elevated failure rate) but not yet OPEN, allowing requests through with warnings. | Inspect `state` field in `getAllCircuitBreakerStatuses()` output. |
| Critical failures never open the circuit | Missing `immediateOpen` flag for the specific failure kind in `kindThresholds`. | Verify `kindThresholds[<kind>].immediateOpen` is set to `true` for critical error types. |
| Back-off period exceeds expected duration | `maxBackoffMultiplier` or `backoffEscalationCount` configured too low. | Check `breaker.maxBackoffMultiplier` via `getStatus()`. |

## Summary

Debugging OmniRoute circuit breaker failures requires understanding both the generic request-level breaker and the authentication-specific token refresh blocker:

- **Primary breaker** lives in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) and manages four states (CLOSED, DEGRADED, OPEN, HALF_OPEN) with per-kind thresholds.
- **Token refresh blocker** resides in [`open-sse/services/tokenRefresh.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/tokenRefresh.ts) and locks providers using `CIRCUIT_BREAKER_THRESHOLD` and `CIRCUIT_BREAKER_COOLDOWN`.
- **Diagnostics** are available via `getAllCircuitBreakerStatuses()`, `getCircuitBreakerStatus()`, and `transitionHistory` inspection.
- **Recovery** can be accelerated using `resetAllCircuitBreakers()` or individual `breaker.reset()` calls.
- **Persistence** to SQLite via the `domainState` table ensures diagnostics survive restarts.

## Frequently Asked Questions

### How do I check if a specific provider is currently locked out?

Import `isProviderBlocked` from `@/open-sse/services/tokenRefresh` and pass the provider ID. This function checks the `_circuitBreaker[provider].blockedUntil` timestamp against `Date.now()`. For request-level circuit breakers, use `getCircuitBreaker(provider).getStatus().state` to see if it is OPEN or HALF_OPEN.

### What is the difference between the DEGRADED and OPEN states?

**DEGRADED** allows requests to continue passing through while logging warnings and tracking elevated failure rates; it serves as an early warning before full lockout. **OPEN** immediately short-circuits all requests to the provider, returning errors without attempting the external call. The transition from DEGRADED to OPEN occurs when `failureCount` reaches the configured `failureThreshold`.

### Why does my provider stay blocked after only one HTTP 502 error?

The token refresh circuit breaker in [`open-sse/services/tokenRefresh.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/tokenRefresh.ts) uses a simple counter (`_circuitBreaker[provider].failures`) that persists across operations. If the counter was previously elevated and reached `CIRCUIT_BREAKER_THRESHOLD`, the provider remains blocked until `blockedUntil` exceeds the current time. Check the failure count using `getCircuitBreakerStatus()` and reset if necessary.

### How do I permanently reset a circuit breaker after fixing an upstream issue?

For a single provider, retrieve the breaker instance via `getCircuitBreaker('providerId')` and call `.reset()`. To clear all breakers simultaneously, import `resetAllCircuitBreakers` from `@/shared/utils/circuitBreaker`. Both operations clear the in-memory counters and update the SQLite persistence layer in the `domainState` table.