# How to Debug Provider Connection Failures and Circuit Breaker Behavior in OmniRoute

> Debug OmniRoute provider connection failures and circuit breaker behavior. Learn how OmniRoute short-circuits connections and uses failure classification for cooldown periods. Monitor with getStatus() and transition history.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: debugging-guide
- Published: 2026-07-26

---

**OmniRoute's circuit breaker automatically short-circuits failing provider connections and uses failure classification to trigger specific cooldown periods, with full observability available via `getStatus()` and transition history.**

When upstream providers in OmniRoute start failing, the built-in circuit breaker prevents cascade failures by rejecting requests until the provider recovers. Learning how to debug provider connection failures and circuit breaker behavior requires understanding the failure classification system, the four-state machine, and the diagnostic tools exposed by the `CircuitBreaker` class.

## Understanding the Circuit Breaker Architecture

OmniRoute implements provider resilience through three interconnected components that classify errors, configure thresholds, and manage state transitions.

### Failure Classification in classify429.ts

All HTTP 429 responses are analyzed by the `classify429` function at `src/shared/utils/classify429.ts#L123`. This examines response headers and provider-specific hints to return a `FailureKind`—categorizing errors as rate limits, quota exhaustion, or transient issues.

### Provider Configuration Hints

Per-provider thresholds live in [`src/shared/utils/providerHints.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/providerHints.ts). This module exports the `failureThreshold`, `resetTimeout`, and `cooldownByKind` values that determine how aggressively the breaker trips for each upstream provider.

### Core Implementation in circuitBreaker.ts

The generic `CircuitBreaker` class at `src/shared/utils/circuitBreaker.ts#L124` manages the state machine, tracks failure counts by kind, and implements exponential backoff for reset timeouts.

## How Requests Flow Through the Circuit Breaker

Every provider request in OmniRoute wraps execution through the circuit breaker. In `src/sse/handlers/chat.ts#L96`, the handler instantiates a breaker for the specific provider and executes the request:

```typescript
// src/sse/handlers/chat.ts (excerpt)
const breaker = getCircuitBreaker(providerName, {
  failureThreshold: profile.circuitBreakerThreshold,
  resetTimeout: profile.circuitBreakerReset,
  classifyError: classify429FromError,
});
await breaker.execute(() => executor.execute(request));

```

The `execute` method at `src/shared/utils/circuitBreaker.ts#L52-L63` handles the protection logic:

```typescript
// src/shared/utils/circuitBreaker.ts#L52-L63
async execute<T>(fn: () => Promise<T>): Promise<T> {
  this._refreshOpenState();
  if (this.state === STATE.OPEN) { 
    throw new CircuitBreakerOpenError(...); 
  }
  
  try {
    const result = await fn();
    this._onSuccess();
    return result;
  } catch (error) {
    if (this.isFailure(error)) {
      const kind = this.classifyError?.(error);
      this._onFailure(kind);
    }
    throw error;
  }
}

```

## Circuit Breaker States and Transitions

The breaker maintains four distinct states that control request flow:

- **CLOSED**: Normal operation; requests proceed to the provider.
- **DEGRADED**: Failure rate exceeds the degradation threshold (default 60% of `failureThreshold`). Requests still flow but warnings are emitted.
- **OPEN**: Requests are short-circuited immediately with a `CircuitBreakerOpenError`.
- **HALF_OPEN**: After the cooldown expires, a limited number of probe requests (`halfOpenRequests`) test provider recovery.

State transitions are recorded in `transitionHistory`. The `_effectiveResetTimeout` method at `src/shared/utils/circuitBreaker.ts#L41-L50` implements exponential backoff for repeated failures:

```typescript
_effectiveResetTimeout(): number {
  if (this.openCycleCount <= this.backoffEscalationCount) return this.resetTimeout;
  const escalationFactor = Math.pow(2, this.openCycleCount - this.backoffEscalationCount);
  return Math.min(this.resetTimeout * escalationFactor, this.resetTimeout * this.maxBackoffMultiplier);
}

```

### Per-Failure-Kind Handling

Different error types can have distinct thresholds via `kindThresholds`. Quota-exhausted errors often use longer cooldowns than generic rate limits. The interface at lines 75-85 defines:

```typescript
interface FailureKindThresholds {
  threshold: number;
  cooldown?: number;
  immediateOpen?: boolean;
}

```

## Debugging Provider Connection Failures

Use these specific steps to diagnose why a provider is being rejected.

### Inspect the Breaker Status

Call `getStatus()` to retrieve current state, failure counts, and retry timing. The implementation at `src/shared/utils/circuitBreaker.ts#L103` returns a snapshot including `state`, `failureCount`, and `retryAfterMs`:

```typescript
const status = getCircuitBreaker('openai').getStatus();
console.log(status);
// { name: 'openai', state: 'OPEN', failureCount: 7, retryAfterMs: 12345, ... }

```

### Verify Failure Classification

Ensure errors are categorized correctly by checking the output of `classify429FromError`. If classifications seem incorrect, review the provider-specific patterns in [`src/shared/utils/classify429.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/classify429.ts).

### Review Transition History

The breaker preserves up to `maxTransitionHistory` (default 20) state changes. Access this array to trace exactly when and why a circuit opened:

```typescript
const cb = getCircuitBreaker('anthropic');
console.log(cb.transitionHistory);

```

### Reset Circuits and Tune Thresholds

Force an immediate reset after resolving upstream issues:

```typescript
import { resetAllCircuitBreakers } from '@/shared/utils/circuitBreaker';
resetAllCircuitBreakers(); // Clears every registered breaker

```

Customize behavior per provider in [`src/shared/utils/providerHints.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/providerHints.ts) by adjusting `circuitBreakerThreshold` (default 5) or `circuitBreakerReset` (default 30s).

## Common Pitfalls and Solutions

| Symptom | Likely Cause | Verification |
|---------|--------------|--------------|
| Circuit opens immediately after single request | `immediateOpen` flag set for the failure kind | Inspect `kindThresholds` in status for `immediateOpen: true` |
| Breaker never closes | Cooldown too long or failure kind never cleared | Verify `retryAfterMs` or manually call `reset()` |
| Transient glitches trigger circuit | `isFailure` defaults to `() => true` | Use `isLocalStreamLifecycleError` to filter client-side aborts |
| Stuck in DEGRADED state | Failure count between degradation and open thresholds | Compare `failureCount` vs `degradationThreshold` in status |

## Summary

- OmniRoute wraps provider calls through `getCircuitBreaker(...).execute()` in [`src/sse/handlers/chat.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/handlers/chat.ts).
- The breaker classifies failures using `classify429FromError` and tracks four states: CLOSED, DEGRADED, OPEN, and HALF_OPEN.
- Debug using `getStatus()` for current state, `transitionHistory` for change logs, and `resetAllCircuitBreakers()` to clear stuck states.
- Configure per-provider behavior via [`src/shared/utils/providerHints.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/providerHints.ts) using `failureThreshold`, `resetTimeout`, and `cooldownByKind`.
- Avoid false positives by customizing `isFailure` to ignore local stream lifecycle errors.

## Frequently Asked Questions

### How do I check if a specific provider's circuit breaker is currently open?

Call `getCircuitBreaker('providerName').getStatus()` and inspect the `state` property. If it returns `'OPEN'`, the circuit is actively short-circuiting requests. The returned object also includes `retryAfterMs`, indicating when the next probe attempt will be permitted according to the exponential backoff calculation.

### Why does my circuit breaker immediately open after just one failure?

Check the `kindThresholds` configuration for that provider. Certain failure kinds—such as quota exhaustion—may have `immediateOpen: true` set, which bypasses the DEGRADED state and opens the circuit immediately upon detection. Review [`src/shared/utils/classify429.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/classify429.ts) to see how specific HTTP 429 responses map to these failure kinds.

### Can I prevent local network errors from triggering the circuit breaker?

Yes. Pass a custom `isFailure` predicate to `getCircuitBreaker()` that filters out local stream lifecycle errors. Use the `isLocalStreamLifecycleError` utility (lines 46-65) to identify client-side aborts and return `false` for those errors, preventing them from incrementing the failure count while still catching legitimate upstream failures.

### How do I reset the circuit breaker after fixing an upstream provider issue?

Import `resetAllCircuitBreakers` from [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) and invoke it to clear all registered breakers globally. For a single provider, obtain the specific instance via `getCircuitBreaker(name)` and call `.reset()` on it. This immediately transitions the state to CLOSED, clears all failure counters, and resets the `openCycleCount` to zero.