How to Debug Provider Connection Failures and Circuit Breaker Behavior in OmniRoute

OmniRoute's circuit breaker automatically short-circuits failing provider connections and uses failure classification to trigger specific cooldown periods, with full observability available via getStatus() and transition history.

When upstream providers in OmniRoute start failing, the built-in circuit breaker prevents cascade failures by rejecting requests until the provider recovers. Learning how to debug provider connection failures and circuit breaker behavior requires understanding the failure classification system, the four-state machine, and the diagnostic tools exposed by the CircuitBreaker class.

Understanding the Circuit Breaker Architecture

OmniRoute implements provider resilience through three interconnected components that classify errors, configure thresholds, and manage state transitions.

Failure Classification in classify429.ts

All HTTP 429 responses are analyzed by the classify429 function at src/shared/utils/classify429.ts#L123. This examines response headers and provider-specific hints to return a FailureKind—categorizing errors as rate limits, quota exhaustion, or transient issues.

Provider Configuration Hints

Per-provider thresholds live in src/shared/utils/providerHints.ts. This module exports the failureThreshold, resetTimeout, and cooldownByKind values that determine how aggressively the breaker trips for each upstream provider.

Core Implementation in circuitBreaker.ts

The generic CircuitBreaker class at src/shared/utils/circuitBreaker.ts#L124 manages the state machine, tracks failure counts by kind, and implements exponential backoff for reset timeouts.

How Requests Flow Through the Circuit Breaker

Every provider request in OmniRoute wraps execution through the circuit breaker. In src/sse/handlers/chat.ts#L96, the handler instantiates a breaker for the specific provider and executes the request:

// src/sse/handlers/chat.ts (excerpt)
const breaker = getCircuitBreaker(providerName, {
  failureThreshold: profile.circuitBreakerThreshold,
  resetTimeout: profile.circuitBreakerReset,
  classifyError: classify429FromError,
});
await breaker.execute(() => executor.execute(request));

The execute method at src/shared/utils/circuitBreaker.ts#L52-L63 handles the protection logic:

// src/shared/utils/circuitBreaker.ts#L52-L63
async execute<T>(fn: () => Promise<T>): Promise<T> {
  this._refreshOpenState();
  if (this.state === STATE.OPEN) { 
    throw new CircuitBreakerOpenError(...); 
  }
  
  try {
    const result = await fn();
    this._onSuccess();
    return result;
  } catch (error) {
    if (this.isFailure(error)) {
      const kind = this.classifyError?.(error);
      this._onFailure(kind);
    }
    throw error;
  }
}

Circuit Breaker States and Transitions

The breaker maintains four distinct states that control request flow:

• CLOSED: Normal operation; requests proceed to the provider.
• DEGRADED: Failure rate exceeds the degradation threshold (default 60% of failureThreshold). Requests still flow but warnings are emitted.
• OPEN: Requests are short-circuited immediately with a CircuitBreakerOpenError.
• HALF_OPEN: After the cooldown expires, a limited number of probe requests (halfOpenRequests) test provider recovery.

State transitions are recorded in transitionHistory. The _effectiveResetTimeout method at src/shared/utils/circuitBreaker.ts#L41-L50 implements exponential backoff for repeated failures:

_effectiveResetTimeout(): number {
  if (this.openCycleCount <= this.backoffEscalationCount) return this.resetTimeout;
  const escalationFactor = Math.pow(2, this.openCycleCount - this.backoffEscalationCount);
  return Math.min(this.resetTimeout * escalationFactor, this.resetTimeout * this.maxBackoffMultiplier);
}

Per-Failure-Kind Handling

Different error types can have distinct thresholds via kindThresholds. Quota-exhausted errors often use longer cooldowns than generic rate limits. The interface at lines 75-85 defines:

interface FailureKindThresholds {
  threshold: number;
  cooldown?: number;
  immediateOpen?: boolean;
}

Debugging Provider Connection Failures

Use these specific steps to diagnose why a provider is being rejected.

Inspect the Breaker Status

Call getStatus() to retrieve current state, failure counts, and retry timing. The implementation at src/shared/utils/circuitBreaker.ts#L103 returns a snapshot including state, failureCount, and retryAfterMs:

const status = getCircuitBreaker('openai').getStatus();
console.log(status);
// { name: 'openai', state: 'OPEN', failureCount: 7, retryAfterMs: 12345, ... }

Verify Failure Classification

Ensure errors are categorized correctly by checking the output of classify429FromError. If classifications seem incorrect, review the provider-specific patterns in src/shared/utils/classify429.ts.

Review Transition History

The breaker preserves up to maxTransitionHistory (default 20) state changes. Access this array to trace exactly when and why a circuit opened:

const cb = getCircuitBreaker('anthropic');
console.log(cb.transitionHistory);

Reset Circuits and Tune Thresholds

Force an immediate reset after resolving upstream issues:

import { resetAllCircuitBreakers } from '@/shared/utils/circuitBreaker';
resetAllCircuitBreakers(); // Clears every registered breaker

Customize behavior per provider in src/shared/utils/providerHints.ts by adjusting circuitBreakerThreshold (default 5) or circuitBreakerReset (default 30s).

Common Pitfalls and Solutions

Symptom	Likely Cause	Verification
Circuit opens immediately after single request	immediateOpen flag set for the failure kind	Inspect kindThresholds in status for immediateOpen: true
Breaker never closes	Cooldown too long or failure kind never cleared	Verify retryAfterMs or manually call reset()
Transient glitches trigger circuit	isFailure defaults to () => true	Use isLocalStreamLifecycleError to filter client-side aborts
Stuck in DEGRADED state	Failure count between degradation and open thresholds	Compare failureCount vs degradationThreshold in status

Summary

• OmniRoute wraps provider calls through getCircuitBreaker(...).execute() in src/sse/handlers/chat.ts.
• The breaker classifies failures using classify429FromError and tracks four states: CLOSED, DEGRADED, OPEN, and HALF_OPEN.
• Debug using getStatus() for current state, transitionHistory for change logs, and resetAllCircuitBreakers() to clear stuck states.
• Configure per-provider behavior via src/shared/utils/providerHints.ts using failureThreshold, resetTimeout, and cooldownByKind.
• Avoid false positives by customizing isFailure to ignore local stream lifecycle errors.

Frequently Asked Questions

How do I check if a specific provider's circuit breaker is currently open?

Call getCircuitBreaker('providerName').getStatus() and inspect the state property. If it returns 'OPEN', the circuit is actively short-circuiting requests. The returned object also includes retryAfterMs, indicating when the next probe attempt will be permitted according to the exponential backoff calculation.

Why does my circuit breaker immediately open after just one failure?

Check the kindThresholds configuration for that provider. Certain failure kinds—such as quota exhaustion—may have immediateOpen: true set, which bypasses the DEGRADED state and opens the circuit immediately upon detection. Review src/shared/utils/classify429.ts to see how specific HTTP 429 responses map to these failure kinds.

Can I prevent local network errors from triggering the circuit breaker?

Yes. Pass a custom isFailure predicate to getCircuitBreaker() that filters out local stream lifecycle errors. Use the isLocalStreamLifecycleError utility (lines 46-65) to identify client-side aborts and return false for those errors, preventing them from incrementing the failure count while still catching legitimate upstream failures.

How do I reset the circuit breaker after fixing an upstream provider issue?

Import resetAllCircuitBreakers from src/shared/utils/circuitBreaker.ts and invoke it to clear all registered breakers globally. For a single provider, obtain the specific instance via getCircuitBreaker(name) and call .reset() on it. This immediately transitions the state to CLOSED, clears all failure counters, and resets the openCycleCount to zero.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}