How to Debug OmniRoute Circuit Breaker Failures and Understand Provider Lockout

OmniRoute protects downstream providers using a robust circuit breaker in src/shared/utils/circuitBreaker.ts that tracks four distinct states (CLOSED, DEGRADED, OPEN, HALF_OPEN), while a separate lightweight breaker in open-sse/services/tokenRefresh.ts handles authentication-specific lockouts via isProviderBlocked() checks.

OmniRoute implements a sophisticated resilience pattern to prevent cascading failures when calling external AI providers. The system maintains two distinct circuit breaker implementations: a generic request-level breaker for routing failures and a specialized token-refresh blocker for authentication issues. Both mechanisms persist their state to a SQLite-backed domainState table, enabling diagnostics across application restarts.

Understanding the Circuit Breaker State Machine

The core circuit breaker implementation in src/shared/utils/circuitBreaker.ts manages provider health through a finite state machine driven by failure counts and per-failure-kind thresholds.

The Four States of Resilience

Each provider (or named resource) receives its own CircuitBreaker instance via the registry function getCircuitBreaker(). The breaker transitions between these states based on accumulated failures:

• CLOSED: Normal operation where all requests pass through to the provider.
• DEGRADED: Failure rate is elevated but below the critical threshold; requests continue while warnings are logged.
• OPEN: The provider is fully locked out; all requests are short-circuited immediately to prevent overload.
• HALF_OPEN: Recovery probing state where a limited number of test requests are allowed to check if the provider has recovered.

State transitions occur when failureCount reaches configured thresholds. When the count hits failureThreshold, the breaker moves to OPEN (or DEGRADED first if only the degradation threshold is met).

Key Implementation Methods

The breaker logic centers on three critical methods in src/shared/utils/circuitBreaker.ts:

• execute(fn): Wraps async calls, applying state checks and transition logic before invoking the wrapped function.
• _onFailure(kind?): Tallies failures, applies per-kind thresholds from kindThresholds, and decides whether to open the circuit based on immediateOpen flags.
• _refreshOpenState(): Monitors the back-off timeout when in OPEN state; moves to HALF_OPEN once the reset period elapses.

Token Refresh and Provider Lockout Mechanisms

Authentication failures trigger a separate lightweight circuit breaker located in open-sse/services/tokenRefresh.ts. This mechanism specifically handles token refresh failures through a simple counter-based approach.

When recordFailure() detects consecutive failures for a provider, it increments a counter stored in _circuitBreaker[provider].failures. Once this exceeds CIRCUIT_BREAKER_THRESHOLD, the provider is marked as blocked:

// open-sse/services/tokenRefresh.ts
if (_circuitBreaker[provider].failures >= CIRCUIT_BREAKER_THRESHOLD) {
  _circuitBreaker[provider].blockedUntil = Date.now() + CIRCUIT_BREAKER_COOLDOWN;
}

The helper isProviderBlocked(provider) checks this timestamp, returning true while the lockout persists:

// open-sse/services/tokenRefresh.ts
if (!state) return false;
if (!state.blockedUntil) return false;
if (state.blockedUntil > Date.now()) return true;

Debugging Circuit Breaker Failures Step-by-Step

Inspect Global Breaker Status

To diagnose which providers are currently affected, use the global registry helper that returns a JSON array containing state, failureCount, lastFailureTime, retryAfterMs, and openCycleCount:

import { getAllCircuitBreakerStatuses } from '@/shared/utils/circuitBreaker';

function dumpStatuses() {
  const list = getAllCircuitBreakerStatuses();
  list.forEach((s) => {
    console.log(
      `${s.name}: ${s.state} – failures:${s.failureCount} – retryAfter:${s.retryAfterMs}ms`
    );
  });
}
dumpStatuses();

This queries the SQLite persistence layer and reflects the current in-memory state as implemented in src/shared/utils/circuitBreaker.ts (lines 885-898).

Check Specific Provider Lockout

For authentication-specific blocks, import the token refresh utilities to verify lockout status:

import { getCircuitBreakerStatus, isProviderBlocked } from '@/open-sse/services/tokenRefresh';

console.log(getCircuitBreakerStatus()['openai']);
console.log(isProviderBlocked('openai')); // Returns true if blockedUntil > Date.now()

Analyze Transition History

Each CircuitBreaker instance records the last 20 state transitions (configurable via maxTransitionHistory). Access this history to understand when and why a provider entered OPEN state:

import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';

function showHistory(id: string) {
  const breaker = getCircuitBreaker(id);
  console.table(breaker.getStatus().transitionHistory);
}
showHistory('openai');

Force Reset After Recovery

When you confirm an upstream service has recovered, manually reset the breaker to clear failure counts and move to CLOSED:

import { resetAllCircuitBreakers } from '@/shared/utils/circuitBreaker';

resetAllCircuitBreakers(); // Clears all breakers and DB rows

For a single provider:

import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';

function resetProviderBreaker(id: string) {
  const breaker = getCircuitBreaker(id);
  breaker.reset(); // Clears counts, moves to CLOSED
  console.info(`Breaker for ${id} has been reset`);
}

Tune Per-Failure-Kind Thresholds

Configure different thresholds for specific error types using kindThresholds. This allows rate-limit errors (429s) to trigger degradation faster than transient 5xx errors:

import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';

const breaker = getCircuitBreaker('myProvider', {
  failureThreshold: 10,
  kindThresholds: {
    rate_limit: { threshold: 3, immediateOpen: false },
    transient:  { threshold: 5, immediateOpen: true },
  },
});

The immediateOpen flag determines whether reaching that specific threshold forces an immediate transition to OPEN state.

Monitor Back-off Escalation

After repeated OPEN→HALF_OPEN→OPEN cycles, the reset timeout grows exponentially via _effectiveResetTimeout. Check the current effective timeout:

const breaker = getCircuitBreaker('myProvider');
console.log(breaker.getStatus().effectiveResetTimeout);

This value increases based on maxBackoffMultiplier and backoffEscalationCount settings as implemented in src/shared/utils/circuitBreaker.ts (lines 223-232).

Common Circuit Breaker Pitfalls

Symptom	Likely Cause	Verification Method
Provider remains blocked after a single 502	The token-refresh breaker treats consecutive failures as immediate blocks; if the counter was left at >= CIRCUIT_BREAKER_THRESHOLD, the provider stays locked.	Check recordFailure counters via getCircuitBreakerStatus().
Requests receive 429s despite breaker showing CLOSED	The breaker may be in DEGRADED state (elevated failure rate) but not yet OPEN, allowing requests through with warnings.	Inspect state field in getAllCircuitBreakerStatuses() output.
Critical failures never open the circuit	Missing immediateOpen flag for the specific failure kind in kindThresholds.	Verify kindThresholds[<kind>].immediateOpen is set to true for critical error types.
Back-off period exceeds expected duration	maxBackoffMultiplier or backoffEscalationCount configured too low.	Check breaker.maxBackoffMultiplier via getStatus().

Summary

Debugging OmniRoute circuit breaker failures requires understanding both the generic request-level breaker and the authentication-specific token refresh blocker:

• Primary breaker lives in src/shared/utils/circuitBreaker.ts and manages four states (CLOSED, DEGRADED, OPEN, HALF_OPEN) with per-kind thresholds.
• Token refresh blocker resides in open-sse/services/tokenRefresh.ts and locks providers using CIRCUIT_BREAKER_THRESHOLD and CIRCUIT_BREAKER_COOLDOWN.
• Diagnostics are available via getAllCircuitBreakerStatuses(), getCircuitBreakerStatus(), and transitionHistory inspection.
• Recovery can be accelerated using resetAllCircuitBreakers() or individual breaker.reset() calls.
• Persistence to SQLite via the domainState table ensures diagnostics survive restarts.

Frequently Asked Questions

How do I check if a specific provider is currently locked out?

Import isProviderBlocked from @/open-sse/services/tokenRefresh and pass the provider ID. This function checks the _circuitBreaker[provider].blockedUntil timestamp against Date.now(). For request-level circuit breakers, use getCircuitBreaker(provider).getStatus().state to see if it is OPEN or HALF_OPEN.

What is the difference between the DEGRADED and OPEN states?

DEGRADED allows requests to continue passing through while logging warnings and tracking elevated failure rates; it serves as an early warning before full lockout. OPEN immediately short-circuits all requests to the provider, returning errors without attempting the external call. The transition from DEGRADED to OPEN occurs when failureCount reaches the configured failureThreshold.

Why does my provider stay blocked after only one HTTP 502 error?

The token refresh circuit breaker in open-sse/services/tokenRefresh.ts uses a simple counter (_circuitBreaker[provider].failures) that persists across operations. If the counter was previously elevated and reached CIRCUIT_BREAKER_THRESHOLD, the provider remains blocked until blockedUntil exceeds the current time. Check the failure count using getCircuitBreakerStatus() and reset if necessary.

How do I permanently reset a circuit breaker after fixing an upstream issue?

For a single provider, retrieve the breaker instance via getCircuitBreaker('providerId') and call .reset(). To clear all breakers simultaneously, import resetAllCircuitBreakers from @/shared/utils/circuitBreaker. Both operations clear the in-memory counters and update the SQLite persistence layer in the domainState table.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}