How to Debug OmniRoute Circuit Breaker Issues: A Complete Troubleshooting Guide

Use the health dashboard, API endpoints, and structured logs in src/shared/utils/circuitBreaker.ts to identify open breakers, inspect failure codes, and reset provider states when upstream LLM services recover.

OmniRoute's provider circuit breaker sits at the top of its three-layer resilience system, blocking traffic to entire LLM providers when repeated upstream failures occur. Understanding how this state machine operates—where it lives, how it transitions, and where it guards the request pipeline—lets you quickly diagnose why providers appear offline and restore normal routing.

Circuit Breaker Architecture in OmniRoute

The circuit breaker implementation spans five key components across the codebase:

Component	Role	Implementation File
CircuitBreaker class	Core state machine tracking failures, timestamps, and states (CLOSED, OPEN, HALF_OPEN)	src/shared/utils/circuitBreaker.ts
Provider-level registry	Isolates one misbehaving provider from others via per-provider instances	src/lib/warmupScheduler/circuitBreakerFactory.ts
AccountFallback logic	Guards routing by checking breaker.canExecute() before request dispatch	open-sse/services/accountFallback.ts
Chat-handler guard	Prevents SSE streams from inadvertently tripping the breaker	src/sse/handlers/chatHelpers.ts
Health & reset API	Exposes breaker status and allows manual recovery	/api/resilience/reset

The CircuitBreaker class in src/shared/utils/circuitBreaker.ts implements a standard state machine with three states:

• CLOSED: Normal operation, requests pass through
• OPEN: Blocking all traffic, storing an expiration timestamp in resetAfterMs
• HALF_OPEN: Single probe request allowed to test recovery

Each provider receives its own isolated breaker instance through the factory in src/lib/warmupScheduler/circuitBreakerFactory.ts, ensuring one failing LLM service doesn't cascade to others.

How Provider Circuit Breaker Failures Flow

Understanding the lifecycle helps you identify where to intervene:

1. Upstream error detection: When a provider returns failure codes (408, 500, 502, 503, 504), CircuitBreaker.recordFailure() increments the failure counter.

2. State transition: After the configured threshold (default 3 for OAuth providers, 5 for API-key providers), the breaker moves CLOSED → OPEN and records resetAfterMs.

3. Lazy recovery: The next breaker.canExecute() call checks the timeout. If elapsed, it flips to HALF_OPEN and allows one probe request.

4. Resolution: Success closes the breaker; failure resets the timeout and returns to OPEN.

Critical distinction: Only provider-level failures trigger this flow. Connection-level rate-limits (429) and model-specific quota errors are handled by the connection cooldown and model lockout layers—not by the provider breaker. This separation explains why a provider might appear "offline" while individual connections still work.

Step-by-Step Debugging Process

1. Check the Health Dashboard

The UI (/dashboard/health) and CLI provide immediate visibility into breaker states:

omniroute health components

Look for the circuit breaker card showing each provider's current state (CLOSED, HALF_OPEN, OPEN).

2. Query the Health API Directly

For programmatic access to raw metrics:

GET /api/monitoring/health
Accept: application/json

The response contains a providers map with circuitBreaker fields:

{
  "providers": {
    "openai": {
      "circuitBreaker": {
        "state": "OPEN",
        "consecutiveFailures": 5,
        "nextRetryMs": 30000
      }
    }
  }
}

3. Inspect Structured Logs

src/shared/utils/circuitBreaker.ts logs every transition via pino. Search logs for "circuitBreaker" to find:

• When the breaker opened
• Which error code caused the transition
• The resetAfterMs value set

4. Validate Failure Classification

Ensure the triggering error is truly a provider failure. The classification predicate lives in src/shared/utils/circuitBreaker.ts as PROVIDER_FAILURE_ERROR_CODES.

Known issue: If 429 appears in this list, the breaker trips on normal rate-limit responses. This regression was fixed in #1767.

5. Manually Reset When Safe

If the provider has recovered, bypass the timeout:

POST /api/resilience/reset
Content-Type: application/json

{ "provider": "openai" }

This calls resetProviderCircuitBreaker(provider), clearing consecutiveFailures and forcing state to CLOSED.

6. Reproduce in Isolation

Test the breaker logic directly using the exported CircuitBreaker class:

import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';

async function logBreakerState(provider: string) {
  const breaker = getCircuitBreaker(provider);
  console.log(`[${provider}] state = ${breaker.getStatus()}`);
  console.log(`failures = ${breaker.consecutiveFailures}`);
  console.log(`next retry in ${breaker.getRetryAfterMs()} ms`);
}

await logBreakerState('openai');

The getStatus() and getRetryAfterMs() helpers expose internal state for debugging.

Common Circuit Breaker Issues and Fixes

Symptom	Root Cause	Solution
Provider stays OPEN despite healthy upstream	Clock skew or overridden resetAfterMs	Verify system clock; use reset API
All models blocked, other providers work	Provider breaker opened on valid failure	Check error code isn't 429; reference #1767
Single account blocks entire provider	Account cooldown propagating incorrectly	Review open-sse/services/accountFallback.ts guard order
Never recovers after transient outage	HALF_OPEN probe keeps failing	Increase reset timeout via environment variables
Frequent trips for stable provider	Overly aggressive threshold	Tune in src/shared/constants/resilience.ts

Code Examples for Debugging OmniRoute Circuit Breakers

CLI Script to Reset a Provider

import fetch from 'node-fetch';

async function resetBreaker(provider: string) {
  const res = await fetch('http://localhost:20128/api/resilience/reset', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ provider })
  });
  const json = await res.json();
  console.log('Reset response:', json);
}

resetBreaker('anthropic');

Bash One-Liner for Health Checks

curl -s http://localhost:20128/api/monitoring/health \
  | jq '.providers.openai.circuitBreaker'

Sample output:

{
  "state": "OPEN",
  "consecutiveFailures": 5,
  "nextRetryMs": 30000
}

Unit Test Pattern for Recovery Validation

import { getCircuitBreaker } from '@/shared/utils/circuitBreaker';
import assert from 'assert';

describe('Provider breaker half-open recovery', () => {
  const provider = 'gemini';
  const breaker = getCircuitBreaker(provider);

  it('opens after threshold', () => {
    for (let i = 0; i < 5; i++) breaker.recordFailure(new Error('500'));
    assert.equal(breaker.getStatus(), 'OPEN');
  });

  it('recovers after timeout', async () => {
    breaker.forceOpenUntil(Date.now() - 1);
    assert.equal(breaker.canExecute(), true);
    assert.equal(breaker.getStatus(), 'CLOSED');
  });
});

Reference: tests/unit/provider-breaker-halfopen-recovery.test.ts

Key Source Files for Circuit Breaker Debugging

File	Purpose
src/shared/utils/circuitBreaker.ts	Core CircuitBreaker class with state machine and public helpers
src/lib/warmupScheduler/circuitBreakerFactory.ts	Factory for per-provider breaker instances
open-sse/services/accountFallback.ts	Routing guard calling breaker.canExecute()
src/sse/handlers/chatHelpers.ts	SSE stream guard respecting breaker state
docs/architecture/RESILIENCE_GUIDE.md	Three-layer resilience design overview
docs/reference/API_REFERENCE.md	Health and reset endpoint documentation
src/shared/constants/resilience.ts	Default thresholds (OAuth: 3, API-key: 5)

Summary

• OmniRoute's provider circuit breaker in src/shared/utils/circuitBreaker.ts uses a three-state machine (CLOSED → OPEN → HALF_OPEN) to protect against upstream LLM failures
• Debug via: health dashboard (/dashboard/health), CLI (omniroute health), API (/api/monitoring/health), and structured logs
• Reset manually through /api/resilience/reset when providers recover before timeout
• Distinguish provider failures (408, 500, 502, 503, 504) from connection rate-limits (429) to avoid misconfiguration
• Tune thresholds in src/shared/constants/resilience.ts if defaults prove too aggressive for your providers

Frequently Asked Questions

Why is my provider stuck in OPEN state even though the service is healthy?

The breaker may have a stale resetAfterMs timestamp due to clock skew, or the timeout was manually extended. Check system time synchronization, then use the reset API to force CLOSED state. Inspect logs for the original error code that triggered the open—if it was 429, see #1767 for the fix.

How do I distinguish a provider circuit breaker from connection cooldown?

Provider breakers live in src/shared/utils/circuitBreaker.ts and track consecutiveFailures with recordFailure(). Connection cooldown is a separate layer managing individual account rate-limits. The health API shows both: circuitBreaker.state for the provider, connectionStatus.cooldownUntil for accounts. Provider breakers only react to PROVIDER_FAILURE_ERROR_CODES, not 429 responses.

Can I disable the circuit breaker for a specific provider?

No direct disable exists, but you can set an effectively infinite threshold in src/shared/constants/resilience.ts or environment configuration. The breaker still initializes but won't open. For production, prefer tuning resetAfterMs rather than disabling—this preserves the safety net while reducing recovery time.

Where does the breaker guard actually block requests?

Two critical checkpoints enforce breaker.canExecute(): open-sse/services/accountFallback.ts for standard requests and src/sse/handlers/chatHelpers.ts for streaming responses. Both call getCircuitBreaker(provider) from the factory and reject routing when OPEN. If requests bypass these guards, they won't respect breaker state.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}