How to Debug OmniRoute Routing Issues Using the Circuit Breaker Dashboard

You can debug OmniRoute routing failures by querying the /api/monitoring/health endpoint to inspect real-time provider circuit breaker states, failure counts, and automatic recovery timestamps.

OmniRoute implements resilient request routing through layered protection mechanisms that isolate failing providers. When upstream services return HTTP 5xx errors or time out, the system activates circuit breakers to prevent cascading failures. The circuit breaker dashboard exposes internal health metrics through a dedicated API endpoint, allowing you to diagnose why requests are being dropped or rerouted.

Understanding OmniRoute's Circuit Breaker Architecture

OmniRoute protects request routing with three distinct resilience layers: provider circuit breakers, connection cooldowns, and model lockouts. When a provider repeatedly fails, the circuit breaker flips to OPEN and temporarily blocks all traffic to that provider while the routing engine falls back to healthy candidates.

The core implementation resides in src/shared/utils/circuitBreaker.ts, which manages state tracking, back-off timers, and failure thresholds. According to the Resilience Guide in docs/architecture/RESILIENCE_GUIDE.md, these mechanisms work together to ensure that transient outages do not degrade overall system availability.

Accessing the Circuit Breaker Dashboard

The dashboard exposes provider health data through the internal endpoint /api/monitoring/health. The route handler is implemented in src/app/api/monitoring/health/route.ts, which serializes the in-memory breaker states into a JSON payload.

When you query this endpoint, the response contains an array of provider objects with the following diagnostic fields:

• provider: Human-readable identifier (e.g., openai, anthropic).
• circuitBreakerState: Current state as "CLOSED", "OPEN", or "HALF_OPEN".
• circuitOpenUntil: Epoch-millisecond timestamp indicating when the OPEN state expires.
• failureCount: Consecutive failures that triggered the current state.
• resetTimeoutMs: Configured back-off timeout for this provider.
• lastError: The most recent error message (e.g., "504 Gateway Timeout").

Interpreting Dashboard Metrics

Circuit Breaker States

The dashboard reports three distinct states that indicate provider health:

• CLOSED: The provider is healthy and accepting traffic. If routing issues persist while the state is closed, investigate rate limits or model lockouts instead.
• OPEN: The provider is blocked due to consecutive failures. Check circuitOpenUntil to determine when automatic recovery will occur.
• HALF_OPEN: The system is probing the provider with a test request. A successful probe closes the circuit; a failure reopens it.

Key Diagnostic Fields

The lastError field reveals the root cause of the failure streak, such as "502 Bad Gateway" or connection timeouts. Correlate this with failureCount to determine if the provider is approaching its threshold (OAuth providers default to 3 failures, API-key providers to 5, as defined in src/shared/config/constants.ts).

The circuitOpenUntil timestamp implements lazy recovery, ensuring the provider remains isolated until the configured resetTimeoutMs elapses.

Step-by-Step Debugging Workflow

Follow this systematic approach to diagnose routing anomalies using the dashboard:

1. Query the health endpoint using curl http://localhost:20128/api/monitoring/health or fetch it from your browser console.

2. Locate the target provider in the JSON response array.

3. Check circuitBreakerState:

   • If CLOSED, examine lastError and request logs for non-circuit issues.
   • If OPEN, note the circuitOpenUntil timestamp to estimate recovery time.
   • If HALF_OPEN, wait for the probe to complete or investigate immediate failures.

4. Inspect lastError to identify whether the issue stems from network timeouts, HTTP errors, or authentication failures.

5. Correlate with request logs in open-sse/utils/requestLogger.ts to confirm the request was short-circuited before reaching the upstream.

6. Adjust configuration if thresholds are too aggressive by modifying environment variables or constants in src/shared/config/constants.ts.

Manual Reset and Configuration Tuning

If a circuit breaker is open but you have verified the provider is healthy, you can force recovery by invoking resetAllCircuitBreakers() from src/shared/utils/circuitBreaker.ts. This function clears the in-memory state immediately, though the breaker will reopen if failures persist.

Provider-specific thresholds reside in src/shared/config/constants.ts, where OAuth providers default to 3 consecutive failures and API-key providers default to 5. Override these values via environment variables to tune sensitivity based on your reliability requirements.

Practical Code Examples

Fetch the dashboard programmatically using Node.js:

import fetch from 'node-fetch';

async function getCircuitStatus() {
  const res = await fetch('http://localhost:20128/api/monitoring/health');
  const data = await res.json();

  data.providers.forEach(p => {
    console.log(
      `${p.provider}: ${p.circuitBreakerState}` +
      (p.circuitBreakerState === 'OPEN' ? ` (unblocks @ ${new Date(p.circuitOpenUntil)})` : '')
    );
  });
}

getCircuitStatus();

Quick command-line inspection with curl and jq:

curl -s http://localhost:20128/api/monitoring/health | jq .

Render provider status in a React component:

import useSWR from 'swr';

export function ProviderBadge({ provider }: { provider: string }) {
  const { data } = useSWR('/api/monitoring/health', url => fetch(url).then(r => r.json()));
  const info = data?.providers?.find(p => p.provider === provider);

  if (!info) return null;
  const color = info.circuitBreakerState === 'OPEN' ? 'red' : 'green';
  return <span style={{ color }}>{info.circuitBreakerState}</span>;
}

Summary

• Query /api/monitoring/health (implemented in src/app/api/monitoring/health/route.ts) to retrieve real-time circuit breaker states for all providers.
• Interpret circuitBreakerState values (CLOSED, OPEN, HALF_OPEN) to determine if routing issues stem from circuit isolation or other factors.
• Use circuitOpenUntil and lastError to diagnose recovery timelines and root causes.
• Invoke resetAllCircuitBreakers() from src/shared/utils/circuitBreaker.ts for manual recovery after verifying provider health.
• Adjust failure thresholds in src/shared/config/constants.ts to balance resilience against sensitivity.

Frequently Asked Questions

How do I know if a routing failure is caused by a circuit breaker or a rate limit?

Check the circuitBreakerState field in the health endpoint response. If the state is CLOSED, the provider is healthy and the failure likely stems from rate limiting or model lockouts. If the state is OPEN or HALF_OPEN, the circuit breaker has isolated the provider due to consecutive errors, which you can verify by inspecting the lastError field for HTTP 5xx or timeout messages.

What is the difference between HALF_OPEN and OPEN states?

An OPEN circuit completely blocks traffic to the provider until the circuitOpenUntil timestamp passes. HALF_OPEN indicates the system is testing the provider with a single probe request. If the probe succeeds, the state returns to CLOSED; if it fails, the circuit reverts to OPEN with a refreshed timeout. This allows automatic recovery without manual intervention.

Can I manually close a circuit breaker before the timeout expires?

Yes. You can manually reset the circuit by calling resetAllCircuitBreakers() from src/shared/utils/circuitBreaker.ts, which clears the in-memory state immediately. However, only use this after confirming the provider is genuinely healthy, as premature reset will cause the breaker to reopen quickly if failures persist.

Where are the circuit breaker thresholds configured?

Default failure thresholds are defined in src/shared/config/constants.ts, with OAuth providers typically set to 3 failures and API-key providers to 5. You can override these values through environment variables to adjust how quickly the system isolates unreliable providers based on your specific reliability requirements.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}