How to Debug Provider Connection Issues Using OmniRoute's Cooldown Tracker Logs

OmniRoute's cooldown tracker logs and health endpoint let you identify failing providers, inspect retry-after timing, and verify circuit-breaker recovery without reading the full log stream.

In diegosouzapw/OmniRoute, every provider connection is guarded by a circuit-breaker that records a cooldown state after repeated failures such as HTTP 429 or 503. You can debug provider connection issues using OmniRoute's provider cooldown tracker logs, the structured health endpoint, and the error classification logic in the source tree. The sections below walk through the exact log formats, source files, and debugging steps as implemented in the OmniRoute source code.

How OmniRoute Tracks Provider Cooldowns

OmniRoute monitors every provider connection with a circuit-breaker that aggregates cooldown counts, retry-after timestamps, and internal state across several observable layers.

Connection-Cooldown Summary

The core aggregation logic lives in src/sse/services/cooldownAwareRetry.ts. This service maintains per-provider counters for total connections, how many are currently in cooldown, and the next eligible retry time. The same aggregation is validated in src/lib/db/connection-cooldown-summary.test.ts.

Health Endpoint Provider Badges

The /api/monitoring/health route—implemented in src/app/api/monitoring/health/route.ts—returns a JSON payload that includes a provider badge such as "cooldown 2/3 · 28s". This badge indicates that 2 of 3 connections are cooling down and the next retry will be attempted in 28 seconds. The UI test file tests/unit/ui/comboFlowModel-cooldown.test.ts demonstrates how these badges render in combo flow diagrams.

Debug Log Format

Whenever a cooldown is applied or cleared, OmniRoute emits a debug line through the central logger in src/lib/logger.ts. The log format is:

provider=<id> cooldownCount=<n>/<total> retryAfterMs=<ms>

Internal Circuit-Breaker State

Inside src/sse/services/cooldownAwareRetry.ts, the CircuitBreaker objects expose cbState (OPEN, HALF_OPEN, CLOSED) and rateLimitedUntil. These fields are updated after each request and determine whether traffic is routed to a provider.

Step-by-Step Debugging Workflow

Follow these steps to trace a provider connection issue from symptom to root cause.

1. Query the health endpoint

   Start by checking which providers are currently in cooldown. Run:

   curl -s http://localhost:3000/api/monitoring/health | jq '.providers[] | select(.issues|contains("cooldown"))'

   The response includes cooldownCount, cooldownTotal, and cooldownRetryAfterMs for each affected provider. These values are aggregated by the logic in src/sse/services/cooldownAwareRetry.ts.

2. Inspect the server logs

   Look for debug lines containing the string cooldown. A typical entry looks like this:

   2024-07-27T12:34:56Z DEBUG provider=openai cooldownCount=2/3 retryAfterMs=28000

   These entries are emitted from the cooldown-aware retry service via src/lib/logger.ts and confirm exactly when a cooldown was added or cleared.

3. Identify the triggering error

   Cooldowns originate from upstream error responses such as HTTP 429, 503, or 401. The classifyFailKind function in src/open-sse/handlers/chatCore/cooldownClassification.ts maps these error messages to the "cooldown" kind. Cross-reference the timestamp of the debug log with your request trace—or reproduce the call with curl -v—to see the raw provider response.

4. Verify cooldown clearance

   After the retry-after period expires, the circuit-breaker automatically transitions from OPEN to HALF_OPEN, and then to CLOSED on the next successful request. The health endpoint badge will disappear, and the logs will show:

   2024-07-27T12:36:01Z DEBUG provider=openai cooldown cleared

   You can poll the health endpoint periodically to confirm this state change.

5. Force a cooldown reset

   If a provider remains stuck in cooldown due to a stale state after a crash, OmniRoute clears stale entries on startup via clearStaleCrashCooldowns in src/tests/unit/startup-stale-cooldown-recovery.test.ts. Restarting the server triggers this cleanup automatically.

Practical Code Examples

Retrieve Cooldown Info Programmatically

Use the health endpoint to fetch structured cooldown data for a specific provider:

import fetch from 'node-fetch';

async function getProviderCooldown(providerId: string) {
  const res = await fetch('http://localhost:3000/api/monitoring/health');
  const data = await res.json();
  const provider = data.providers.find((p: any) => p.id === providerId);
  return provider?.cooldownCount
    ? {
        count: provider.cooldownCount,
        total: provider.cooldownTotal,
        retryAfterMs: provider.cooldownRetryAfterMs,
      }
    : null;
}

getProviderCooldown('openai').then(console.log);

This script uses the same fields defined in src/sse/services/cooldownAwareRetry.ts.

Simulate a Cooldown Trigger

You can force a provider into cooldown by sending a request that elicits a 429 response:

# Send a request that forces a 429 response from the provider

curl -X POST http://localhost:3000/v1/chat/completions \
  -H "Authorization: Bearer $OMNIRoute_API_KEY" \
  -d '{"model":"gpt-4","messages":[{"role":"user","content":"spam"}]}'

# Immediately query the health endpoint to see the cooldown badge

curl -s http://localhost:3000/api/monitoring/health | jq '.providers[] | select(.id=="openai")'

Clear Stale Cooldowns on Startup

The following utility demonstrates how OmniRoute recovers from stale crash states:

import { clearStaleCrashCooldowns } from '../tests/unit/startup-stale-cooldown-recovery.test';

await clearStaleCrashCooldowns(); // runs automatically on server boot

In production, the startup script invokes this routine automatically.

Summary

• src/sse/services/cooldownAwareRetry.ts tracks per-provider cooldown counts, retry-after timestamps, and circuit-breaker state (OPEN, HALF_OPEN, CLOSED).
• The /api/monitoring/health endpoint exposes provider badges like "cooldown 2/3 · 28s" for real-time monitoring.
• Debug logs in src/lib/logger.ts print provider=<id> cooldownCount=<n>/<total> retryAfterMs=<ms> every time a cooldown is applied or cleared.
• The classifyFailKind function in src/open-sse/handlers/chatCore/cooldownClassification.ts maps upstream errors to cooldown events.
• Restarting the server triggers clearStaleCrashCooldowns in src/tests/unit/startup-stale-cooldown-recovery.test.ts to clear stale entries left by crashes.

Frequently Asked Questions

How do I know which provider is in cooldown without reading all logs?

Query the /api/monitoring/health endpoint and filter for providers with cooldown badges. The JSON fields cooldownCount, cooldownTotal, and cooldownRetryAfterMs tell you exactly how many connections are affected and when the next retry occurs according to the logic in src/sse/services/cooldownAwareRetry.ts.

What do the circuit-breaker states OPEN, HALF_OPEN, and CLOSED mean in OmniRoute?

OPEN means the provider is actively cooling down and receiving no traffic. HALF_OPEN means the retry-after window has expired and OmniRoute is testing the provider with a single request. CLOSED means the provider is healthy and fully available. These states are managed in src/sse/services/cooldownAwareRetry.ts.

Why did my provider enter cooldown even though my request seemed valid?

OmniRoute classifies upstream responses like HTTP 429, 503, and certain 401 errors as cooldown triggers through classifyFailKind in src/open-sse/handlers/chatCore/cooldownClassification.ts. Even if your local payload looks correct, the upstream provider may have rate-limited or rejected the request, causing the circuit-breaker to open.

Can I manually clear a cooldown without restarting the server?

The source code does not expose a public manual-clear API in the production path. The intended recovery path is automatic: the circuit-breaker transitions to HALF_OPEN after retryAfterMs and to CLOSED on success. If a cooldown is stuck due to a crash, restart the server to invoke clearStaleCrashCooldowns from src/tests/unit/startup-stale-cooldown-recovery.test.ts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}