# How to Debug Provider Connection Issues Using OmniRoute's Cooldown Tracker Logs

> Debug OmniRoute provider connection issues fast. Use cooldown tracker logs and health endpoint to pinpoint failing providers and verify circuit-breaker recovery. Avoid reading full logs.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: how-to-guide
- Published: 2026-07-27

---

**OmniRoute's cooldown tracker logs and health endpoint let you identify failing providers, inspect retry-after timing, and verify circuit-breaker recovery without reading the full log stream.**

In `diegosouzapw/OmniRoute`, every provider connection is guarded by a **circuit-breaker** that records a **cooldown** state after repeated failures such as HTTP 429 or 503. You can debug provider connection issues using OmniRoute's provider cooldown tracker logs, the structured health endpoint, and the error classification logic in the source tree. The sections below walk through the exact log formats, source files, and debugging steps as implemented in the OmniRoute source code.

## How OmniRoute Tracks Provider Cooldowns

OmniRoute monitors every provider connection with a circuit-breaker that aggregates cooldown counts, retry-after timestamps, and internal state across several observable layers.

### Connection-Cooldown Summary

The core aggregation logic lives in [`src/sse/services/cooldownAwareRetry.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/cooldownAwareRetry.ts). This service maintains per-provider counters for total connections, how many are currently in cooldown, and the next eligible retry time. The same aggregation is validated in [`src/lib/db/connection-cooldown-summary.test.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/db/connection-cooldown-summary.test.ts).

### Health Endpoint Provider Badges

The `/api/monitoring/health` route—implemented in [`src/app/api/monitoring/health/route.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/app/api/monitoring/health/route.ts)—returns a JSON payload that includes a provider badge such as `"cooldown 2/3 · 28s"`. This badge indicates that 2 of 3 connections are cooling down and the next retry will be attempted in 28 seconds. The UI test file [`tests/unit/ui/comboFlowModel-cooldown.test.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/tests/unit/ui/comboFlowModel-cooldown.test.ts) demonstrates how these badges render in combo flow diagrams.

### Debug Log Format

Whenever a cooldown is applied or cleared, OmniRoute emits a **debug** line through the central logger in [`src/lib/logger.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/logger.ts). The log format is:

```text
provider=<id> cooldownCount=<n>/<total> retryAfterMs=<ms>

```

### Internal Circuit-Breaker State

Inside [`src/sse/services/cooldownAwareRetry.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/cooldownAwareRetry.ts), the `CircuitBreaker` objects expose **`cbState`** (`OPEN`, `HALF_OPEN`, `CLOSED`) and **`rateLimitedUntil`**. These fields are updated after each request and determine whether traffic is routed to a provider.

## Step-by-Step Debugging Workflow

Follow these steps to trace a provider connection issue from symptom to root cause.

1. **Query the health endpoint**

   Start by checking which providers are currently in cooldown. Run:

   ```bash
   curl -s http://localhost:3000/api/monitoring/health | jq '.providers[] | select(.issues|contains("cooldown"))'
   ```

   The response includes `cooldownCount`, `cooldownTotal`, and `cooldownRetryAfterMs` for each affected provider. These values are aggregated by the logic in [`src/sse/services/cooldownAwareRetry.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/cooldownAwareRetry.ts).

2. **Inspect the server logs**

   Look for debug lines containing the string `cooldown`. A typical entry looks like this:

   ```text
   2024-07-27T12:34:56Z DEBUG provider=openai cooldownCount=2/3 retryAfterMs=28000
   ```

   These entries are emitted from the cooldown-aware retry service via [`src/lib/logger.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/logger.ts) and confirm exactly when a cooldown was added or cleared.

3. **Identify the triggering error**

   Cooldowns originate from upstream error responses such as HTTP 429, 503, or 401. The **`classifyFailKind`** function in [`src/open-sse/handlers/chatCore/cooldownClassification.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/open-sse/handlers/chatCore/cooldownClassification.ts) maps these error messages to the `"cooldown"` kind. Cross-reference the timestamp of the debug log with your request trace—or reproduce the call with `curl -v`—to see the raw provider response.

4. **Verify cooldown clearance**

   After the retry-after period expires, the circuit-breaker automatically transitions from `OPEN` to `HALF_OPEN`, and then to `CLOSED` on the next successful request. The health endpoint badge will disappear, and the logs will show:

   ```text
   2024-07-27T12:36:01Z DEBUG provider=openai cooldown cleared
   ```

   You can poll the health endpoint periodically to confirm this state change.

5. **Force a cooldown reset**

   If a provider remains stuck in cooldown due to a stale state after a crash, OmniRoute clears stale entries on startup via **`clearStaleCrashCooldowns`** in [`src/tests/unit/startup-stale-cooldown-recovery.test.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/tests/unit/startup-stale-cooldown-recovery.test.ts). Restarting the server triggers this cleanup automatically.

## Practical Code Examples

### Retrieve Cooldown Info Programmatically

Use the health endpoint to fetch structured cooldown data for a specific provider:

```typescript
import fetch from 'node-fetch';

async function getProviderCooldown(providerId: string) {
  const res = await fetch('http://localhost:3000/api/monitoring/health');
  const data = await res.json();
  const provider = data.providers.find((p: any) => p.id === providerId);
  return provider?.cooldownCount
    ? {
        count: provider.cooldownCount,
        total: provider.cooldownTotal,
        retryAfterMs: provider.cooldownRetryAfterMs,
      }
    : null;
}

getProviderCooldown('openai').then(console.log);

```

This script uses the same fields defined in [`src/sse/services/cooldownAwareRetry.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/cooldownAwareRetry.ts).

### Simulate a Cooldown Trigger

You can force a provider into cooldown by sending a request that elicits a 429 response:

```bash

# Send a request that forces a 429 response from the provider

curl -X POST http://localhost:3000/v1/chat/completions \
  -H "Authorization: Bearer $OMNIRoute_API_KEY" \
  -d '{"model":"gpt-4","messages":[{"role":"user","content":"spam"}]}'

# Immediately query the health endpoint to see the cooldown badge

curl -s http://localhost:3000/api/monitoring/health | jq '.providers[] | select(.id=="openai")'

```

### Clear Stale Cooldowns on Startup

The following utility demonstrates how OmniRoute recovers from stale crash states:

```typescript
import { clearStaleCrashCooldowns } from '../tests/unit/startup-stale-cooldown-recovery.test';

await clearStaleCrashCooldowns(); // runs automatically on server boot

```

In production, the startup script invokes this routine automatically.

## Summary

- **[`src/sse/services/cooldownAwareRetry.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/cooldownAwareRetry.ts)** tracks per-provider cooldown counts, retry-after timestamps, and circuit-breaker state (`OPEN`, `HALF_OPEN`, `CLOSED`).
- The **`/api/monitoring/health`** endpoint exposes provider badges like `"cooldown 2/3 · 28s"` for real-time monitoring.
- **Debug logs** in [`src/lib/logger.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/logger.ts) print `provider=<id> cooldownCount=<n>/<total> retryAfterMs=<ms>` every time a cooldown is applied or cleared.
- The **`classifyFailKind`** function in [`src/open-sse/handlers/chatCore/cooldownClassification.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/open-sse/handlers/chatCore/cooldownClassification.ts) maps upstream errors to cooldown events.
- Restarting the server triggers **`clearStaleCrashCooldowns`** in [`src/tests/unit/startup-stale-cooldown-recovery.test.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/tests/unit/startup-stale-cooldown-recovery.test.ts) to clear stale entries left by crashes.

## Frequently Asked Questions

### How do I know which provider is in cooldown without reading all logs?

Query the `/api/monitoring/health` endpoint and filter for providers with cooldown badges. The JSON fields `cooldownCount`, `cooldownTotal`, and `cooldownRetryAfterMs` tell you exactly how many connections are affected and when the next retry occurs according to the logic in [`src/sse/services/cooldownAwareRetry.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/cooldownAwareRetry.ts).

### What do the circuit-breaker states OPEN, HALF_OPEN, and CLOSED mean in OmniRoute?

`OPEN` means the provider is actively cooling down and receiving no traffic. `HALF_OPEN` means the retry-after window has expired and OmniRoute is testing the provider with a single request. `CLOSED` means the provider is healthy and fully available. These states are managed in [`src/sse/services/cooldownAwareRetry.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/cooldownAwareRetry.ts).

### Why did my provider enter cooldown even though my request seemed valid?

OmniRoute classifies upstream responses like HTTP 429, 503, and certain 401 errors as cooldown triggers through `classifyFailKind` in [`src/open-sse/handlers/chatCore/cooldownClassification.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/open-sse/handlers/chatCore/cooldownClassification.ts). Even if your local payload looks correct, the upstream provider may have rate-limited or rejected the request, causing the circuit-breaker to open.

### Can I manually clear a cooldown without restarting the server?

The source code does not expose a public manual-clear API in the production path. The intended recovery path is automatic: the circuit-breaker transitions to `HALF_OPEN` after `retryAfterMs` and to `CLOSED` on success. If a cooldown is stuck due to a crash, restart the server to invoke `clearStaleCrashCooldowns` from [`src/tests/unit/startup-stale-cooldown-recovery.test.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/tests/unit/startup-stale-cooldown-recovery.test.ts).