# How to Debug Connection Cooldown Issues When Keys Are Being Skipped in OmniRoute

> Debug OmniRoute connection cooldown issues and skipped keys. Learn how temporary exclusions work and get your API keys routing again.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: how-to-guide
- Published: 2026-08-11

---

**Connection cooldown in OmniRoute temporarily excludes API keys from routing after recoverable errors like 429s or quota exhaustion, causing them to appear "skipped" until the cooldown timestamp expires.**

OmniRoute implements **connection cooldown** (also called *rate-limited-until*) as a core resilience mechanism that protects upstream providers from repeated failed requests. When a key displays unexpected skip behavior, understanding the cooldown pipeline helps you diagnose whether the cause is legitimate backpressure, misconfigured quotas, or stale database state. This guide walks through the architecture, diagnostic steps, and resolution methods based on the diegosouzapw/OmniRoute source code.

---

## Understanding the Connection Cooldown Architecture

OmniRoute's cooldown system spans multiple layers with clear separation of concerns:

| Layer | Key File | Purpose |
|-------|----------|---------|
| **Authentication & marking** | [`src/sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/auth.ts) | `markAccountUnavailable()` writes `rateLimitedUntil` timestamps to the database |
| **Lazy recovery** | [`src/lib/quota/connectionRecovery.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/quota/connectionRecovery.ts) | Background task clears expired cooldowns on `OMNIROUTE_CONNECTION_RECOVERY_INTERVAL_MS` |
| **Persistence** | [`src/lib/db/providers.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/db/providers.ts) | Stores `rate_limited_until` column for each connection |
| **Health visibility** | [`src/lib/monitoring/providerHealthMatrix.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/monitoring/providerHealthMatrix.ts) | Exposes cooldown state via `/api/monitoring/health` |
| **Management** | `/api/resilience` and `/api/providers/[id]` | REST endpoints to view and clear cooldowns |

The three resilience mechanisms—**circuit breaker** (provider-wide), **connection cooldown** (per-connection), and **model lockout** (per-model in-memory)—are documented in [`docs/architecture/RESILIENCE_GUIDE.md`](https://github.com/diegosouzapw/OmniRoute/blob/main/docs/architecture/RESILIENCE_GUIDE.md). Connection cooldown specifically targets individual API keys to allow graceful degradation without disabling entire providers.

---

## Why Keys Appear Skipped: The Cooldown Pipeline

A key becomes unavailable to the routing algorithm through four sequential steps:

1. **Error classification** — `checkFallbackError` in [`src/sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/accountFallback.ts) identifies recoverable errors (429, 503, quota-exhausted)
2. **Cooldown marking** — The handler calls `markAccountUnavailable(connectionId, cooldownMs)` which persists a future timestamp
3. **Routing filtration** — `getProviderCredentials` filters out connections where `isAccountUnavailable()` returns true
4. **Lazy re-entry** — The connection recovers when `rateLimitedUntil` passes, either on next read or via background recovery

If your cooldown duration exceeds expectations, keys remain skipped for extended periods.

---

## Common Cooldown Triggers and Their Signatures

| Trigger | Default `cooldownMs` | Log Indicator |
|---------|----------------------|---------------|
| **Standard 429** (no `Retry-After`) | `baseCooldownMs * 2^n` exponential: 3s (API key), 5s (OAuth) start | `connection cooldown active` |
| **429 with `Retry-After`** | Header value, capped at `MAX_RETRY_INTERVAL_SEC` | `cooldown 28s` badge in health UI |
| **5xx errors** | Same exponential back-off | `connection cooldown active` |
| **Quota exhaustion** | Provider-specific windows (e.g., 1h for daily quota) | `QUOTA_EXHAUSTED` badge |

The **back-off level** increments with each consecutive failure and resets only on successful requests via `clearAccountError`.

---

## Step-by-Step Debug Workflow

### Verify Current Cooldown State via Health Endpoint

```bash
curl -s http://localhost:20128/api/monitoring/health | jq .providers

```

Inspect the response for `rateLimitedUntil` fields. Active cooldowns display badges like `cooldown 2/3 • 28s` indicating level 2 of 3 with 28 seconds remaining.

*Source*: [`src/lib/monitoring/providerHealthMatrix.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/monitoring/providerHealthMatrix.ts)

---

### Query the Database Directly

```bash
npm run db:cli

```

```sql
SELECT id, provider, rate_limited_until, backoff_level
FROM provider_connections
WHERE provider = 'openai';

```

The `rate_limited_until` column stores ISO 8601 timestamps or numeric epochs. The `backoff_level` column reveals accumulated penalty severity.

*Source*: [`src/lib/db/providers.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/db/providers.ts)

---

### Analyze Application Logs

```bash
grep -i "cooldown\|unavailable" -R logs/ | tail -n 20

```

Expected output format:

```

A3 guard: marking connection 7f3a… unavailable for 28 000 ms (connection cooldown active)

```

Log emissions originate from `markAccountUnavailable` in [`src/sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/auth.ts).

---

### Identify the Root Cause

Three diagnostic approaches determine why cooldown was applied:

1. **Capture upstream headers** — Use `curl -v` to inspect `Retry-After` on failed requests
2. **Parse quota messages** — Check response bodies for "daily quota exceeded"; [`src/sse/services/quotaTextCooldowns.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/quotaTextCooldowns.ts) converts these to extended cooldowns
3. **Check error codes** — 5xx responses always trigger exponential back-off regardless of body content

---

### Clear Stale Cooldowns via Management API

For a specific connection:

```bash
curl -X PATCH http://localhost:20128/api/providers/<connection-id> \
  -H "Authorization: Bearer <management-token>" \
  -H "Content-Type: application/json" \
  -d '{"rateLimitedUntil": null}'

```

To clear **model lockouts only** (in-memory, does not affect connection cooldowns):

```bash
curl -X DELETE http://localhost:20128/api/resilience/model-cooldowns \
  -H "Authorization: Bearer <management-token>"

```

*Source*: [`docs/reference/API_REFERENCE.md`](https://github.com/diegosouzapw/OmniRoute/blob/main/docs/reference/API_REFERENCE.md)

---

### Force Immediate Recovery (Optional)

When background recovery appears stalled:

```bash
npm run ts-node src/lib/quota/connectionRecovery.ts --run-once

```

This executes the same logic as the scheduled task: queries all connections, nullifies expired `rateLimitedUntil` values, and commits transactions.

*Source*: [`src/lib/quota/connectionRecovery.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/quota/connectionRecovery.ts)

---

### Validate the Fix

Re-run the health endpoint and confirm `rateLimitedUntil` is null or in the past. Submit a test request that previously triggered fallback to verify the original key is now selected.

---

## Programmatic Cooldown Operations

### Manually Trigger Cooldown

```typescript
import { markAccountUnavailable } from "@/sse/services/auth";

await markAccountUnavailable(connectionId, 30_000); // 30 second cooldown

```

*Reference*: `src/sse/services/auth.ts::markAccountUnavailable` (line ~1927)

---

### Parse Retry-After Headers

```typescript
import { parseRetryAfterHeader } from "@/sse/services/quotaResetParsing";

const cooldownMs = parseRetryAfterHeader(response.headers.get("Retry-After"));
// Returns milliseconds from seconds or HTTP-date strings

```

*Reference*: [`src/sse/services/quotaResetParsing.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/quotaResetParsing.ts)

---

### Check Availability in Custom Logic

```typescript
import { isAccountUnavailable } from "@/sse/services/auth";

if (isAccountUnavailable(connection.rateLimitedUntil)) {
  // Exclude from routing candidate pool
}

```

*Reference*: `src/sse/services/auth.ts::isAccountUnavailable` (line ~105)

---

## Key Files Reference

| File | Function |
|------|----------|
| [`src/sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/auth.ts) | Core cooldown marking and availability checking |
| [`src/lib/quota/connectionRecovery.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/quota/connectionRecovery.ts) | Expired cooldown cleanup |
| [`src/lib/db/providers.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/db/providers.ts) | Schema and persistence |
| [`src/lib/monitoring/providerHealthMatrix.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/monitoring/providerHealthMatrix.ts) | Health endpoint implementation |
| [`docs/architecture/RESILIENCE_GUIDE.md`](https://github.com/diegosouzapw/OmniRoute/blob/main/docs/architecture/RESILIENCE_GUIDE.md) | Design documentation |
| [`docs/reference/API_REFERENCE.md`](https://github.com/diegosouzapw/OmniRoute/blob/main/docs/reference/API_REFERENCE.md) | Management API specification |

---

## Quick Troubleshooting Reference

| Symptom | Cause | Resolution |
|---------|-------|------------|
| Immediate skip after 429 with `Retry-After: 60` | Header-respected cooldown | Wait or PATCH `/api/providers/:id` |
| Multi-hour skips without header | Quota-exhaustion parser triggered | Inspect error body; clear if safe |
| Numeric epoch string in `rateLimitedUntil` | Legacy storage format | Upgrade to release with fix #3995 |
| All provider connections cooldown after single 503 | Anti-thundering-herd with escalated `backoffLevel` | Check and reset `backoffLevel` via PATCH |
| Permanent cooldown after server restart | Far-future timestamp from crash burst | Run `clearStaleCrashCooldowns` or manual PATCH |

---

## Summary

- **Connection cooldown** excludes individual API keys from routing via the `rateLimitedUntil` timestamp in `provider_connections`
- **Debug sequence**: health endpoint → database query → log analysis → root cause identification → management API clearance
- **Recovery** occurs lazily on read or via [`connectionRecovery.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/connectionRecovery.ts) background task
- **Programmatic control** available through `markAccountUnavailable`, `isAccountUnavailable`, and PATCH `/api/providers/:id`
- **Distinguish** connection cooldown (persistent) from model lockout (in-memory) and circuit breaker (provider-wide)

---

## Frequently Asked Questions

### Does cooling one connection affect others in the same provider?

No. Cooldown is strictly **per-connection**. Only the circuit breaker mechanism disables entire providers. Multiple connections for the same provider route independently unless all simultaneously enter cooldown.

### Can cooldown be disabled for specific connections?

Yes. Enable the **"disable cooldown"** toggle in the connection dashboard, which sets `providerSpecificData.disableCooling = true`. This bypasses `markAccountUnavailable` calls for that connection.

### How does back-off escalation work?

Each `markAccountUnavailable` call increments `backoffLevel`. The cooldown duration calculates as `baseCooldownMs * Math.pow(2, backoffLevel)` with a ceiling at `MAX_RETRY_INTERVAL_SEC`. Successful requests trigger `clearAccountError` which resets `backoffLevel` to zero.

### Where are model lockouts stored?

Model lockouts reside **in-memory only** and never persist to `rate_limited_until`. Access them via `GET /api/resilience/model-cooldowns`. These reflect temporary model-specific failures distinct from connection-level rate limiting.