How to Debug Connection Cooldown Issues When Keys Are Being Skipped in OmniRoute

Connection cooldown in OmniRoute temporarily excludes API keys from routing after recoverable errors like 429s or quota exhaustion, causing them to appear "skipped" until the cooldown timestamp expires.

OmniRoute implements connection cooldown (also called rate-limited-until) as a core resilience mechanism that protects upstream providers from repeated failed requests. When a key displays unexpected skip behavior, understanding the cooldown pipeline helps you diagnose whether the cause is legitimate backpressure, misconfigured quotas, or stale database state. This guide walks through the architecture, diagnostic steps, and resolution methods based on the diegosouzapw/OmniRoute source code.

---

Understanding the Connection Cooldown Architecture

OmniRoute's cooldown system spans multiple layers with clear separation of concerns:

Layer	Key File	Purpose
Authentication & marking	src/sse/services/auth.ts	markAccountUnavailable() writes rateLimitedUntil timestamps to the database
Lazy recovery	src/lib/quota/connectionRecovery.ts	Background task clears expired cooldowns on OMNIROUTE_CONNECTION_RECOVERY_INTERVAL_MS
Persistence	src/lib/db/providers.ts	Stores rate_limited_until column for each connection
Health visibility	src/lib/monitoring/providerHealthMatrix.ts	Exposes cooldown state via /api/monitoring/health
Management	/api/resilience and /api/providers/[id]	REST endpoints to view and clear cooldowns

The three resilience mechanisms—circuit breaker (provider-wide), connection cooldown (per-connection), and model lockout (per-model in-memory)—are documented in docs/architecture/RESILIENCE_GUIDE.md. Connection cooldown specifically targets individual API keys to allow graceful degradation without disabling entire providers.

---

Why Keys Appear Skipped: The Cooldown Pipeline

A key becomes unavailable to the routing algorithm through four sequential steps:

1. Error classification — checkFallbackError in src/sse/services/accountFallback.ts identifies recoverable errors (429, 503, quota-exhausted)
2. Cooldown marking — The handler calls markAccountUnavailable(connectionId, cooldownMs) which persists a future timestamp
3. Routing filtration — getProviderCredentials filters out connections where isAccountUnavailable() returns true
4. Lazy re-entry — The connection recovers when rateLimitedUntil passes, either on next read or via background recovery

If your cooldown duration exceeds expectations, keys remain skipped for extended periods.

---

Common Cooldown Triggers and Their Signatures

Trigger	Default cooldownMs	Log Indicator
Standard 429 (no Retry-After)	baseCooldownMs * 2^n exponential: 3s (API key), 5s (OAuth) start	connection cooldown active
429 with Retry-After	Header value, capped at MAX_RETRY_INTERVAL_SEC	cooldown 28s badge in health UI
5xx errors	Same exponential back-off	connection cooldown active
Quota exhaustion	Provider-specific windows (e.g., 1h for daily quota)	QUOTA_EXHAUSTED badge

The back-off level increments with each consecutive failure and resets only on successful requests via clearAccountError.

---

Step-by-Step Debug Workflow

Verify Current Cooldown State via Health Endpoint

curl -s http://localhost:20128/api/monitoring/health | jq .providers

Inspect the response for rateLimitedUntil fields. Active cooldowns display badges like cooldown 2/3 • 28s indicating level 2 of 3 with 28 seconds remaining.

Source: src/lib/monitoring/providerHealthMatrix.ts

---

Query the Database Directly

npm run db:cli

SELECT id, provider, rate_limited_until, backoff_level
FROM provider_connections
WHERE provider = 'openai';

The rate_limited_until column stores ISO 8601 timestamps or numeric epochs. The backoff_level column reveals accumulated penalty severity.

Source: src/lib/db/providers.ts

---

Analyze Application Logs

grep -i "cooldown\|unavailable" -R logs/ | tail -n 20

Expected output format:

A3 guard: marking connection 7f3a… unavailable for 28 000 ms (connection cooldown active)

Log emissions originate from markAccountUnavailable in src/sse/services/auth.ts.

---

Identify the Root Cause

Three diagnostic approaches determine why cooldown was applied:

1. Capture upstream headers — Use curl -v to inspect Retry-After on failed requests
2. Parse quota messages — Check response bodies for "daily quota exceeded"; src/sse/services/quotaTextCooldowns.ts converts these to extended cooldowns
3. Check error codes — 5xx responses always trigger exponential back-off regardless of body content

---

Clear Stale Cooldowns via Management API

For a specific connection:

curl -X PATCH http://localhost:20128/api/providers/<connection-id> \
  -H "Authorization: Bearer <management-token>" \
  -H "Content-Type: application/json" \
  -d '{"rateLimitedUntil": null}'

To clear model lockouts only (in-memory, does not affect connection cooldowns):

curl -X DELETE http://localhost:20128/api/resilience/model-cooldowns \
  -H "Authorization: Bearer <management-token>"

Source: docs/reference/API_REFERENCE.md

---

Force Immediate Recovery (Optional)

When background recovery appears stalled:

npm run ts-node src/lib/quota/connectionRecovery.ts --run-once

This executes the same logic as the scheduled task: queries all connections, nullifies expired rateLimitedUntil values, and commits transactions.

Source: src/lib/quota/connectionRecovery.ts

---

Validate the Fix

Re-run the health endpoint and confirm rateLimitedUntil is null or in the past. Submit a test request that previously triggered fallback to verify the original key is now selected.

---

Programmatic Cooldown Operations

Manually Trigger Cooldown

import { markAccountUnavailable } from "@/sse/services/auth";

await markAccountUnavailable(connectionId, 30_000); // 30 second cooldown

Reference: src/sse/services/auth.ts::markAccountUnavailable (line ~1927)

---

Parse Retry-After Headers

import { parseRetryAfterHeader } from "@/sse/services/quotaResetParsing";

const cooldownMs = parseRetryAfterHeader(response.headers.get("Retry-After"));
// Returns milliseconds from seconds or HTTP-date strings

Reference: src/sse/services/quotaResetParsing.ts

---

Check Availability in Custom Logic

import { isAccountUnavailable } from "@/sse/services/auth";

if (isAccountUnavailable(connection.rateLimitedUntil)) {
  // Exclude from routing candidate pool
}

Reference: src/sse/services/auth.ts::isAccountUnavailable (line ~105)

---

Key Files Reference

File	Function
src/sse/services/auth.ts	Core cooldown marking and availability checking
src/lib/quota/connectionRecovery.ts	Expired cooldown cleanup
src/lib/db/providers.ts	Schema and persistence
src/lib/monitoring/providerHealthMatrix.ts	Health endpoint implementation
docs/architecture/RESILIENCE_GUIDE.md	Design documentation
docs/reference/API_REFERENCE.md	Management API specification

---

Quick Troubleshooting Reference

Symptom	Cause	Resolution
Immediate skip after 429 with Retry-After: 60	Header-respected cooldown	Wait or PATCH /api/providers/:id
Multi-hour skips without header	Quota-exhaustion parser triggered	Inspect error body; clear if safe
Numeric epoch string in rateLimitedUntil	Legacy storage format	Upgrade to release with fix #3995
All provider connections cooldown after single 503	Anti-thundering-herd with escalated backoffLevel	Check and reset backoffLevel via PATCH
Permanent cooldown after server restart	Far-future timestamp from crash burst	Run clearStaleCrashCooldowns or manual PATCH

---

Summary

• Connection cooldown excludes individual API keys from routing via the rateLimitedUntil timestamp in provider_connections
• Debug sequence: health endpoint → database query → log analysis → root cause identification → management API clearance
• Recovery occurs lazily on read or via connectionRecovery.ts background task
• Programmatic control available through markAccountUnavailable, isAccountUnavailable, and PATCH /api/providers/:id
• Distinguish connection cooldown (persistent) from model lockout (in-memory) and circuit breaker (provider-wide)

---

Frequently Asked Questions

Does cooling one connection affect others in the same provider?

No. Cooldown is strictly per-connection. Only the circuit breaker mechanism disables entire providers. Multiple connections for the same provider route independently unless all simultaneously enter cooldown.

Can cooldown be disabled for specific connections?

Yes. Enable the "disable cooldown" toggle in the connection dashboard, which sets providerSpecificData.disableCooling = true. This bypasses markAccountUnavailable calls for that connection.

How does back-off escalation work?

Each markAccountUnavailable call increments backoffLevel. The cooldown duration calculates as baseCooldownMs * Math.pow(2, backoffLevel) with a ceiling at MAX_RETRY_INTERVAL_SEC. Successful requests trigger clearAccountError which resets backoffLevel to zero.

Where are model lockouts stored?

Model lockouts reside in-memory only and never persist to rate_limited_until. Access them via GET /api/resilience/model-cooldowns. These reflect temporary model-specific failures distinct from connection-level rate limiting.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}