How to Debug Common OmniRoute Issues: Connection Cooldown, Model Lockout, and 429 Rate Limiting

Use OmniRoute's health and resilience APIs to isolate whether 429 errors stem from provider-wide circuit breakers, per-connection cooldowns, or per-model lockouts, then apply targeted resets via the admin endpoints.

OmniRoute implements a layered fault-tolerance architecture to manage upstream LLM provider failures gracefully. When you encounter repeated rate limits or specific models becoming unavailable, knowing how to inspect and reset the Connection Cooldown, Model Lockout, and 429 Rate Limiting mechanisms will minimize downtime and restore routing reliability.

Understanding OmniRoute's Three-Layer Resilience Architecture

OmniRoute protects the routing pipeline through three distinct scopes of failure handling. Each mechanism triggers under specific conditions and stores state that you can inspect via the monitoring API.

Provider Circuit Breaker

Scope: Entire provider (e.g., openai, glm).

Trigger: Accumulates provider-wide failures (HTTP 408/500/502/503/504) exceeding the configured failureThreshold.

Implementation: The core logic resides in [src/shared/utils/circuitBreaker.ts](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.50/src/shared/utils/circuitBreaker.ts). When triggered, the breaker prevents all traffic to that provider until the half-open probe succeeds.

Connection Cooldown

Scope: Single provider connection or API key.

Trigger: Retryable failures (408/429/5xx) on a specific connection, including upstream Retry-After hints.

Implementation: The markAccountUnavailable() function in src/sse/services/auth.ts marks the connection unavailable, while checkFallbackError() in [open-sse/services/accountFallback.ts](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.50/open-sse/services/accountFallback.ts) calculates exponential back-off as baseCooldownMs * 2 ** failureIndex (see lines 73-79).

Model Lockout

Scope: Provider + connection + model triple.

Trigger: Per-model quota or permission errors (403/404/429/502/503/504).

Implementation: The lockModel() and lockModelIfPerModelQuota() functions store a ModelLockoutEntry in an in-memory Map<string, ModelLockoutEntry>. Lockouts expire by timer or "success-decay," where successful requests halve the failure count (see [RESILIENCE_GUIDE.md](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.50/docs/architecture/RESILIENCE_GUIDE.md) lines 196-202).

Diagnosing 429 Rate Limiting in OmniRoute

HTTP 429 responses indicate rate limiting, but OmniRoute handles these differently depending on whether the limit is per-connection or per-model.

Why 429 Errors Trigger Different Mechanisms

• Per-connection 429: Exhausted account quotas trigger Connection Cooldown with a base duration of 3 seconds for API-key providers, growing exponentially thereafter.
• Per-model 429: Model-specific quotas trigger Model Lockout, routing subsequent requests for that model tuple to fallback connections or failing fast until the lockout expires.

The classification logic that distinguishes these cases lives in [src/shared/utils/classify429.ts](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.50/src/shared/utils/classify429.ts) (line 365).

How to Verify Active Rate Limits

Query the health and resilience endpoints to identify which layer is throttling traffic:

# Check provider-level circuit breaker status

curl -s http://localhost:20128/api/monitoring/health | jq '.providers["openai"]'

# Inspect specific connection cooldown state

curl -s http://localhost:20128/api/resilience/connections | jq '.connections[] | select(.id=="<connection-id>")'

If the response contains rateLimitedUntil with a future timestamp, the connection is actively cooling down. The system respects upstream Retry-After headers when calculating this timestamp.

Debugging Connection Cooldown Issues

Connection Cooldown isolates failing keys to prevent cascading failures across the provider pool.

Inspecting Connection State Programmatically

Use the auth service to retrieve detailed credential status including back-off levels:

import { getProviderCredentials } from '@/sse/services/auth';

async function diagnoseCooldown(provider: string, authType: string) {
  const creds = await getProviderCredentials({ provider, authType });
  
  console.log(`Connection ID: ${creds.id}`);
  console.log(`Status: ${creds.testStatus}`);
  console.log(`Cooldown until: ${new Date(Number(creds.rateLimitedUntil ?? 0))}`);
  console.log(`Backoff level: ${creds.backoffLevel}`);
}

// Example usage
diagnoseCooldown('openai', 'apiKey');

The backoffLevel indicates the current exponential back-off tier, while testStatus: "unavailable" confirms the connection is temporarily removed from rotation.

Remediation Steps

1. Wait for automatic recovery: A successful request automatically clears the error state via clearAccountError().
2. Manual reset: If the connection remains stuck after the cooldown window, force a reset via the admin API:

curl -X POST http://localhost:20128/api/resilience/reset \
  -H "Content-Type: application/json" \
  -d '{"provider":"openai","connectionId":"<id>"}'

Resolving Model Lockout Errors

Model Lockout prevents repeated attempts against models that return persistent quota or permission errors.

Viewing Active Model Lockouts

List all currently locked model tuples to identify which specific provider-connection-model combinations are restricted:

curl -s http://localhost:20128/api/resilience/model-cooldowns | jq

The response includes reason, until, and the calculated expiration based on exponential back-off settings configured in src/lib/resilience/modelLockoutSettings.ts.

Clearing Model Lockouts

If a model requires immediate availability before the automatic timer expires, delete the specific lockout entry:

curl -X DELETE http://localhost:20128/api/resilience/model-cooldowns \
  -H "Content-Type: application/json" \
  -d '{"provider":"openai","connection":"<connection-id>","model":"gpt-4o-mini"}'

Alternatively, clear lockouts programmatically:

async function clearModelLockout(provider: string, connectionId: string, model: string) {
  const resp = await fetch('http://localhost:20128/api/resilience/model-cooldowns', {
    method: 'DELETE',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ provider, connection: connectionId, model })
  });
  return resp.json();
}

If a model repeatedly re-locks, inspect the upstream provider's per-model quota dashboard—OmniRoute respects maxCooldownMs settings that may require manual quota expansion.

Complete Debugging Workflow

Follow this sequence to isolate and resolve resilience-related failures:

1. Check provider health: Query /api/monitoring/health to verify if the provider circuit breaker is open.
2. Inspect connections: Use /api/resilience/connections to identify cooling-down keys and their rateLimitedUntil timestamps.
3. List model lockouts: Verify /api/resilience/model-cooldowns for per-model restrictions.
4. Correlate headers: If upstream logs show Retry-After headers, confirm OmniRoute honored them in the cooldown calculation (see [accountFallback.ts](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.50/open-sse/services/accountFallback.ts#L73-L79) lines 73-79).
5. Apply resets: Use manual reset endpoints only when automatic back-off fails to recover within the expected window.

Summary

• Provider Circuit Breakers protect entire providers after systemic failures and are queryable via /api/monitoring/health.
• Connection Cooldown isolates individual API keys with exponential back-off calculated in [open-sse/services/accountFallback.ts](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.50/open-sse/services/accountFallback.ts).
• Model Lockout restricts specific model tuples using in-memory state with success-decay recovery.
• Use getProviderCredentials() to inspect connection states programmatically, and the /api/resilience/* endpoints to view and reset cooldowns.
• Always check for upstream Retry-After headers when debugging persistent 429 errors, as OmniRoute incorporates these into cooldown calculations.

Frequently Asked Questions

How do I know if a 429 error is from Connection Cooldown or Model Lockout?

Query /api/resilience/connections for the specific connection ID. If rateLimitedUntil is present, the 429 triggered a Connection Cooldown. If the connection shows healthy but requests still fail for a specific model, check /api/resilience/model-cooldowns for a Model Lockout entry on that model tuple.

Why is my connection stuck in "unavailable" status after the cooldown period expires?

The testStatus remains "unavailable" until a successful request clears the error via clearAccountError(). If automatic recovery fails, manually reset the connection using POST /api/resilience/reset with the provider and connection ID.

Can I adjust the exponential back-off duration for model lockouts?

Yes. Modify the maxCooldownMs and base timing values in src/lib/resilience/modelLockoutSettings.ts, or adjust per-model thresholds in the dashboard under Settings → Model Lockout. The back-off calculation follows baseCooldownMs * 2 ** failureIndex pattern defined in [accountFallback.ts](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.50/open-sse/services/accountFallback.ts#L73-L79).

Where does OmniRoute store the Model Lockout state?

Model lockouts are stored in an in-memory Map<string, ModelLockoutEntry> within the lockModel() function implementation. This state is not persisted to disk, meaning lockouts clear on server restart, though they typically expire via timer or success-decay before that becomes necessary.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}