How OmniRoute's Connection Cooldown Mechanism Prevents Cascade Failures

OmniRoute implements a per-connection cooldown system that temporarily disables failing provider accounts using exponential backoff, storing a rateLimitedUntil timestamp in the provider_connections SQLite table and checking it before routing each request.

The open-source AI request router diegosouzapw/OmniRoute protects upstream providers from traffic hammering through a sophisticated connection cooldown mechanism. When transient errors occur—such as gateway timeouts or rate limits—the system automatically applies time-based penalties to specific credentials. This ensures that flaky accounts are temporarily sidelined while healthy connections continue serving traffic, preventing cascade failures across the request pipeline.

How the Cooldown Mechanism Works

At the core of OmniRoute's resilience strategy is a per-connection cooldown state tracked in the SQLite database. Each provider connection row maintains a rateLimitedUntil timestamp and a backoffLevel counter. When the error classifier detects a transient failure, the system invokes markAccountUnavailable in src/sse/services/auth.ts (around line 2437) to compute a future timestamp when the connection can be reused.

The cooldown follows an exponential backoff formula: baseCooldown × 2^backoffLevel. The backoffLevel increments with each consecutive failure and resets only after a successful request completes. This creates progressively longer exclusion windows—starting at 3–5 seconds and potentially extending to the configured maxCooldownMs cap defined in src/lib/resilience/settings.ts.

Error Classification and Cooldown Triggers

Identifying Retryable Errors

Before applying a cooldown, OmniRoute classifies the upstream error to determine if it represents a temporary condition. The classifyProviderError function in src/open-sse/services/errorClassifier.ts flags specific HTTP status codes and scenarios as retryable:

• 502, 503, 504: Gateway and service unavailable errors
• 429 without Retry-After header: Rate limited without explicit guidance from upstream
• 404 model-lockout: Local model unavailability signals

Only errors matching these criteria trigger the cooldown mechanism, distinguishing transient infrastructure issues from permanent authentication failures or invalid requests.

Recording the Cooldown Timestamp

When an error qualifies as retryable, the router calls markAccountUnavailable in src/sse/services/auth.ts. This function updates the connection record via updateProviderConnection in src/lib/db/providers.ts, setting the rateLimitedUntil field to a calculated future time:

// src/sse/services/auth.ts – marking a connection unavailable
await auth.markAccountUnavailable(
  connection.id,            // connection ID
  502,                      // HTTP status code
  "Bad gateway",            // error message
  "openai",                 // provider name
  "gpt-4o-mini"            // optional model
);

Simultaneously, the function increments the backoffLevel field in the database, ensuring the next failure triggers a longer exclusion period.

Exponential Backoff Calculation

Base Cooldown Values

OmniRoute defines base cooldown durations in src/open-sse/config/constants.ts through the COOLDOWN_MS constant. The system applies different base values depending on authentication type:

• OAuth providers: 5 seconds base cooldown
• API-key providers: 3 seconds base cooldown

The Backoff Formula

The actual cooldown duration applies exponential scaling based on the connection's current backoffLevel (n):

cooldownDuration = COOLDOWN_MS × 2^n

For an OAuth connection experiencing consecutive failures, this produces sequences such as 5s → 10s → 20s → 40s, capped at the maxCooldownMs value specified in resilience settings. This aggressive backoff prevents repeatedly hammering endpoints that are clearly experiencing distress while allowing automatic recovery attempts at increasing intervals.

Filtering Cooled-Down Connections

Before routing a request, OmniRoute checks credential availability through getProviderCredentials in src/sse/services/auth.ts (approximately lines 2260–2270). This function queries the rateLimitedUntil timestamp and compares it against the current system time:

// src/sse/services/auth.ts – connection-cooldown check
if (new Date(connection.rateLimitedUntil).getTime() > Date.now()) {
  // skip this connection, try another one
}

If the timestamp lies in the future, the connection is excluded from the routing pool. This check occurs atomically during credential selection, ensuring that cooled-down accounts never receive new requests while maintaining zero-downtime operation for healthy connections.

Recovery and Cleanup

Resetting After Success

When a request eventually succeeds through a previously failing connection, OmniRoute immediately clears the penalty state. The resetConnectionBackoff function in src/lib/db/providers.ts removes the rateLimitedUntil timestamp and resets backoffLevel to 0:

await resetConnectionBackoff(conn.id); // clears rateLimitedUntil and backoffLevel

This "lazy-recovery" approach ensures that connections re-enter the pool the moment they demonstrate health, without waiting for the full cooldown period to expire.

Startup Cleanup

To handle stale cooldown entries from unclean shutdowns, OmniRoute includes a cleanup routine in src/instrumentation-node.ts (around line 307). During process startup, this code identifies and removes transient cooldowns that have expired while the system was offline, preventing permanently stuck connections after crashes or restarts.

Configuration and Tuning

Operators can adjust cooldown behavior through src/lib/resilience/settings.ts, which exposes maxCooldownMs to cap exponential growth. This prevents individual connections from being excluded indefinitely during prolonged outages while maintaining sufficient backoff to protect upstream providers.

Summary

• Error Detection: OmniRoute uses classifyProviderError in src/open-sse/services/errorClassifier.ts to identify transient failures (502/503/504, 429 without Retry-After, 404 lockouts).
• Cooldown Recording: The markAccountUnavailable function in src/sse/services/auth.ts sets rateLimitedUntil using exponential backoff based on COOLDOWN_MS from src/open-sse/config/constants.ts.
• Request Filtering: getProviderCredentials checks rateLimitedUntil during routing to skip cooled-down connections.
• Automatic Recovery: Successful requests trigger resetConnectionBackoff in src/lib/db/providers.ts to immediately restore connection eligibility.
• Startup Safety: src/instrumentation-node.ts cleans up stale cooldowns on process initialization.

Frequently Asked Questions

What errors trigger a connection cooldown in OmniRoute?

OmniRoute applies cooldowns to transient infrastructure errors classified by classifyProviderError in src/open-sse/services/errorClassifier.ts. These include HTTP 502, 503, and 504 gateway errors; HTTP 429 rate-limit responses lacking a Retry-After header; and local 404 model-lockout conditions. Permanent failures like authentication errors do not trigger cooldowns.

How long does the OmniRoute connection cooldown last?

The cooldown duration follows an exponential backoff pattern starting at 3 seconds for API-key providers or 5 seconds for OAuth providers (defined in src/open-sse/config/constants.ts), multiplied by 2 raised to the power of the current backoffLevel. Each consecutive failure doubles the wait time until reaching the maxCooldownMs cap configured in src/lib/resilience/settings.ts.

How does OmniRoute recover from a cooldown state?

Recovery occurs through lazy evaluation. When getProviderCredentials in src/sse/services/auth.ts detects that rateLimitedUntil has passed, it considers the connection available again. Additionally, upon any successful request, resetConnectionBackoff in src/lib/db/providers.ts immediately clears the timestamp and resets the backoff counter to zero, restoring full eligibility without waiting for the cooldown period to expire.

Where is the cooldown state stored in OmniRoute?

OmniRoute persists cooldown data in the SQLite provider_connections table. Each connection row maintains a rateLimitedUntil datetime field and an integer backoffLevel counter. This durable storage allows the cooldown mechanism to survive process restarts, with cleanup handled during startup via src/instrumentation-node.ts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}