# How OmniRoute's Connection Cooldown Mechanism Prevents Cascade Failures

> Discover how OmniRoute's connection cooldown mechanism prevents cascade failures. Learn about its exponential backoff and rate limiting for robust provider connections.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: how-to-guide
- Published: 2026-08-21

---

**OmniRoute implements a per-connection cooldown system that temporarily disables failing provider accounts using exponential backoff, storing a `rateLimitedUntil` timestamp in the `provider_connections` SQLite table and checking it before routing each request.**

The open-source AI request router diegosouzapw/OmniRoute protects upstream providers from traffic hammering through a sophisticated connection cooldown mechanism. When transient errors occur—such as gateway timeouts or rate limits—the system automatically applies time-based penalties to specific credentials. This ensures that flaky accounts are temporarily sidelined while healthy connections continue serving traffic, preventing cascade failures across the request pipeline.

## How the Cooldown Mechanism Works

At the core of OmniRoute's resilience strategy is a **per-connection cooldown state** tracked in the SQLite database. Each provider connection row maintains a `rateLimitedUntil` timestamp and a `backoffLevel` counter. When the error classifier detects a transient failure, the system invokes `markAccountUnavailable` in [`src/sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/auth.ts) (around line 2437) to compute a future timestamp when the connection can be reused.

The cooldown follows an exponential backoff formula: `baseCooldown × 2^backoffLevel`. The `backoffLevel` increments with each consecutive failure and resets only after a successful request completes. This creates progressively longer exclusion windows—starting at 3–5 seconds and potentially extending to the configured `maxCooldownMs` cap defined in [`src/lib/resilience/settings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/settings.ts).

## Error Classification and Cooldown Triggers

### Identifying Retryable Errors

Before applying a cooldown, OmniRoute classifies the upstream error to determine if it represents a temporary condition. The `classifyProviderError` function in [`src/open-sse/services/errorClassifier.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/open-sse/services/errorClassifier.ts) flags specific HTTP status codes and scenarios as retryable:

- **502, 503, 504**: Gateway and service unavailable errors
- **429 without Retry-After header**: Rate limited without explicit guidance from upstream
- **404 model-lockout**: Local model unavailability signals

Only errors matching these criteria trigger the cooldown mechanism, distinguishing transient infrastructure issues from permanent authentication failures or invalid requests.

### Recording the Cooldown Timestamp

When an error qualifies as retryable, the router calls `markAccountUnavailable` in [`src/sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/auth.ts). This function updates the connection record via `updateProviderConnection` in [`src/lib/db/providers.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/db/providers.ts), setting the `rateLimitedUntil` field to a calculated future time:

```ts
// src/sse/services/auth.ts – marking a connection unavailable
await auth.markAccountUnavailable(
  connection.id,            // connection ID
  502,                      // HTTP status code
  "Bad gateway",            // error message
  "openai",                 // provider name
  "gpt-4o-mini"            // optional model
);

```

Simultaneously, the function increments the `backoffLevel` field in the database, ensuring the next failure triggers a longer exclusion period.

## Exponential Backoff Calculation

### Base Cooldown Values

OmniRoute defines base cooldown durations in [`src/open-sse/config/constants.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/open-sse/config/constants.ts) through the `COOLDOWN_MS` constant. The system applies different base values depending on authentication type:

- **OAuth providers**: 5 seconds base cooldown
- **API-key providers**: 3 seconds base cooldown

### The Backoff Formula

The actual cooldown duration applies exponential scaling based on the connection's current `backoffLevel` (n):

```

cooldownDuration = COOLDOWN_MS × 2^n

```

For an OAuth connection experiencing consecutive failures, this produces sequences such as 5s → 10s → 20s → 40s, capped at the `maxCooldownMs` value specified in resilience settings. This aggressive backoff prevents repeatedly hammering endpoints that are clearly experiencing distress while allowing automatic recovery attempts at increasing intervals.

## Filtering Cooled-Down Connections

Before routing a request, OmniRoute checks credential availability through `getProviderCredentials` in [`src/sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/auth.ts) (approximately lines 2260–2270). This function queries the `rateLimitedUntil` timestamp and compares it against the current system time:

```ts
// src/sse/services/auth.ts – connection-cooldown check
if (new Date(connection.rateLimitedUntil).getTime() > Date.now()) {
  // skip this connection, try another one
}

```

If the timestamp lies in the future, the connection is excluded from the routing pool. This check occurs atomically during credential selection, ensuring that cooled-down accounts never receive new requests while maintaining zero-downtime operation for healthy connections.

## Recovery and Cleanup

### Resetting After Success

When a request eventually succeeds through a previously failing connection, OmniRoute immediately clears the penalty state. The `resetConnectionBackoff` function in [`src/lib/db/providers.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/db/providers.ts) removes the `rateLimitedUntil` timestamp and resets `backoffLevel` to 0:

```ts
await resetConnectionBackoff(conn.id); // clears rateLimitedUntil and backoffLevel

```

This "lazy-recovery" approach ensures that connections re-enter the pool the moment they demonstrate health, without waiting for the full cooldown period to expire.

### Startup Cleanup

To handle stale cooldown entries from unclean shutdowns, OmniRoute includes a cleanup routine in [`src/instrumentation-node.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/instrumentation-node.ts) (around line 307). During process startup, this code identifies and removes transient cooldowns that have expired while the system was offline, preventing permanently stuck connections after crashes or restarts.

## Configuration and Tuning

Operators can adjust cooldown behavior through [`src/lib/resilience/settings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/settings.ts), which exposes `maxCooldownMs` to cap exponential growth. This prevents individual connections from being excluded indefinitely during prolonged outages while maintaining sufficient backoff to protect upstream providers.

## Summary

- **Error Detection**: OmniRoute uses `classifyProviderError` in [`src/open-sse/services/errorClassifier.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/open-sse/services/errorClassifier.ts) to identify transient failures (502/503/504, 429 without Retry-After, 404 lockouts).
- **Cooldown Recording**: The `markAccountUnavailable` function in [`src/sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/auth.ts) sets `rateLimitedUntil` using exponential backoff based on `COOLDOWN_MS` from [`src/open-sse/config/constants.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/open-sse/config/constants.ts).
- **Request Filtering**: `getProviderCredentials` checks `rateLimitedUntil` during routing to skip cooled-down connections.
- **Automatic Recovery**: Successful requests trigger `resetConnectionBackoff` in [`src/lib/db/providers.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/db/providers.ts) to immediately restore connection eligibility.
- **Startup Safety**: [`src/instrumentation-node.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/instrumentation-node.ts) cleans up stale cooldowns on process initialization.

## Frequently Asked Questions

### What errors trigger a connection cooldown in OmniRoute?

OmniRoute applies cooldowns to transient infrastructure errors classified by `classifyProviderError` in [`src/open-sse/services/errorClassifier.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/open-sse/services/errorClassifier.ts). These include HTTP 502, 503, and 504 gateway errors; HTTP 429 rate-limit responses lacking a `Retry-After` header; and local 404 model-lockout conditions. Permanent failures like authentication errors do not trigger cooldowns.

### How long does the OmniRoute connection cooldown last?

The cooldown duration follows an exponential backoff pattern starting at 3 seconds for API-key providers or 5 seconds for OAuth providers (defined in [`src/open-sse/config/constants.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/open-sse/config/constants.ts)), multiplied by 2 raised to the power of the current `backoffLevel`. Each consecutive failure doubles the wait time until reaching the `maxCooldownMs` cap configured in [`src/lib/resilience/settings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/settings.ts).

### How does OmniRoute recover from a cooldown state?

Recovery occurs through lazy evaluation. When `getProviderCredentials` in [`src/sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/auth.ts) detects that `rateLimitedUntil` has passed, it considers the connection available again. Additionally, upon any successful request, `resetConnectionBackoff` in [`src/lib/db/providers.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/db/providers.ts) immediately clears the timestamp and resets the backoff counter to zero, restoring full eligibility without waiting for the cooldown period to expire.

### Where is the cooldown state stored in OmniRoute?

OmniRoute persists cooldown data in the SQLite `provider_connections` table. Each connection row maintains a `rateLimitedUntil` datetime field and an integer `backoffLevel` counter. This durable storage allows the cooldown mechanism to survive process restarts, with cleanup handled during startup via [`src/instrumentation-node.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/instrumentation-node.ts).