# How OmniRoute Connection Cooldowns Work: A Technical Deep-Dive

> Explore OmniRoute connection cooldowns and their per-account exponential backoff. Learn how OmniRoute isolates failing connections while keeping healthy ones active for the same provider.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: deep-dive
- Published: 2026-08-26

---

**OmniRoute connection cooldowns implement per-account exponential backoff that automatically isolates failing OAuth or API-key connections while keeping other healthy connections active for the same provider.**

OmniRoute's routing engine treats each authenticated connection—whether an OAuth account or an API-key account—as an independently recoverable resource. When a specific connection encounters rate limits, quota exhaustion, or service errors, the system calculates a cooldown window and marks that single connection unavailable. This design prevents one failing key from degrading traffic for an entire provider. The following sections trace the complete lifecycle from error detection through automatic recovery, citing the actual source implementation in `diegosouzapw/OmniRoute`.

## Error Detection and Cooldown Triggering

The entry point for all connection cooldowns is `markAccountUnavailable()` in [`src/sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/auth.ts). This function receives the HTTP response status, an optional `Retry-After` header, the provider name, and the specific model that failed.

The function performs three critical decisions:

- **Connection-wide cooldown** — For transient errors like `429 Too Many Requests` or `503 Service Unavailable`, it initiates a timed backoff on the connection itself
- **Model-only lockout** — For errors like `404 Model Not Found`, it marks just that model as unavailable for this connection without triggering a full cooldown
- **Permanent disablement** — For authentication failures or other hard errors, it may permanently deactivate the connection

This granularity ensures that a rate limit on `gpt-4o-mini` does not prevent the same OpenAI connection from serving `gpt-3.5-turbo` requests.

## Calculating the Cooldown Duration

The backoff calculation lives in [`open-sse/services/accountFallback/cooldownCap.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback/cooldownCap.ts). The `getCooldownMs()` function implements **exponential backoff** using provider-specific base values:

```typescript
// From open-sse/services/accountFallback/cooldownCap.ts
// Base cooldowns defined per credential type
const baseCooldowns = {
  OAuth: 5_000,      // 5 seconds
  apiKey: 3_000,     // 3 seconds
};

// Exponential calculation: base * 2 ** failureIndex
const cooldownMs = baseCooldownMs * Math.pow(2, connection.backoffLevel);

```

The `backoffLevel` (failure index) increments with each consecutive failure on the same connection and resets only after a successful request. This means:

- First failure: 3–5 seconds
- Second failure: 6–10 seconds
- Third failure: 12–20 seconds
- And so on...

Operators can override these defaults through environment variables like `OMNIROUTE_OPENAI_COOLDOWN_MS` as documented in [`docs/reference/FEATURE_FLAGS.md`](https://github.com/diegosouzapw/OmniRoute/blob/main/docs/reference/FEATURE_FLAGS.md).

## Persisting and Enforcing Cooldown State

Once calculated, the cooldown expiry timestamp is written to two locations:

1. **SQLite database** — The `rateLimitedUntil` field in the `connections` table (schema defined in `src/lib/db/migrations/*`)
2. **In-memory object** — `connection.rateLimitedUntil` on the active connection instance

This dual storage ensures cooldowns survive process restarts. The startup routine `clearStaleCrashCooldowns` in [`src/sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/auth.ts) performs a lazy cleanup: it removes only expired timestamps while preserving future ones, verified by [`tests/unit/startup-stale-cooldown-recovery.test.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/tests/unit/startup-stale-cooldown-recovery.test.ts).

When the router builds candidate lists, the filtering occurs in `getProviderCredentials()`:

```typescript
// From src/sse/services/auth.ts — automatic filtering logic
const activeConnections = connections.filter(
  conn => !conn.rateLimitedUntil || new Date(conn.rateLimitedUntil) <= new Date()
);

```

The combo runner in [`open-sse/services/combo.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/combo.ts) calls `resolveComboTargets()`, which relies on this filtering to exclude cooled-down connections from rotation.

## Recovery and Anti-Thundering-Herd Protection

OmniRoute uses **lazy recovery**—no background timers or polling. When a new request arrives, the timestamp comparison happens inline. If `rateLimitedUntil` has passed, the connection immediately re-enters the candidate pool.

To prevent race conditions during concurrent error handling, `markAccountUnavailable()` contains a critical guard: it **never shortens an existing numeric-epoch cooldown**. This prevents two simultaneous 429 responses from resetting each other's backoff calculations. The test suite validates this in [`tests/unit/mark-account-unavailable-numeric-epoch-guard.test.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/tests/unit/mark-account-unavailable-numeric-epoch-guard.test.ts).

## Practical Code Examples

### Manually Triggering a Cooldown (Testing or Custom Logic)

```typescript
import { markAccountUnavailable } from "@/sse/services/auth";

await markAccountUnavailable(
  connectionId,          // UUID from connections table
  429,                  // HTTP status that triggered cooldown
  "rate limit hit",     // Descriptive error message
  "openai",             // Provider identifier
  "gpt-4o-mini"         // Optional: specific model affected
);

```

### Inspecting Connection Cooldown State

```typescript
import { getDbInstance } from "@/lib/db/core";

const db = await getDbInstance();
const conn = await db.getConnectionById(connectionId);

console.log({
  isCoolingDown: conn.rateLimitedUntil && new Date(conn.rateLimitedUntil) > new Date(),
  cooldownExpires: conn.rateLimitedUntil,
  currentBackoffLevel: conn.backoffLevel,
});

```

### Observing Router Behavior

```typescript
import { resolveComboTargets } from "@/open-sse/services/combo";

const candidates = await resolveComboTargets({
  provider: "openai",
  model: "gpt-4o-mini",
});

// candidates excludes any connection where rateLimitedUntil is in the future
console.log(`Available connections: ${candidates.length}`);

```

## Key Implementation Files

| Path | Responsibility |
|------|----------------|
| [`src/sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/auth.ts) | `markAccountUnavailable()`, cooldown persistence, candidate filtering |
| [`open-sse/services/accountFallback/cooldownCap.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback/cooldownCap.ts) | `getCooldownMs()` exponential backoff calculation |
| [`src/sse/services/cooldownAwareRetry.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/cooldownAwareRetry.ts) | Helper for combo runner to respect short cooldowns before fallback |
| `src/lib/db/migrations/*` | Schema definitions for `rateLimitedUntil` column |
| [`tests/unit/startup-stale-cooldown-recovery.test.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/tests/unit/startup-stale-cooldown-recovery.test.ts) | Lazy recovery after process restart |
| [`tests/unit/mark-account-unavailable-numeric-epoch-guard.test.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/tests/unit/mark-account-unavailable-numeric-epoch-guard.test.ts) | Concurrent modification protection |
| [`tests/unit/combo-cooldown-retry.test.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/tests/unit/combo-cooldown-retry.test.ts) | End-to-end combo flow validation |

## Summary

- **Connection cooldowns in OmniRoute isolate individual failing accounts** without disabling entire providers, maintaining system throughput during partial outages
- **Detection** occurs in [`src/sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/auth.ts) via `markAccountUnavailable()`, which classifies errors and determines cooldown scope
- **Duration calculation** uses exponential backoff with provider-specific bases in [`open-sse/services/accountFallback/cooldownCap.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback/cooldownCap.ts)
- **Persistence** spans SQLite and in-memory storage, surviving restarts through lazy cleanup
- **Enforcement** happens at routing time through timestamp comparison in `getProviderCredentials()`
- **Recovery** is automatic and lazy—connections rejoin the pool once their expiry passes, with guards against thundering-herd race conditions

## Frequently Asked Questions

### How does OmniRoute handle cooldowns after a server restart?

Cooldowns persist in the SQLite `connections` table. On startup, `clearStaleCrashCooldowns()` in [`src/sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/auth.ts) scans all records and removes only expired `rateLimitedUntil` timestamps. Future timestamps remain intact, so legitimate ongoing cooldowns continue. The test in [`tests/unit/startup-stale-cooldown-recovery.test.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/tests/unit/startup-stale-cooldown-recovery.test.ts) verifies this behavior.

### What is the difference between connection cooldowns and model lockouts?

Connection cooldowns are time-based backoffs triggered by transient errors (`429`, `503`) and apply to the entire connection for all models. Model lockouts are status-based flags for specific errors like missing models (`404`) and only disable that single model on the connection. Both are handled in `markAccountUnavailable()` but write to different fields.

### Can I adjust cooldown durations without modifying source code?

Yes. Each provider profile in [`src/shared/constants/providers.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/constants/providers.ts) defines `baseCooldownMs` values that can be overridden via environment variables following the pattern `OMNIROUTE_{PROVIDER}_COOLDOWN_MS`. These are documented in [`docs/reference/FEATURE_FLAGS.md`](https://github.com/diegosouzapw/OmniRoute/blob/main/docs/reference/FEATURE_FLAGS.md).

### Why doesn't OmniRoute use a background timer for cooldown recovery?

The lazy recovery design eliminates complexity from timer management, race conditions, and cross-process synchronization. The timestamp comparison at request time is deterministic, stateless, and naturally handles process restarts without additional coordination logic.