# What Is Connection Cooldown in OmniRoute? A Deep Dive into Rate Limiting

> Discover OmniRoute connection cooldown, its rate limiting strategy using exponential back-off and Retry-After hints to manage provider connections after errors.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: deep-dive
- Published: 2026-08-13

---

**Connection cooldown is OmniRoute's per-connection back-off mechanism that temporarily suspends individual provider connections after HTTP 429 or 5xx errors, using exponential back-off with optional upstream Retry-After hints.**

In high-traffic AI proxy deployments, rate limiting can cripple throughput if handled poorly. OmniRoute solves this with **connection cooldown** — a precise, database-persisted throttling system that isolates failing connections without marking entire providers as dead. This article explains exactly how the mechanism works, where it's implemented in the codebase, and how to configure it for your deployment.

---

## The Core Concept: Per-Connection Isolation

OmniRoute treats rate limit errors as **transient, connection-specific problems**. Rather than global failure, the system:

1. Calculates a cooldown deadline for the exact connection that errored
2. Persists that deadline to `provider_connections.rate_limited_until` in SQLite
3. Excludes the connection from selection until the deadline passes
4. Returns a unified "all rate limited" response when no connections remain available

This design lets healthy connections from the same provider continue serving requests while the throttled connection recovers.

---

## How Cooldown Duration Is Calculated

The calculation logic lives in [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts) and follows a configurable exponential back-off policy.

### Base Configuration

Default values are defined in [`src/lib/resilience/settings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/settings.ts) (lines 64-74):

```typescript
connectionCooldown: {
  oauth: { 
    baseCooldownMs: PROVIDER_PROFILES.oauth.transientCooldown, 
    useUpstreamRetryHints: PROVIDER_PROFILES.oauth.rateLimitCooldown === 0, 
    maxBackoffSteps: PROVIDER_PROFILES.oauth.maxBackoffLevel 
  },
  apikey: { 
    baseCooldownMs: PROVIDER_PROFILES.apikey.transientCooldown, 
    useUpstreamRetryHints: PROVIDER_PROFILES.apikey.rateLimitCooldown === 0, 
    maxBackoffSteps: PROVIDER_PROFILES.apikey.maxBackoffLevel 
  },
},

```

**Key parameters:**

- **baseCooldownMs**: Starting wait time before retry
- **useUpstreamRetryHints**: Whether to respect upstream `Retry-After` headers
- **maxBackoffSteps`: Ceiling for exponential growth (2^n multiplier)

### The Back-Off Formula

When `calculateBackoffCooldown` triggers (referenced at [`accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/accountFallback.ts) lines 177-185), the effective cooldown is:

```

effectiveMs = baseCooldownMs * (2 ** backoffLevel)

```

If `useUpstreamRetryHints` is true and the upstream returns a `Retry-After` header (or JSON body hint), that value **overrides** the calculated duration — capped at `MAX_SHORT_RETRY_HINT_MS` to prevent abuse.

---

## Persisting and Enforcing Cooldown

### Writing the Deadline to Database

When an error qualifies for cooldown, [`accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/accountFallback.ts) (lines 2005-2024) persists the state:

```typescript
// Persist the cooldown …
if (typeof connId === "string" && effectiveCooldownMs > 0 && nextState.rateLimitedUntil) {
  const untilMs = cooldownUntilMs(nextState.rateLimitedUntil);
  if (Number.isFinite(untilMs) && untilMs > Date.now()) {
    setConnectionRateLimitUntil(connId, untilMs); // ← DB write
  }
}

```

This ensures **survival across requests** — the cooldown survives even if the server process restarts.

### Filtering During Connection Selection

The auth layer in [`src/sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/auth.ts) (lines 1337-1380) enforces cooldown exclusion:

```typescript
const connectionCooldownMs = parseFutureDateMs(connection.rateLimitedUntil);
…
if (state.connectionCooldownMs !== null) { 
  candidates.push({ ms: state.connectionCooldownMs, connection: state.connection }); 
}
…
if (earliest) { 
  return { allRateLimited: true, retryAfter: earliest, … }; 
}

```

The `isAccountUnavailable(connection.rateLimitedUntil)` helper automatically clears expired cooldowns on read — **lazy recovery** without background jobs.

---

## Handling Total Provider Cooldown

When **every** connection for a provider is cooling down, OmniRoute returns a structured response (lines 1393-1406):

```typescript
const authResult = await selectConnection(provider, options);
if (authResult?.allRateLimited) {
  res.status(429).json({
    error: "rate_limited",
    retry_after: authResult.retryAfter,
    message: authResult.lastError ?? "All accounts are cooling down",
  });
  return;
}

```

This gives callers **actionable retry timing** rather than opaque failures.

---

## Practical Code Examples

### Manually Trigger Cooldown

```typescript
import { setConnectionRateLimitUntil } from "@/lib/db/providers";

async function markCooldown(connId: string, seconds: number) {
  const untilMs = Date.now() + seconds * 1_000;
  await setConnectionRateLimitUntil(connId, untilMs);
}

```

Mirrors internal usage in [`accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/accountFallback.ts).

### Check Connection Availability

```typescript
import { isAccountUnavailable } from "@omniroute/open-sse/services/accountFallback";

function canUse(connection) {
  return !isAccountUnavailable(connection.rateLimitedUntil);
}

```

Used throughout credential selection logic.

---

## Files Defining Connection Cooldown

| File | Purpose |
|------|---------|
| [`src/lib/resilience/settings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/settings.ts) | Default cooldown configuration |
| [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts) | Error classification, back-off calculation, persistence |
| [`src/sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/auth.ts) | Connection filtering and "all rate limited" response |
| [`src/lib/db/providers.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/db/providers.ts) | Database helpers for `rate_limited_until` updates |

---

## Summary

- **Connection cooldown** isolates rate-limited connections without killing provider health
- **Exponential back-off** scales from `baseCooldownMs` with capped doubling steps
- **Upstream hints** override calculations when `useUpstreamRetryHints` is enabled
- **SQLite persistence** survives process restarts via `provider_connections.rate_limited_until`
- **Lazy recovery** automatically re-enables connections when deadlines expire
- **Unified responses** expose `retryAfter` when all connections throttle simultaneously

---

## Frequently Asked Questions

### How does connection cooldown differ from circuit breaker pattern?

Circuit breakers stop all traffic to a failing dependency. OmniRoute's connection cooldown quarantines only the specific connection that errored, allowing sibling connections from the same provider to continue. This maximizes throughput during partial rate limiting.

### Can I disable upstream Retry-After hints entirely?

Yes. Set `rateLimitCooldown` to a non-zero value in your provider profile. In [`settings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/settings.ts), `useUpstreamRetryHints` evaluates to `PROVIDER_PROFILES.oauth.rateLimitCooldown === 0` — any non-zero value disables hint consumption and forces strict exponential back-off.

### What happens if the database write for cooldown fails?

The cooldown operates in-memory as a fallback. While less durable, the connection selection logic in [`auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/auth.ts) still filters based on `rateLimitedUntil` if present in the connection object, providing graceful degradation.

### Where is the maximum back-off level configured?

The `maxBackoffSteps` property in `connectionCooldown.oauth` and `connectionCooldown.apikey` (defined in [`src/lib/resilience/settings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/settings.ts)) caps the exponent. A value of 5 means maximum cooldown is `baseCooldownMs * 32`.