What Is Connection Cooldown in OmniRoute? A Deep Dive into Rate Limiting

Connection cooldown is OmniRoute's per-connection back-off mechanism that temporarily suspends individual provider connections after HTTP 429 or 5xx errors, using exponential back-off with optional upstream Retry-After hints.

In high-traffic AI proxy deployments, rate limiting can cripple throughput if handled poorly. OmniRoute solves this with connection cooldown — a precise, database-persisted throttling system that isolates failing connections without marking entire providers as dead. This article explains exactly how the mechanism works, where it's implemented in the codebase, and how to configure it for your deployment.

---

The Core Concept: Per-Connection Isolation

OmniRoute treats rate limit errors as transient, connection-specific problems. Rather than global failure, the system:

1. Calculates a cooldown deadline for the exact connection that errored
2. Persists that deadline to provider_connections.rate_limited_until in SQLite
3. Excludes the connection from selection until the deadline passes
4. Returns a unified "all rate limited" response when no connections remain available

This design lets healthy connections from the same provider continue serving requests while the throttled connection recovers.

---

How Cooldown Duration Is Calculated

The calculation logic lives in open-sse/services/accountFallback.ts and follows a configurable exponential back-off policy.

Base Configuration

Default values are defined in src/lib/resilience/settings.ts (lines 64-74):

connectionCooldown: {
  oauth: { 
    baseCooldownMs: PROVIDER_PROFILES.oauth.transientCooldown, 
    useUpstreamRetryHints: PROVIDER_PROFILES.oauth.rateLimitCooldown === 0, 
    maxBackoffSteps: PROVIDER_PROFILES.oauth.maxBackoffLevel 
  },
  apikey: { 
    baseCooldownMs: PROVIDER_PROFILES.apikey.transientCooldown, 
    useUpstreamRetryHints: PROVIDER_PROFILES.apikey.rateLimitCooldown === 0, 
    maxBackoffSteps: PROVIDER_PROFILES.apikey.maxBackoffLevel 
  },
},

Key parameters:

• baseCooldownMs: Starting wait time before retry
• useUpstreamRetryHints: Whether to respect upstream Retry-After headers
• **maxBackoffSteps`: Ceiling for exponential growth (2^n multiplier)

The Back-Off Formula

When calculateBackoffCooldown triggers (referenced at accountFallback.ts lines 177-185), the effective cooldown is:

effectiveMs = baseCooldownMs * (2 ** backoffLevel)

If useUpstreamRetryHints is true and the upstream returns a Retry-After header (or JSON body hint), that value overrides the calculated duration — capped at MAX_SHORT_RETRY_HINT_MS to prevent abuse.

---

Persisting and Enforcing Cooldown

Writing the Deadline to Database

When an error qualifies for cooldown, accountFallback.ts (lines 2005-2024) persists the state:

// Persist the cooldown …
if (typeof connId === "string" && effectiveCooldownMs > 0 && nextState.rateLimitedUntil) {
  const untilMs = cooldownUntilMs(nextState.rateLimitedUntil);
  if (Number.isFinite(untilMs) && untilMs > Date.now()) {
    setConnectionRateLimitUntil(connId, untilMs); // ← DB write
  }
}

This ensures survival across requests — the cooldown survives even if the server process restarts.

Filtering During Connection Selection

The auth layer in src/sse/services/auth.ts (lines 1337-1380) enforces cooldown exclusion:

const connectionCooldownMs = parseFutureDateMs(connection.rateLimitedUntil);
…
if (state.connectionCooldownMs !== null) { 
  candidates.push({ ms: state.connectionCooldownMs, connection: state.connection }); 
}
…
if (earliest) { 
  return { allRateLimited: true, retryAfter: earliest, … }; 
}

The isAccountUnavailable(connection.rateLimitedUntil) helper automatically clears expired cooldowns on read — lazy recovery without background jobs.

---

Handling Total Provider Cooldown

When every connection for a provider is cooling down, OmniRoute returns a structured response (lines 1393-1406):

const authResult = await selectConnection(provider, options);
if (authResult?.allRateLimited) {
  res.status(429).json({
    error: "rate_limited",
    retry_after: authResult.retryAfter,
    message: authResult.lastError ?? "All accounts are cooling down",
  });
  return;
}

This gives callers actionable retry timing rather than opaque failures.

---

Practical Code Examples

Manually Trigger Cooldown

import { setConnectionRateLimitUntil } from "@/lib/db/providers";

async function markCooldown(connId: string, seconds: number) {
  const untilMs = Date.now() + seconds * 1_000;
  await setConnectionRateLimitUntil(connId, untilMs);
}

Mirrors internal usage in accountFallback.ts.

Check Connection Availability

import { isAccountUnavailable } from "@omniroute/open-sse/services/accountFallback";

function canUse(connection) {
  return !isAccountUnavailable(connection.rateLimitedUntil);
}

Used throughout credential selection logic.

---

Files Defining Connection Cooldown

File	Purpose
src/lib/resilience/settings.ts	Default cooldown configuration
open-sse/services/accountFallback.ts	Error classification, back-off calculation, persistence
src/sse/services/auth.ts	Connection filtering and "all rate limited" response
src/lib/db/providers.ts	Database helpers for rate_limited_until updates

---

Summary

• Connection cooldown isolates rate-limited connections without killing provider health
• Exponential back-off scales from baseCooldownMs with capped doubling steps
• Upstream hints override calculations when useUpstreamRetryHints is enabled
• SQLite persistence survives process restarts via provider_connections.rate_limited_until
• Lazy recovery automatically re-enables connections when deadlines expire
• Unified responses expose retryAfter when all connections throttle simultaneously

---

Frequently Asked Questions

How does connection cooldown differ from circuit breaker pattern?

Circuit breakers stop all traffic to a failing dependency. OmniRoute's connection cooldown quarantines only the specific connection that errored, allowing sibling connections from the same provider to continue. This maximizes throughput during partial rate limiting.

Can I disable upstream Retry-After hints entirely?

Yes. Set rateLimitCooldown to a non-zero value in your provider profile. In settings.ts, useUpstreamRetryHints evaluates to PROVIDER_PROFILES.oauth.rateLimitCooldown === 0 — any non-zero value disables hint consumption and forces strict exponential back-off.

What happens if the database write for cooldown fails?

The cooldown operates in-memory as a fallback. While less durable, the connection selection logic in auth.ts still filters based on rateLimitedUntil if present in the connection object, providing graceful degradation.

Where is the maximum back-off level configured?

The maxBackoffSteps property in connectionCooldown.oauth and connectionCooldown.apikey (defined in src/lib/resilience/settings.ts) caps the exponent. A value of 5 means maximum cooldown is baseCooldownMs * 32.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}