How OmniRoute Connection Cooldowns Work: A Technical Deep-Dive

OmniRoute connection cooldowns implement per-account exponential backoff that automatically isolates failing OAuth or API-key connections while keeping other healthy connections active for the same provider.

OmniRoute's routing engine treats each authenticated connection—whether an OAuth account or an API-key account—as an independently recoverable resource. When a specific connection encounters rate limits, quota exhaustion, or service errors, the system calculates a cooldown window and marks that single connection unavailable. This design prevents one failing key from degrading traffic for an entire provider. The following sections trace the complete lifecycle from error detection through automatic recovery, citing the actual source implementation in diegosouzapw/OmniRoute.

Error Detection and Cooldown Triggering

The entry point for all connection cooldowns is markAccountUnavailable() in src/sse/services/auth.ts. This function receives the HTTP response status, an optional Retry-After header, the provider name, and the specific model that failed.

The function performs three critical decisions:

• Connection-wide cooldown — For transient errors like 429 Too Many Requests or 503 Service Unavailable, it initiates a timed backoff on the connection itself
• Model-only lockout — For errors like 404 Model Not Found, it marks just that model as unavailable for this connection without triggering a full cooldown
• Permanent disablement — For authentication failures or other hard errors, it may permanently deactivate the connection

This granularity ensures that a rate limit on gpt-4o-mini does not prevent the same OpenAI connection from serving gpt-3.5-turbo requests.

Calculating the Cooldown Duration

The backoff calculation lives in open-sse/services/accountFallback/cooldownCap.ts. The getCooldownMs() function implements exponential backoff using provider-specific base values:

// From open-sse/services/accountFallback/cooldownCap.ts
// Base cooldowns defined per credential type
const baseCooldowns = {
  OAuth: 5_000,      // 5 seconds
  apiKey: 3_000,     // 3 seconds
};

// Exponential calculation: base * 2 ** failureIndex
const cooldownMs = baseCooldownMs * Math.pow(2, connection.backoffLevel);

The backoffLevel (failure index) increments with each consecutive failure on the same connection and resets only after a successful request. This means:

• First failure: 3–5 seconds
• Second failure: 6–10 seconds
• Third failure: 12–20 seconds
• And so on...

Operators can override these defaults through environment variables like OMNIROUTE_OPENAI_COOLDOWN_MS as documented in docs/reference/FEATURE_FLAGS.md.

Persisting and Enforcing Cooldown State

Once calculated, the cooldown expiry timestamp is written to two locations:

1. SQLite database — The rateLimitedUntil field in the connections table (schema defined in src/lib/db/migrations/*)
2. In-memory object — connection.rateLimitedUntil on the active connection instance

This dual storage ensures cooldowns survive process restarts. The startup routine clearStaleCrashCooldowns in src/sse/services/auth.ts performs a lazy cleanup: it removes only expired timestamps while preserving future ones, verified by tests/unit/startup-stale-cooldown-recovery.test.ts.

When the router builds candidate lists, the filtering occurs in getProviderCredentials():

// From src/sse/services/auth.ts — automatic filtering logic
const activeConnections = connections.filter(
  conn => !conn.rateLimitedUntil || new Date(conn.rateLimitedUntil) <= new Date()
);

The combo runner in open-sse/services/combo.ts calls resolveComboTargets(), which relies on this filtering to exclude cooled-down connections from rotation.

Recovery and Anti-Thundering-Herd Protection

OmniRoute uses lazy recovery—no background timers or polling. When a new request arrives, the timestamp comparison happens inline. If rateLimitedUntil has passed, the connection immediately re-enters the candidate pool.

To prevent race conditions during concurrent error handling, markAccountUnavailable() contains a critical guard: it never shortens an existing numeric-epoch cooldown. This prevents two simultaneous 429 responses from resetting each other's backoff calculations. The test suite validates this in tests/unit/mark-account-unavailable-numeric-epoch-guard.test.ts.

Practical Code Examples

Manually Triggering a Cooldown (Testing or Custom Logic)

import { markAccountUnavailable } from "@/sse/services/auth";

await markAccountUnavailable(
  connectionId,          // UUID from connections table
  429,                  // HTTP status that triggered cooldown
  "rate limit hit",     // Descriptive error message
  "openai",             // Provider identifier
  "gpt-4o-mini"         // Optional: specific model affected
);

Inspecting Connection Cooldown State

import { getDbInstance } from "@/lib/db/core";

const db = await getDbInstance();
const conn = await db.getConnectionById(connectionId);

console.log({
  isCoolingDown: conn.rateLimitedUntil && new Date(conn.rateLimitedUntil) > new Date(),
  cooldownExpires: conn.rateLimitedUntil,
  currentBackoffLevel: conn.backoffLevel,
});

Observing Router Behavior

import { resolveComboTargets } from "@/open-sse/services/combo";

const candidates = await resolveComboTargets({
  provider: "openai",
  model: "gpt-4o-mini",
});

// candidates excludes any connection where rateLimitedUntil is in the future
console.log(`Available connections: ${candidates.length}`);

Key Implementation Files

Path	Responsibility
src/sse/services/auth.ts	markAccountUnavailable(), cooldown persistence, candidate filtering
open-sse/services/accountFallback/cooldownCap.ts	getCooldownMs() exponential backoff calculation
src/sse/services/cooldownAwareRetry.ts	Helper for combo runner to respect short cooldowns before fallback
src/lib/db/migrations/*	Schema definitions for rateLimitedUntil column
tests/unit/startup-stale-cooldown-recovery.test.ts	Lazy recovery after process restart
tests/unit/mark-account-unavailable-numeric-epoch-guard.test.ts	Concurrent modification protection
tests/unit/combo-cooldown-retry.test.ts	End-to-end combo flow validation

Summary

• Connection cooldowns in OmniRoute isolate individual failing accounts without disabling entire providers, maintaining system throughput during partial outages
• Detection occurs in src/sse/services/auth.ts via markAccountUnavailable(), which classifies errors and determines cooldown scope
• Duration calculation uses exponential backoff with provider-specific bases in open-sse/services/accountFallback/cooldownCap.ts
• Persistence spans SQLite and in-memory storage, surviving restarts through lazy cleanup
• Enforcement happens at routing time through timestamp comparison in getProviderCredentials()
• Recovery is automatic and lazy—connections rejoin the pool once their expiry passes, with guards against thundering-herd race conditions

Frequently Asked Questions

How does OmniRoute handle cooldowns after a server restart?

Cooldowns persist in the SQLite connections table. On startup, clearStaleCrashCooldowns() in src/sse/services/auth.ts scans all records and removes only expired rateLimitedUntil timestamps. Future timestamps remain intact, so legitimate ongoing cooldowns continue. The test in tests/unit/startup-stale-cooldown-recovery.test.ts verifies this behavior.

What is the difference between connection cooldowns and model lockouts?

Connection cooldowns are time-based backoffs triggered by transient errors (429, 503) and apply to the entire connection for all models. Model lockouts are status-based flags for specific errors like missing models (404) and only disable that single model on the connection. Both are handled in markAccountUnavailable() but write to different fields.

Can I adjust cooldown durations without modifying source code?

Yes. Each provider profile in src/shared/constants/providers.ts defines baseCooldownMs values that can be overridden via environment variables following the pattern OMNIROUTE_{PROVIDER}_COOLDOWN_MS. These are documented in docs/reference/FEATURE_FLAGS.md.

Why doesn't OmniRoute use a background timer for cooldown recovery?

The lazy recovery design eliminates complexity from timer management, race conditions, and cross-process synchronization. The timestamp comparison at request time is deterministic, stateless, and naturally handles process restarts without additional coordination logic.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}