How the Provider Cooldown Tracker Prevents 429 Rate Limit Errors in OmniRoute

OmniRoute's provider cooldown tracker prevents 429 errors by storing a rateLimitedUntil deadline for each provider and account that hits a rate limit, causing the router to skip that provider entirely until the cooldown expires.

The provider cooldown tracker in the open-source OmniRoute project (diegosouzapw/OmniRoute) is a lightweight, cross-request service designed to eliminate aggressive retry storms against upstream APIs. When a provider returns an HTTP 429 response, the tracker records a cooldown deadline that subsequent routing decisions consult before dispatching any new requests.

Core Mechanism of the Provider Cooldown Tracker

The tracker operates as a centralized state manager that makes unified decisions across all routing components. It exposes three primary functions for managing cooldown state: recordProviderCooldown, isProviderInCooldown, and getRemainingCooldownMs.

Recording Rate Limit Events

When a provider responds with a 429 status, the system immediately records the cooldown using the recordProviderCooldown function defined in open-sse/services/providerCooldownTracker.ts. This function accepts the provider name, account identifier, and the retryAfterMs duration to calculate the rateLimitedUntil timestamp.

// Record a 429 for a provider/account
import { recordProviderCooldown } from '@/open-sse/services/providerCooldownTracker';

recordProviderCooldown('openai', 'account-123', /* retryAfterMs */ 30_000);

This stores a deadline representing when the provider will become available again, protecting the upstream service from immediate subsequent requests.

Checking Cooldown State Before Requests

Before dispatching any request, the router consults isProviderInCooldown to determine if the deadline is still in the future. If active, the router retrieves the remaining wait time via getRemainingCooldownMs and treats the provider as unavailable.

// Later, before routing a request:
import { isProviderInCooldown, getRemainingCooldownMs } from '@/open-sse/services/providerCooldownTracker';

if (isProviderInCooldown('openai')) {
  const wait = getRemainingCooldownMs('openai'); // e.g. 28 000 ms
  // Skip this provider, try the next one in the combo
}

This short-circuit behavior ensures that requests never hit a provider that is known to be rate-limited, eliminating redundant 429 responses and maintaining low overall latency.

Integration with Combo Routing

The cooldown tracker integrates directly with OmniRoute's combo routing logic in open-sse/services/combo.ts. During combo execution, the system iterates through target providers and actively filters out any that are currently in cooldown.

// Combo routing snippet (simplified)
import { recordProviderCooldown, isProviderInCooldown } from '@/open-sse/services/providerCooldownTracker';

async function handleComboChat(combo, request) {
  for (const target of combo.targets) {
    if (isProviderInCooldown(target.provider)) continue;   // avoid 429‑prone provider
    const result = await dispatchToProvider(target, request);
    if (result.status === 429) {
      // Store cooldown and move on to the next target
      recordProviderCooldown(target.provider, target.account, result.retryAfterMs);
      continue;
    }
    return result; // success
  }
  throw new Error('All combo targets are in cooldown');
}

This integration ensures that the fallback chain automatically excludes rate-limited providers, allowing traffic to flow immediately to healthy alternatives without wasting time on doomed requests.

Housekeeping and Cleanup

The tracker performs automatic maintenance to prevent memory leaks and handle server restarts. The cleanupExpiredCooldownEntries function removes entries whose deadlines have passed, while clearStaleCrashCooldowns prunes stale cooldowns on startup to recover from unexpected crashes.

Additionally, the health monitoring system in open-sse/services/webSessionPoolHealth.ts uses the tracker to surface "cooldown" badges in the dashboard UI, giving operators real-time visibility into which providers are currently throttled.

Summary

• Centralized state: The provider cooldown tracker maintains rateLimitedUntil deadlines in open-sse/services/providerCooldownTracker.ts, offering a single source of truth for rate-limit status across the application.
• Short-circuit protection: Functions isProviderInCooldown and getRemainingCooldownMs prevent requests from reaching providers that will return 429 errors, reducing latency and protecting upstream quotas.
• Automatic fallback: The combo routing system in open-sse/services/combo.ts integrates the tracker to skip cooldown providers and immediately attempt the next target in the chain.
• Maintenance utilities: cleanupExpiredCooldownEntries and clearStaleCrashCooldowns ensure the cooldown map remains clean across application lifecycles.

Frequently Asked Questions

What happens when a provider returns a 429 error?

When a provider returns HTTP 429, the system calls recordProviderCooldown with the provider name, account ID, and retry-after duration. This stores a deadline timestamp after which the provider becomes eligible for new requests. Until that deadline passes, isProviderInCooldown returns true, causing the router to skip that provider entirely.

How does the cooldown tracker integrate with combo routing?

The combo routing logic in open-sse/services/combo.ts imports the cooldown API and checks each target via isProviderInCooldown before dispatching. If a target is in cooldown, the loop continues to the next target. If a request returns 429 during execution, the system records the cooldown immediately and continues to the next fallback target.

Where is the cooldown state stored?

The cooldown state is managed within the provider cooldown tracker service located at open-sse/services/providerCooldownTracker.ts. This cross-request service maintains the rateLimitedUntil timestamps in memory, making the state available to all routing components, health monitors, and UI renderers without requiring external database calls.

How does the system handle stale cooldown entries?

The tracker provides cleanupExpiredCooldownEntries to remove expired deadlines during runtime, and clearStaleCrashCooldowns to prune entries on startup. This ensures that temporary rate limits do not persist indefinitely after application restarts or crashes, maintaining accurate availability status for all providers.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}