How the OmniRoute Rate Limit Manager Enforces Per‑Provider and Per‑Key Limits

The OmniRoute rate limit manager implements adaptive throttling using the Bottleneck library to enforce granular per‑provider and per‑connection (API‑key) limits through a three-stage pipeline of initialization, slot acquisition, and response header learning.

OmniRoute is an open‑source AI request routing layer that prevents downstream provider overload while ensuring equitable resource distribution. The OmniRoute rate limit manager, implemented in open‑sse/services/rateLimitManager.ts, orchestrates this protection by dynamically adjusting request throughput based on real‑time provider feedback and configured policies.

Three-Stage Rate Limit Enforcement Pipeline

The enforcement system operates through three distinct stages that manage the lifecycle of rate limiting from startup through request completion.

Stage 1: Initialization and Limiter Creation

On application startup, the manager loads all persisted provider connections and determines which connections require protection. This process explicitly enables protection for API‑key providers or uses auto‑detection heuristics. For each protected connection, the system creates a dedicated Bottleneck limiter instance with default settings.

The initializeRateLimits() function triggers reconcileEnabledConnections() (source lines ≈ 74‑99) to build the initial limiter registry. Each limiter maintains independent state for request queuing, ensuring that congestion on one provider or API key does not affect others.

Stage 2: Slot Acquisition with withRateLimit

Every outbound request flows through withRateLimit() (source lines ≈ 545‑580), which implements the core throttling logic:

• Unprotected bypass: If rate limiting is disabled for the connection, the request executes immediately.
• Default application: Applies global resilience settings via buildLimiterDefaults() (source lines ≈ 154‑176).
• Override merging: Combines defaults with per‑connection overrides from provider_connections.rateLimitOverrides (RPM, minTime, maxConcurrent).
• Queue admission: Validates queue depth against maxQueueDepth via checkQueueAdmission() in open-sse/services/rateLimitManager/admission.ts (source lines ≈ 82‑90), rejecting requests early if the limiter is backlogged.
• Scheduled execution: Schedules the request on the limiter with a global maxWaitMs timeout to prevent indefinite stalls.

Stage 3: Adaptive Learning from Response Headers

After receiving a provider response, the manager parses standard rate‑limit headers (x‑ratelimit‑limit, x‑ratelimit‑remaining, retry‑after) through updateFromHeaders() (source lines ≈ 710‑860) and response bodies via updateFromResponseBody() (source lines ≈ 996‑1010). Header parsing logic utilizes the maps defined in open-sse/services/rateLimitManager/headers.ts (STANDARD_HEADERS and ANTHROPIC_HEADERS) to normalize provider‑specific formats.

The system then updates the Bottleneck limiter with:

• A computed minTime (inter‑request gap) derived from advertised RPM.
• Reservoir settings when remaining quota is low.
• Temporary pausing on HTTP 429 or "over‑limit" signals.

Learned values are cached in learnedLimits and persisted for future restarts.

Core Mechanisms for Per‑Provider and Per‑Key Isolation

Unique Key Generation for Isolation

The manager generates unique limiter keys to guarantee isolation between different providers and API keys. The getLimiterKey() function (source lines ≈ 81‑95) constructs identifiers using the provider ID and connection ID:

const key = `${provider}:${connectionId}`; // base case
// Model‑scoped keys for Codex, Gemini, etc.

For providers with per‑model quotas (such as Codex, Gemini, and Antigravity), the key optionally incorporates the model name, creating distinct limiters for each model tier.

Dynamic Defaults and Connection Overrides

Default limits originate from global resilience settings: requestsPerMinute, minTimeBetweenRequestsMs, and concurrentRequests. The buildLimiterDefaults() function merges these with per‑connection overrides stored in the database. Zero values in override fields are treated as "no override," allowing connections to effectively disable their own limits while retaining global protection. Fallback defaults for providers without explicit headers are defined in open-sse/services/providerDefaultRateLimit.ts.

Queue Admission Control and Early Rejection

Before scheduling, checkQueueAdmission() inspects the current queue depth via limiter.counts().QUEUED. If the depth exceeds maxQueueDepth, the manager throws a fast‑reject error, preventing expensive downstream work such as prompt compression or token estimation from executing on doomed requests.

Watchdog Recovery for Wedged Limiters

A background watchdog runs every 30 seconds via watchdogTick() (source lines ≈ 27‑84). If a limiter has queued jobs but no dispatches for over 120 seconds, the system considers it "wedged" and force‑resets it using stop({ dropWaitingJobs: true }). This prevents indefinite hangs when Bottleneck’s internal state becomes inconsistent.

Persistence of Learned Rate Limits

When providers supply usable rate‑limit metadata, recordLearnedLimit() (source lines ≈ 448‑468) debounces writes to the settings table. On subsequent startups, loadPersistedLimits() reapplies these learned values to the appropriate limiters, allowing the system to retain knowledge of provider constraints across deployments.

Rate Limit Manager Implementation Examples

Wrapping a Provider Request

Use withRateLimit() to ensure requests respect per‑connection limits:

import { withRateLimit } from "@/open-sse/services/rateLimitManager";
import { execute } from "@/open-sse/executors/default";

export async function handleChatCore(provider, connectionId, model, request, abortSignal) {
  return withRateLimit(
    provider,
    connectionId,
    model,
    () => execute(provider, connectionId, model, request, abortSignal),
    abortSignal
  );
}

Updating Limiter State from Headers

After each request, update the limiter based on provider feedback:

import { updateFromHeaders } from "@/open-sse/services/rateLimitManager";

const response = await fetch(url, fetchOpts);
await updateFromHeaders(
  providerId,
  connectionId,
  response.headers,
  response.status,
  modelName
);

Programmatic Protection Control

Enable or disable protection via the manager's API:

import {
  enableRateLimitProtection,
  disableRateLimitProtection,
  isRateLimitEnabled,
} from "@/open-sse/services/rateLimitManager";

if (userWantsProtection) enableRateLimitProtection(connectionId);
else disableRateLimitProtection(connectionId);

console.log(isRateLimitEnabled(connectionId)); // true / false

Summary

• The OmniRoute rate limit manager in open-sse/services/rateLimitManager.ts uses the Bottleneck library to enforce adaptive throttling.
• Per‑provider and per‑key isolation is achieved through unique limiter keys generated from provider ID, connection ID, and optional model names.
• The three-stage pipeline consists of initialization (initializeRateLimits), slot acquisition (withRateLimit), and adaptive learning (updateFromHeaders).
• Queue admission control prevents resource waste by rejecting requests early when backlogs exceed maxQueueDepth.
• A watchdog mechanism automatically recovers "wedged" limiters every 30 seconds to prevent indefinite stalls.
• Learned limits are persisted to the database and restored on startup, allowing the system to retain provider-specific rate limit knowledge.

Frequently Asked Questions

How does OmniRoute isolate rate limits between different API keys?

OmniRoute generates unique limiter keys using getLimiterKey(), which combines the provider ID and connection ID (representing the API key). For providers with per‑model quotas like Codex or Gemini, the model name is appended to create distinct limiters. This ensures that rate limit consumption for one API key does not affect another, even when using the same provider.

What happens when a provider returns a 429 status code?

When updateFromHeaders() detects an HTTP 429 status or "over‑limit" signals in response headers, it immediately pauses the affected limiter and adjusts the minTime parameter to enforce longer gaps between requests. The system also respects retry‑after headers when present, temporarily halting dispatches to that specific provider connection until the cooldown period expires.

How are rate limit settings persisted across restarts?

The manager stores learned rate limits in a learnedLimits cache and debounces writes to the settings table via recordLearnedLimit(). During initialization, loadPersistedLimits() retrieves these values from the database and applies them to the corresponding limiters. This persistence layer ensures that discovered provider constraints survive application restarts and deployments.

Can rate limiting be disabled for specific connections?

Yes. The enabledConnections set tracks which connections have protection active. Administrators can call disableRateLimitProtection(connectionId) to remove a connection from this set, causing withRateLimit() to bypass throttling for that specific connection. Conversely, enableRateLimitProtection(connectionId) adds protection. The function isRateLimitEnabled(connectionId) checks the current state.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}