How the rateLimitManager in OmniRoute Enforces Rate Limits and Parses Headers

The rateLimitManager in OmniRoute protects upstream LLM providers from 429 errors by wrapping every request in a Bottleneck limiter that dynamically adjusts its reservoir and timing constraints based on provider-specific HTTP response headers.

The OmniRoute repository (diegosouzapw/OmniRoute) implements a sophisticated rate-limiting layer to prevent upstream throttling. The rateLimitManager service in open-sse/services/rateLimitManager.ts creates isolated Bottleneck instances for each provider connection and continuously adapts their throughput by parsing standard and custom rate-limit headers using utilities in open-sse/services/rateLimitManager/headers.ts.

Initializing Rate Limit Protection

Rate limiting is initialized through the initializeRateLimits function. The system reconciles which connections receive protection by evaluating two conditions: either rateLimitProtection is explicitly enabled for the connection, or the auto-enable safety net is active and the provider uses API-key authentication (the default for most non-OAuth providers).

For every enabled connection, the manager creates a dedicated Bottleneck limiter on-demand via the getLimiter function. This ensures that rate limits are isolated per provider and connection, preventing one provider's constraints from affecting another.

Configuring Limiter Defaults and Overrides

Global Default Settings

The manager sources baseline configuration from the global resilience settings. The buildLimiterDefaults function (lines 58-66) extracts default values for maxConcurrent, minTime, and reservoir size from DEFAULT_RESILIENCE_SETTINGS.requestQueue.

function buildLimiterDefaults() {
  // Returns settings derived from global request-queue configuration
  // https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.50/open-sse/services/rateLimitManager.ts#L58-L66
}

Per-Connection Overrides

Administrators can override RPM (requests per minute), minTime, and maxConcurrent values by storing configuration in provider_connections.rateLimitOverrides. These overrides are applied when the limiter is first instantiated in getLimiter, allowing fine-grained control over individual provider connections without affecting global defaults.

Deterministic Limiter Keys

The getLimiterKey function (lines 84-102) generates unique keys to isolate limiters by provider, connection, and optionally by model. The key format varies by provider type:

• Codex: codex:${codexKey} (model-scoped)
• Antigravity/AGY: antigravity:${connectionId}:${family} (quota family-scoped)
• Gemini/GitHub: ${provider}:${connectionId}:${model} (model-scoped)
• All others: ${provider}:${connectionId}

This scheme ensures that different models or quota families within the same provider receive independent rate-limit buckets.

Request Flow and Enforcement via withRateLimit

The public API exposed in open-sse/services/rateLimitSemaphore.ts provides the withRateLimit function used by request pipelines. If rate limiting is disabled for a connection, the wrapped function executes directly. Otherwise, the flow follows these steps:

1. Acquire a slot via limiter.schedule(fn) (lines 80-92)
2. Respect current constraints: the limiter enforces its reservoir (remaining requests), minTime (minimum gap between calls), and maxConcurrent settings
3. Honor abort signals: if the request is aborted, the signal is passed through and respected by the scheduler

export async function withRateLimit(provider, connectionId, model, fn, signal = null, retryAfterWedge = true) {
  // https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.50/open-sse/services/rateLimitManager.ts#L80-L92
}

Parsing Rate Limit Headers

The manager adapts to provider-specific limits by parsing HTTP response headers through utilities in open-sse/services/rateLimitManager/headers.ts.

Header Constants and Normalization

The module exports two header dictionaries: STANDARD_HEADERS for common x-ratelimit-* headers and ANTHROPIC_HEADERS for Anthropic-specific fields. These constants map semantic names (like remaining or reset) to the actual header keys used by different providers (lines 6-28).

The toPlainHeaders function converts Header objects (or plain maps) into lower-cased key/value pairs for consistent parsing across different HTTP client implementations.

Reset Time Parsing

The parseResetTime function (lines 30-66) handles multiple timestamp formats returned by various providers:

• Relative seconds: "60" or "1s"
• Unix timestamps: seconds or milliseconds since epoch
• ISO dates: standard ISO-8601 formatted strings

The function returns the reset duration in milliseconds, calculated relative to the current time.

Adaptive Limit Updates

The checkQueueAdmission helper imported from ./rateLimitManager/admission reads parsed header values and invokes limiter.updateSettings to adjust the reservoir, refresh interval, and minTime dynamically. If a provider reports only 10 remaining requests with a reset window of 60 seconds, subsequent calls throttle accordingly to prevent 429 errors.

Watchdog and Lifecycle Management

Idle Eviction and Wedge Detection

The manager runs a watchdog via watchdogTick (lines 27-71) to prevent resource leaks and phantom queues:

• Idle eviction: Limiters unused for >10 minutes are disconnected and removed from memory
• Wedge detection: If a limiter has queued jobs but zero RUNNING or EXECUTING jobs for >120 seconds, the evictWedgeLimiter function stops the instance (dropping waiting jobs) and discards it, preventing permanent stalls

Graceful Shutdown

On SIGTERM or SIGINT signals, the shutdownLimiters function (lines 29-33) stops all active limiters without dropping queued jobs, allowing the process to exit cleanly without aborting in-flight requests.

Practical Implementation Examples

Enabling Protection for a Connection

import { enableRateLimitProtection, withRateLimit } from "@/open-sse/services/rateLimitManager";

// Enable protection for connection ID 42
await enableRateLimitProtection("conn-42");

// Execute with rate limiting
await withRateLimit(
  "openai",
  "conn-42",
  "gpt-4o",
  () => executor.execute(request),
  abortSignal
);

Manually Adapting Limiter from Headers

import { parseResetTime, toPlainHeaders, STANDARD_HEADERS } from "@/open-sse/services/rateLimitManager/headers";

function adaptLimiterFromResponse(limiter, response) {
  const plain = toPlainHeaders(response.headers);
  const remaining = Number(plain[STANDARD_HEADERS.remaining] ?? "0");
  const resetMs = parseResetTime(plain[STANDARD_HEADERS.reset]);

  limiter.updateSettings({
    reservoir: remaining,
    minTime: resetMs ? Math.max(resetMs, limiter.settings.minTime) : limiter.settings.minTime,
  });
}

Disabling Protection

import { disableRateLimitProtection } from "@/open-sse/services/rateLimitManager";

await disableRateLimitProtection("conn-internal");

Summary

• The rateLimitManager creates isolated Bottleneck limiters for each provider connection, keyed by provider, connection ID, and optionally by model or quota family as implemented in getLimiterKey
• Initialization logic in initializeRateLimits protects connections explicitly or via auto-enable for API-key providers, with defaults sourced from DEFAULT_RESILIENCE_SETTINGS
• Request wrapping via withRateLimit in rateLimitSemaphore.ts enforces reservoir, minTime, and maxConcurrent constraints while respecting abort signals
• Header parsing utilities in headers.ts extract rate limit data from standard and Anthropic-specific headers, with parseResetTime converting multiple timestamp formats to milliseconds
• Adaptive updates via checkQueueAdmission adjust limiter settings in real-time based on provider responses
• Watchdog processes in rateLimitManager.ts evict idle limiters after 10 minutes and wedged limiters after 120 seconds of stagnation, with shutdownLimiters handling graceful process termination

Frequently Asked Questions

How does the rateLimitManager determine which connections to protect?

The manager evaluates connections during initialization in initializeRateLimits. A connection receives protection if rateLimitProtection is explicitly enabled in its configuration, or if the auto-enable safety net is active and the provider uses API-key authentication. This default covers most non-OAuth providers while allowing manual opt-in or opt-out for specific connections.

What header formats does OmniRoute support for rate limit resets?

The parseResetTime function in open-sse/services/rateLimitManager/headers.ts supports relative seconds (e.g., "60" or "1s"), Unix timestamps in seconds or milliseconds, and ISO-8601 date strings. The function normalizes all formats to millisecond durations relative to the current time, enabling consistent handling across providers with different header conventions.

How does the watchdog prevent resource leaks?

The watchdogTick function monitors all active limiters every few seconds. It disconnects limiters idle for more than 10 minutes to free memory, and detects "wedged" limiters—those with queued jobs but no running executions for over 120 seconds—using evictWedgeLimiter. This forced eviction drops stale queued jobs and removes the corrupted limiter instance, preventing permanent queue stalls.

Can rate limits be overridden per connection without changing global settings?

Yes. The buildLimiterDefaults function sources global defaults from DEFAULT_RESILIENCE_SETTINGS, but the getLimiter function applies per-connection overrides stored in provider_connections.rateLimitOverrides. These overrides can specify custom RPM, minTime, and maxConcurrent values that take precedence over both global defaults and dynamically learned header values.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}