How to Configure Rate Limiting in OmniRoute: A Complete Guide

OmniRoute protects downstream LLM providers using an adaptive rate-limiting system built on the Bottleneck library that operates at global, per-connection, and API response levels.

To configure rate limiting in OmniRoute, you work with a three-layer resilience architecture defined in the diegosouzapw/OmniRoute repository. The system prevents quota exhaustion by combining global request-queue policies with intelligent per-connection limiters that automatically adapt to upstream rate-limit headers sent by providers like OpenAI and Anthropic.

Understanding the Three-Layer Architecture

OmniRoute implements rate limiting through three coordinated layers, each managed through specific source files in the codebase.

Global Request-Queue Policy

The foundation resides in src/lib/resilience/settings.ts. This layer defines default behaviors such as requests per minute (RPM), minimum time between calls, and maximum concurrent jobs. These settings apply globally unless overridden at the connection level, and are exposed through both the dashboard and environment variables.

Per-Connection Limiters

Individual protection is handled in open-sse/services/rateLimitManager.ts. For every provider + connection pair, the system creates a Bottleneck limiter instance. These limiters learn dynamically from upstream response headers—including x-ratelimit-*, retry-after, and Anthropic-specific headers—and update their internal reservoir, minTime, and maxConcurrent values via updateLimiterSettings().

Rate-Limit Response Handling

When all accounts for a provider are exhausted, src/app/api/v1/_shared/rateLimit.ts generates standardized HTTP 429 responses. The rateLimitedProviderResponse() function ensures consistent error formatting across API routes, including retry-after hints.

Enabling and Disabling Protection

Dashboard Controls

Each provider connection includes a "Rate-limit protection" toggle in the dashboard. When enabled, the connection is added to the enabledConnections set within the rate limit manager, activating its dedicated Bottleneck instance.

Automatic Enablement for API Keys

The system automatically protects API-key-based providers when requestQueue.autoEnableApiKeyProviders is set to true (the default). Override this behavior globally using the RATE_LIMIT_AUTO_ENABLE environment variable (true or false).

Per-Connection Overrides

Fine-tune specific connections by setting values in the rateLimitOverrides column of the provider_connections database table. At startup, initializeRateLimits() loads these into the connectionRateLimitOverrides Map, allowing specific RPM and timing constraints without affecting global defaults.

Key Configuration Parameters

Located in src/lib/resilience/settings.ts, these settings control global queue behavior:

• requestQueue.requestsPerMinute: Base RPM limit when no overrides exist. Default: 0 (infinite).
• requestQueue.minTimeBetweenRequestsMs: Minimum gap in milliseconds between requests. Default: 0.
• requestQueue.concurrentRequests: Maximum concurrent jobs per connection. Default: 0 (infinite).
• requestQueue.maxWaitMs: Maximum time a request may wait in queue before rejection. Default: 15000.
• requestQueue.maxQueueDepth: Maximum queued jobs per connection. Default: 0 (unbounded).
• requestQueue.autoEnableApiKeyProviders: Auto-enable protection for API-key connections. Default: true.

Environment variables override dashboard settings:

• RATE_LIMIT_AUTO_ENABLE: Forces the auto-enable flag on or off.
• RATE_LIMIT_MAX_WAIT_MS: Overrides the global maxWaitMs value.
• RATE_LIMIT_MAX_QUEUE_DEPTH: Overrides the global queue depth limit.

How the Adaptive System Works

The rate limiting lifecycle follows four distinct phases:

1. Initialization: On server startup, initializeRateLimits() reads persisted connections from the database, loads any saved overrides from connectionRateLimitOverrides, and instantiates Bottleneck limiters for each enabled connection.

2. Header Learning: After each successful request, parseResetTime() extracts rate-limit headers from the provider response. The manager calls updateLimiterSettings() to dynamically adjust the limiter's reservoir and timing constraints based on upstream quotas.

3. Admission Control: Before processing requests, checkQueueAdmission() verifies that the limiter can accept new jobs. If the queue depth exceeds limits or the wait time would surpass maxWaitMs, the system returns a RATE_LIMITED error immediately.

4. Watchdog Monitoring: The LimiterWedgeWatchdog periodically scans limiters for "wedged" states (stuck or deadlocked jobs) and performs resets to maintain system health without manual intervention.

Implementation Examples

Initialize Rate-Limit Protection

Call this once during server startup to load settings and start the watchdog:

import { initializeRateLimits } from "@/open-sse/services/rateLimitManager";

await initializeRateLimits(); // Loads settings, creates limiters, starts watchdog

Enable Protection for a Specific Connection

Activate protection programmatically when a user toggles it in your UI:

import { enableRateLimitProtection } from "@/open-sse/services/rateLimitManager";

enableRateLimitProtection("connection-12345");

Configure Per-Connection Overrides

Set specific RPM and timing constraints for individual connections:

import { connectionRateLimitOverrides } from "@/open-sse/services/rateLimitManager";

connectionRateLimitOverrides.set("connection-12345", {
  requestsPerMinute: 100,
  minTimeBetweenRequestsMs: 200,
});

Apply Global Request-Queue Settings

Update system-wide defaults from administrative interfaces:

import { applyRequestQueueSettings } from "@/open-sse/services/rateLimitManager";
import type { RequestQueueSettings } from "@/lib/resilience/settings";

const newSettings: RequestQueueSettings = {
  maxWaitMs: 20000,
  maxQueueDepth: 50,
  requestsPerMinute: 0,
  minTimeBetweenRequestsMs: 0,
  concurrentRequests: 0,
  autoEnableApiKeyProviders: true,
};

await applyRequestQueueSettings(newSettings);

Return Standardized Rate-Limit Responses

Handle exhausted provider quotas in API routes:

import { rateLimitedProviderResponse } from "@/app/api/v1/_shared/rateLimit";

if (allAccountsRateLimited) {
  const credentials = { allRateLimited: true, retryAfter: "60s" };
  return rateLimitedProviderResponse("openai", credentials);
}

Summary

• Three-layer architecture: Global settings in src/lib/resilience/settings.ts, per-connection Bottleneck limiters in open-sse/services/rateLimitManager.ts, and response handling in src/app/api/v1/_shared/rateLimit.ts.
• Adaptive learning: The system parses upstream headers like x-ratelimit-* and retry-after to automatically adjust constraints via updateLimiterSettings().
• Flexible configuration: Control behavior through the dashboard, environment variables (RATE_LIMIT_AUTO_ENABLE, RATE_LIMIT_MAX_WAIT_MS), or per-connection database overrides.
• Health monitoring: LimiterWedgeWatchdog prevents stuck jobs from blocking queues indefinitely.
• Standardized errors: Use rateLimitedProviderResponse() to return consistent HTTP 429 responses when all provider accounts are exhausted.

Frequently Asked Questions

How do I completely disable rate limiting in OmniRoute?

To disable rate limiting entirely, set requestQueue.requestsPerMinute to 0, requestQueue.concurrentRequests to 0, and set the RATE_LIMIT_AUTO_ENABLE environment variable to false. Additionally, ensure no connections have the dashboard toggle enabled and clear any entries in the connectionRateLimitOverrides Map.

Can I set different rate limits for different LLM providers?

Yes. Configure per-provider limits by storing override values in the rateLimitOverrides column of the provider_connections table for each specific connection. When initializeRateLimits() runs, it loads these into the connectionRateLimitOverrides Map, applying unique RPM and concurrency constraints per provider regardless of global settings.

What happens when a request exceeds the maximum wait time?

When checkQueueAdmission() determines that a request would wait longer than requestQueue.maxWaitMs (default 15000ms) or exceed maxQueueDepth, it immediately returns a RATE_LIMITED error instead of queuing the request. This prevents client timeouts and provides immediate feedback that the provider capacity is saturated.

How does OmniRoute handle upstream rate-limit headers?

After each successful request, the system calls parseResetTime() to analyze headers such as x-ratelimit-*, retry-after, and Anthropic-specific rate-limit headers. It then invokes updateLimiterSettings() to dynamically adjust the Bottleneck limiter's reservoir, minTime, and maxConcurrent values, ensuring the system respects the provider's current quota state without manual reconfiguration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}