# How to Configure Rate Limiting in OmniRoute: A Complete Guide

> Learn to configure rate limiting in OmniRoute with this guide. Protect your LLM providers using global, per-connection, and API response level controls built on Bottleneck.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: how-to-guide
- Published: 2026-08-23

---

**OmniRoute protects downstream LLM providers using an adaptive rate-limiting system built on the Bottleneck library that operates at global, per-connection, and API response levels.**

To configure rate limiting in OmniRoute, you work with a three-layer resilience architecture defined in the `diegosouzapw/OmniRoute` repository. The system prevents quota exhaustion by combining global request-queue policies with intelligent per-connection limiters that automatically adapt to upstream rate-limit headers sent by providers like OpenAI and Anthropic.

## Understanding the Three-Layer Architecture

OmniRoute implements rate limiting through three coordinated layers, each managed through specific source files in the codebase.

### Global Request-Queue Policy

The foundation resides in [`src/lib/resilience/settings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/settings.ts). This layer defines default behaviors such as **requests per minute (RPM)**, minimum time between calls, and maximum concurrent jobs. These settings apply globally unless overridden at the connection level, and are exposed through both the dashboard and environment variables.

### Per-Connection Limiters

Individual protection is handled in [`open-sse/services/rateLimitManager.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/rateLimitManager.ts). For every **provider + connection** pair, the system creates a Bottleneck limiter instance. These limiters learn dynamically from upstream response headers—including `x-ratelimit-*`, `retry-after`, and Anthropic-specific headers—and update their internal `reservoir`, `minTime`, and `maxConcurrent` values via `updateLimiterSettings()`.

### Rate-Limit Response Handling

When all accounts for a provider are exhausted, [`src/app/api/v1/_shared/rateLimit.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/app/api/v1/_shared/rateLimit.ts) generates standardized HTTP 429 responses. The `rateLimitedProviderResponse()` function ensures consistent error formatting across API routes, including retry-after hints.

## Enabling and Disabling Protection

### Dashboard Controls

Each provider connection includes a **"Rate-limit protection"** toggle in the dashboard. When enabled, the connection is added to the `enabledConnections` set within the rate limit manager, activating its dedicated Bottleneck instance.

### Automatic Enablement for API Keys

The system automatically protects API-key-based providers when `requestQueue.autoEnableApiKeyProviders` is set to `true` (the default). Override this behavior globally using the **`RATE_LIMIT_AUTO_ENABLE`** environment variable (`true` or `false`).

### Per-Connection Overrides

Fine-tune specific connections by setting values in the `rateLimitOverrides` column of the `provider_connections` database table. At startup, `initializeRateLimits()` loads these into the `connectionRateLimitOverrides` Map, allowing specific RPM and timing constraints without affecting global defaults.

## Key Configuration Parameters

Located in [`src/lib/resilience/settings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/settings.ts), these settings control global queue behavior:

- **`requestQueue.requestsPerMinute`**: Base RPM limit when no overrides exist. Default: `0` (infinite).
- **`requestQueue.minTimeBetweenRequestsMs`**: Minimum gap in milliseconds between requests. Default: `0`.
- **`requestQueue.concurrentRequests`**: Maximum concurrent jobs per connection. Default: `0` (infinite).
- **`requestQueue.maxWaitMs`**: Maximum time a request may wait in queue before rejection. Default: `15000`.
- **`requestQueue.maxQueueDepth`**: Maximum queued jobs per connection. Default: `0` (unbounded).
- **`requestQueue.autoEnableApiKeyProviders`**: Auto-enable protection for API-key connections. Default: `true`.

Environment variables override dashboard settings:

- **`RATE_LIMIT_AUTO_ENABLE`**: Forces the auto-enable flag on or off.
- **`RATE_LIMIT_MAX_WAIT_MS`**: Overrides the global `maxWaitMs` value.
- **`RATE_LIMIT_MAX_QUEUE_DEPTH`**: Overrides the global queue depth limit.

## How the Adaptive System Works

The rate limiting lifecycle follows four distinct phases:

1. **Initialization**: On server startup, `initializeRateLimits()` reads persisted connections from the database, loads any saved overrides from `connectionRateLimitOverrides`, and instantiates Bottleneck limiters for each enabled connection.

2. **Header Learning**: After each successful request, `parseResetTime()` extracts rate-limit headers from the provider response. The manager calls `updateLimiterSettings()` to dynamically adjust the limiter's `reservoir` and timing constraints based on upstream quotas.

3. **Admission Control**: Before processing requests, `checkQueueAdmission()` verifies that the limiter can accept new jobs. If the queue depth exceeds limits or the wait time would surpass `maxWaitMs`, the system returns a `RATE_LIMITED` error immediately.

4. **Watchdog Monitoring**: The `LimiterWedgeWatchdog` periodically scans limiters for "wedged" states (stuck or deadlocked jobs) and performs resets to maintain system health without manual intervention.

## Implementation Examples

### Initialize Rate-Limit Protection

Call this once during server startup to load settings and start the watchdog:

```typescript
import { initializeRateLimits } from "@/open-sse/services/rateLimitManager";

await initializeRateLimits(); // Loads settings, creates limiters, starts watchdog

```

### Enable Protection for a Specific Connection

Activate protection programmatically when a user toggles it in your UI:

```typescript
import { enableRateLimitProtection } from "@/open-sse/services/rateLimitManager";

enableRateLimitProtection("connection-12345");

```

### Configure Per-Connection Overrides

Set specific RPM and timing constraints for individual connections:

```typescript
import { connectionRateLimitOverrides } from "@/open-sse/services/rateLimitManager";

connectionRateLimitOverrides.set("connection-12345", {
  requestsPerMinute: 100,
  minTimeBetweenRequestsMs: 200,
});

```

### Apply Global Request-Queue Settings

Update system-wide defaults from administrative interfaces:

```typescript
import { applyRequestQueueSettings } from "@/open-sse/services/rateLimitManager";
import type { RequestQueueSettings } from "@/lib/resilience/settings";

const newSettings: RequestQueueSettings = {
  maxWaitMs: 20000,
  maxQueueDepth: 50,
  requestsPerMinute: 0,
  minTimeBetweenRequestsMs: 0,
  concurrentRequests: 0,
  autoEnableApiKeyProviders: true,
};

await applyRequestQueueSettings(newSettings);

```

### Return Standardized Rate-Limit Responses

Handle exhausted provider quotas in API routes:

```typescript
import { rateLimitedProviderResponse } from "@/app/api/v1/_shared/rateLimit";

if (allAccountsRateLimited) {
  const credentials = { allRateLimited: true, retryAfter: "60s" };
  return rateLimitedProviderResponse("openai", credentials);
}

```

## Summary

- **Three-layer architecture**: Global settings in [`src/lib/resilience/settings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/settings.ts), per-connection Bottleneck limiters in [`open-sse/services/rateLimitManager.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/rateLimitManager.ts), and response handling in [`src/app/api/v1/_shared/rateLimit.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/app/api/v1/_shared/rateLimit.ts).
- **Adaptive learning**: The system parses upstream headers like `x-ratelimit-*` and `retry-after` to automatically adjust constraints via `updateLimiterSettings()`.
- **Flexible configuration**: Control behavior through the dashboard, environment variables (`RATE_LIMIT_AUTO_ENABLE`, `RATE_LIMIT_MAX_WAIT_MS`), or per-connection database overrides.
- **Health monitoring**: `LimiterWedgeWatchdog` prevents stuck jobs from blocking queues indefinitely.
- **Standardized errors**: Use `rateLimitedProviderResponse()` to return consistent HTTP 429 responses when all provider accounts are exhausted.

## Frequently Asked Questions

### How do I completely disable rate limiting in OmniRoute?

To disable rate limiting entirely, set `requestQueue.requestsPerMinute` to `0`, `requestQueue.concurrentRequests` to `0`, and set the **`RATE_LIMIT_AUTO_ENABLE`** environment variable to `false`. Additionally, ensure no connections have the dashboard toggle enabled and clear any entries in the `connectionRateLimitOverrides` Map.

### Can I set different rate limits for different LLM providers?

Yes. Configure per-provider limits by storing override values in the `rateLimitOverrides` column of the `provider_connections` table for each specific connection. When `initializeRateLimits()` runs, it loads these into the `connectionRateLimitOverrides` Map, applying unique RPM and concurrency constraints per provider regardless of global settings.

### What happens when a request exceeds the maximum wait time?

When `checkQueueAdmission()` determines that a request would wait longer than `requestQueue.maxWaitMs` (default 15000ms) or exceed `maxQueueDepth`, it immediately returns a `RATE_LIMITED` error instead of queuing the request. This prevents client timeouts and provides immediate feedback that the provider capacity is saturated.

### How does OmniRoute handle upstream rate-limit headers?

After each successful request, the system calls `parseResetTime()` to analyze headers such as `x-ratelimit-*`, `retry-after`, and Anthropic-specific rate-limit headers. It then invokes `updateLimiterSettings()` to dynamically adjust the Bottleneck limiter's `reservoir`, `minTime`, and `maxConcurrent` values, ensuring the system respects the provider's current quota state without manual reconfiguration.