# How the OmniRoute Rate Limit Manager Enforces Per‑Provider and Per‑Key Limits

> Learn how the OmniRoute rate limit manager enforces per-provider and per-key limits using adaptive throttling and a three-stage pipeline for effective API management. Boost your application's stability.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: how-to-guide
- Published: 2026-07-25

---

**The OmniRoute rate limit manager implements adaptive throttling using the Bottleneck library to enforce granular per‑provider and per‑connection (API‑key) limits through a three-stage pipeline of initialization, slot acquisition, and response header learning.**

OmniRoute is an open‑source AI request routing layer that prevents downstream provider overload while ensuring equitable resource distribution. The **OmniRoute rate limit manager**, implemented in `open‑sse/services/rateLimitManager.ts`, orchestrates this protection by dynamically adjusting request throughput based on real‑time provider feedback and configured policies.

## Three-Stage Rate Limit Enforcement Pipeline

The enforcement system operates through three distinct stages that manage the lifecycle of rate limiting from startup through request completion.

### Stage 1: Initialization and Limiter Creation

On application startup, the manager loads all persisted provider connections and determines which connections require protection. This process explicitly enables protection for API‑key providers or uses auto‑detection heuristics. For each protected connection, the system creates a dedicated **Bottleneck** limiter instance with default settings.

The `initializeRateLimits()` function triggers `reconcileEnabledConnections()` ([source lines ≈ 74‑99](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.49/open-sse/services/rateLimitManager.ts#L74-L99)) to build the initial limiter registry. Each limiter maintains independent state for request queuing, ensuring that congestion on one provider or API key does not affect others.

### Stage 2: Slot Acquisition with withRateLimit

Every outbound request flows through `withRateLimit()` ([source lines ≈ 545‑580](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.49/open-sse/services/rateLimitManager.ts#L545-L580)), which implements the core throttling logic:

- **Unprotected bypass**: If rate limiting is disabled for the connection, the request executes immediately.
- **Default application**: Applies global resilience settings via `buildLimiterDefaults()` ([source lines ≈ 154‑176](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.49/open-sse/services/rateLimitManager.ts#L154-L176)).
- **Override merging**: Combines defaults with per‑connection overrides from `provider_connections.rateLimitOverrides` (RPM, minTime, maxConcurrent).
- **Queue admission**: Validates queue depth against `maxQueueDepth` via `checkQueueAdmission()` in [`open-sse/services/rateLimitManager/admission.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/rateLimitManager/admission.ts) ([source lines ≈ 82‑90](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.49/open-sse/services/rateLimitManager.ts#L82-L90)), rejecting requests early if the limiter is backlogged.
- **Scheduled execution**: Schedules the request on the limiter with a global `maxWaitMs` timeout to prevent indefinite stalls.

### Stage 3: Adaptive Learning from Response Headers

After receiving a provider response, the manager parses standard rate‑limit headers (`x‑ratelimit‑limit`, `x‑ratelimit‑remaining`, `retry‑after`) through `updateFromHeaders()` ([source lines ≈ 710‑860](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.49/open-sse/services/rateLimitManager.ts#L710-L860)) and response bodies via `updateFromResponseBody()` ([source lines ≈ 996‑1010](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.49/open-sse/services/rateLimitManager.ts#L996-L1010)). Header parsing logic utilizes the maps defined in [`open-sse/services/rateLimitManager/headers.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/rateLimitManager/headers.ts) (`STANDARD_HEADERS` and `ANTHROPIC_HEADERS`) to normalize provider‑specific formats.

The system then updates the Bottleneck limiter with:

- A computed `minTime` (inter‑request gap) derived from advertised RPM.
- **Reservoir settings** when remaining quota is low.
- Temporary pausing on HTTP 429 or "over‑limit" signals.

Learned values are cached in `learnedLimits` and persisted for future restarts.

## Core Mechanisms for Per‑Provider and Per‑Key Isolation

### Unique Key Generation for Isolation

The manager generates unique limiter keys to guarantee isolation between different providers and API keys. The `getLimiterKey()` function ([source lines ≈ 81‑95](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.49/open-sse/services/rateLimitManager.ts#L81-L95)) constructs identifiers using the provider ID and connection ID:

```typescript
const key = `${provider}:${connectionId}`; // base case
// Model‑scoped keys for Codex, Gemini, etc.

```

For providers with per‑model quotas (such as Codex, Gemini, and Antigravity), the key optionally incorporates the model name, creating distinct limiters for each model tier.

### Dynamic Defaults and Connection Overrides

Default limits originate from global resilience settings: `requestsPerMinute`, `minTimeBetweenRequestsMs`, and `concurrentRequests`. The `buildLimiterDefaults()` function merges these with per‑connection overrides stored in the database. Zero values in override fields are treated as "no override," allowing connections to effectively disable their own limits while retaining global protection. Fallback defaults for providers without explicit headers are defined in [`open-sse/services/providerDefaultRateLimit.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/providerDefaultRateLimit.ts).

### Queue Admission Control and Early Rejection

Before scheduling, `checkQueueAdmission()` inspects the current queue depth via `limiter.counts().QUEUED`. If the depth exceeds `maxQueueDepth`, the manager throws a fast‑reject error, preventing expensive downstream work such as prompt compression or token estimation from executing on doomed requests.

### Watchdog Recovery for Wedged Limiters

A background watchdog runs every 30 seconds via `watchdogTick()` ([source lines ≈ 27‑84](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.49/open-sse/services/rateLimitManager.ts#L27-L84)). If a limiter has queued jobs but no dispatches for over 120 seconds, the system considers it "wedged" and force‑resets it using `stop({ dropWaitingJobs: true })`. This prevents indefinite hangs when Bottleneck’s internal state becomes inconsistent.

### Persistence of Learned Rate Limits

When providers supply usable rate‑limit metadata, `recordLearnedLimit()` ([source lines ≈ 448‑468](https://github.com/diegosouzapw/OmniRoute/blob/release/v3.8.49/open-sse/services/rateLimitManager.ts#L448-L468)) debounces writes to the `settings` table. On subsequent startups, `loadPersistedLimits()` reapplies these learned values to the appropriate limiters, allowing the system to retain knowledge of provider constraints across deployments.

## Rate Limit Manager Implementation Examples

### Wrapping a Provider Request

Use `withRateLimit()` to ensure requests respect per‑connection limits:

```typescript
import { withRateLimit } from "@/open-sse/services/rateLimitManager";
import { execute } from "@/open-sse/executors/default";

export async function handleChatCore(provider, connectionId, model, request, abortSignal) {
  return withRateLimit(
    provider,
    connectionId,
    model,
    () => execute(provider, connectionId, model, request, abortSignal),
    abortSignal
  );
}

```

### Updating Limiter State from Headers

After each request, update the limiter based on provider feedback:

```typescript
import { updateFromHeaders } from "@/open-sse/services/rateLimitManager";

const response = await fetch(url, fetchOpts);
await updateFromHeaders(
  providerId,
  connectionId,
  response.headers,
  response.status,
  modelName
);

```

### Programmatic Protection Control

Enable or disable protection via the manager's API:

```typescript
import {
  enableRateLimitProtection,
  disableRateLimitProtection,
  isRateLimitEnabled,
} from "@/open-sse/services/rateLimitManager";

if (userWantsProtection) enableRateLimitProtection(connectionId);
else disableRateLimitProtection(connectionId);

console.log(isRateLimitEnabled(connectionId)); // true / false

```

## Summary

- The **OmniRoute rate limit manager** in [`open-sse/services/rateLimitManager.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/rateLimitManager.ts) uses the **Bottleneck** library to enforce adaptive throttling.
- **Per‑provider and per‑key isolation** is achieved through unique limiter keys generated from provider ID, connection ID, and optional model names.
- The **three-stage pipeline** consists of initialization (`initializeRateLimits`), slot acquisition (`withRateLimit`), and adaptive learning (`updateFromHeaders`).
- **Queue admission control** prevents resource waste by rejecting requests early when backlogs exceed `maxQueueDepth`.
- A **watchdog mechanism** automatically recovers "wedged" limiters every 30 seconds to prevent indefinite stalls.
- **Learned limits** are persisted to the database and restored on startup, allowing the system to retain provider-specific rate limit knowledge.

## Frequently Asked Questions

### How does OmniRoute isolate rate limits between different API keys?

OmniRoute generates unique limiter keys using `getLimiterKey()`, which combines the provider ID and connection ID (representing the API key). For providers with per‑model quotas like Codex or Gemini, the model name is appended to create distinct limiters. This ensures that rate limit consumption for one API key does not affect another, even when using the same provider.

### What happens when a provider returns a 429 status code?

When `updateFromHeaders()` detects an HTTP 429 status or "over‑limit" signals in response headers, it immediately pauses the affected limiter and adjusts the `minTime` parameter to enforce longer gaps between requests. The system also respects `retry‑after` headers when present, temporarily halting dispatches to that specific provider connection until the cooldown period expires.

### How are rate limit settings persisted across restarts?

The manager stores learned rate limits in a `learnedLimits` cache and debounces writes to the `settings` table via `recordLearnedLimit()`. During initialization, `loadPersistedLimits()` retrieves these values from the database and applies them to the corresponding limiters. This persistence layer ensures that discovered provider constraints survive application restarts and deployments.

### Can rate limiting be disabled for specific connections?

Yes. The `enabledConnections` set tracks which connections have protection active. Administrators can call `disableRateLimitProtection(connectionId)` to remove a connection from this set, causing `withRateLimit()` to bypass throttling for that specific connection. Conversely, `enableRateLimitProtection(connectionId)` adds protection. The function `isRateLimitEnabled(connectionId)` checks the current state.