How to Implement Custom Rate Limiting Per API Key Using OmniRoute's RateLimitManager

Store JSON overrides in the rateLimitOverrides column and invoke refreshConnectionRateLimits() to apply per-API-key throttling that supersedes global defaults.

OmniRoute v3.8.49 ships with a RateLimitManager located in open-sse/services/rateLimitManager.ts that orchestrates Bottleneck-based limiters for every provider connection. By leveraging the connectionRateLimitOverrides map and the database persistence layer, you can enforce granular throughput caps—such as requests per minute (RPM) and max concurrency—tailored to individual API keys.

Understanding the RateLimitManager Architecture

The RateLimitManager maintains an in-memory registry of Bottleneck limiters keyed by provider:connectionId[:model]. When a request flows through the system, the manager resolves the effective throttling policy by merging global defaults with per-connection overrides.

Key architectural components include:

• connectionRateLimitOverrides – A Map<string, RateLimitOverrides> populated at startup by reconcileEnabledConnections() and updated on-the-fly via refreshConnectionRateLimits() (lines 84‑87 in rateLimitManager.ts).
• buildLimiterDefaults() – Constructs the configuration object for a new limiter, resolving fields like reservoir (RPM), minTime, and maxConcurrent by checking overrides first, then falling back to global RequestQueueSettings (lines 53‑63).
• limiters – The active registry of Bottleneck instances. When overrides change, the manager calls limiter.updateSettings() to apply new constraints without restarting the service.

Overrides are persisted in the rateLimitOverrides JSON column of the provider_connections table, defined in src/lib/db/providers/columns.ts (lines 44‑53).

Configuring Rate Limit Overrides Per API Key

OmniRoute accepts five override parameters that map directly to Bottleneck configuration options:

Override	Type	Description
rpm	number	Requests per minute (reservoir). Set to 0 for unlimited.
tpm	number	Tokens per minute (tracked for quota accounting).
tpd	number	Tokens per day (quota tracking).
minTime	number	Minimum milliseconds between requests (e.g., 250).
maxConcurrent	number	Maximum concurrent jobs (e.g., 5).

Method 1: Using the Provider Connection REST API

The simplest way to update limits is through the built-in endpoint at src/app/api/providers/[id]/route.ts. Submit a PUT request with a rateLimitOverrides payload:

curl -X PUT "https://your-omniroute.com/api/providers/conn-12345" \
  -H "Content-Type: application/json" \
  -d '{
        "rateLimitOverrides": {
          "rpm": 150,
          "minTime": 400,
          "maxConcurrent": 2
        }
      }'

The route handler writes the JSON to the database and automatically calls refreshConnectionRateLimits(connectionId, overrides) at line 296, ensuring the running limiter picks up the new values immediately.

Method 2: Direct Database Update

For programmatic control, write directly to the rateLimitOverridesJson column (aliased as rateLimitOverrides in the schema) and manually refresh the manager:

import { getDbInstance } from "@/src/lib/db/core";
import { sanitizeRateLimitOverrides } from "@/src/lib/db/providers";
import { refreshConnectionRateLimits } from "open-sse/services/rateLimitManager";

async function setApiKeyLimits(connectionId: string, overrides: Record<string, number>) {
  const db = getDbInstance();
  const safe = sanitizeRateLimitOverrides(overrides);
  
  // Persist to provider_connections table
  db.prepare(`
    UPDATE provider_connections
    SET rateLimitOverridesJson = ?
    WHERE id = ?
  `).run(JSON.stringify(safe), connectionId);

  // Push changes to the in-memory RateLimitManager
  await refreshConnectionRateLimits(connectionId, safe);
}

Refreshing and Activating Custom Limits

The refreshConnectionRateLimits(connectionId, overrides?) function (exposed from rateLimitManager.ts) performs three critical actions:

1. Updates the connectionRateLimitOverrides map with the new configuration.
2. Retrieves the existing Bottleneck limiter for that connection (if any).
3. Invokes limiter.updateSettings() with the resolved maxConcurrent, minTime, and reservoir values.

If you used the REST API method, this happens automatically. For database-direct updates, you must invoke it manually as shown in the previous example.

Verifying Limiter Status and Protection

After applying overrides, confirm the active configuration using getRateLimitStatus():

import { getRateLimitStatus } from "open-sse/services/rateLimitManager";

const status = await getRateLimitStatus("openai", "conn-12345");
console.log(status);
// { reservoir: 150, minTime: 400, maxConcurrent: 2, ... }

Rate-limit protection can be toggled per connection via enableRateLimitProtection(connectionId) and disableRateLimitProtection(connectionId) (lines 391‑398). API-key providers default to enabled protection, while OAuth providers allow disabling.

Complete Implementation Example

The following TypeScript workflow demonstrates end-to-end configuration for a high-priority API key requiring stricter limits than the global defaults:

import {
  refreshConnectionRateLimits,
  enableRateLimitProtection,
  getRateLimitStatus,
} from "open-sse/services/rateLimitManager";
import { getDbInstance } from "@/src/lib/db/core";

async function configureCustomRateLimit(connectionId: string) {
  // 1. Define per-API-key constraints
  const overrides = {
    rpm: 200,          // 200 requests per minute
    minTime: 300,      // 300ms gap between requests
    maxConcurrent: 3,  // Max 3 parallel requests
  };

  // 2. Persist to database
  const db = getDbInstance();
  db.prepare(`
    UPDATE provider_connections
    SET rateLimitOverridesJson = ?
    WHERE id = ?
  `).run(JSON.stringify(overrides), connectionId);

  // 3. Apply to running RateLimitManager
  await refreshConnectionRateLimits(connectionId, overrides);

  // 4. Ensure protection is active
  await enableRateLimitProtection(connectionId);

  // 5. Verify
  const status = await getRateLimitStatus("anthropic", connectionId);
  console.assert(status.reservoir === 200, "RPM override applied");
}

Summary

• Store overrides in the rateLimitOverrides JSON column of the provider_connections table to persist per-API-key configuration.
• Call refreshConnectionRateLimits() after any database update to synchronize the in-memory Bottleneck limiter with new values.
• Use buildLimiterDefaults() logic (lines 53‑63) to understand how rpm, minTime, and maxConcurrent resolve into Bottleneck settings.
• Verify active limits via getRateLimitStatus() and toggle protection with enableRateLimitProtection() as needed.
• Prefer the REST API at src/app/api/providers/[id]/route.ts for dashboard-driven changes, or use direct DB access for automated provisioning.

Frequently Asked Questions

What override values can I specify for custom rate limiting per API key?

You can provide any combination of rpm (requests per minute), tpm (tokens per minute), tpd (tokens per day), minTime (minimum interval in milliseconds), and maxConcurrent (simultaneous requests). These map directly to Bottleneck configuration parameters in open-sse/services/rateLimitManager.ts.

How does OmniRoute handle conflicts between global settings and per-API-key overrides?

The buildLimiterDefaults() function (lines 53‑63) prioritizes per-connection overrides stored in connectionRateLimitOverrides. If an override is undefined, the manager falls back to global RequestQueueSettings defined in src/lib/resilience/settings.ts.

Can I disable rate limiting entirely for a specific API key?

Yes, but only for OAuth-based providers. Invoke disableRateLimitProtection(connectionId) to remove the connection from the enabledConnections set. API-key providers enforce rate-limit protection by default and should use rpm: 0 in overrides to simulate unlimited throughput rather than disabling protection.

Where are the custom rate limit overrides physically stored?

Overrides are serialized as JSON in the rateLimitOverridesJson column (exposed as rateLimitOverrides) of the provider_connections table, defined in src/lib/db/providers/columns.ts. The RateLimitManager hydrates these values into its connectionRateLimitOverrides Map at startup and refreshes them on-demand via refreshConnectionRateLimits().

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}