# How to Implement Custom Rate Limiting Per API Key Using OmniRoute's RateLimitManager

> Implement custom rate limiting per API key with OmniRoute's RateLimitManager. Store JSON overrides and refresh connections to apply per-key throttling that supersedes global defaults.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: how-to-guide
- Published: 2026-07-19

---

**Store JSON overrides in the `rateLimitOverrides` column and invoke `refreshConnectionRateLimits()` to apply per-API-key throttling that supersedes global defaults.**

OmniRoute v3.8.49 ships with a **RateLimitManager** located in [`open-sse/services/rateLimitManager.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/rateLimitManager.ts) that orchestrates Bottleneck-based limiters for every provider connection. By leveraging the `connectionRateLimitOverrides` map and the database persistence layer, you can enforce granular throughput caps—such as requests per minute (RPM) and max concurrency—tailored to individual API keys.

## Understanding the RateLimitManager Architecture

The RateLimitManager maintains an in-memory registry of **Bottleneck** limiters keyed by `provider:connectionId[:model]`. When a request flows through the system, the manager resolves the effective throttling policy by merging global defaults with per-connection overrides.

Key architectural components include:

- **`connectionRateLimitOverrides`** – A `Map<string, RateLimitOverrides>` populated at startup by `reconcileEnabledConnections()` and updated on-the-fly via `refreshConnectionRateLimits()` (lines 84‑87 in [`rateLimitManager.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/rateLimitManager.ts)).
- **`buildLimiterDefaults()`** – Constructs the configuration object for a new limiter, resolving fields like `reservoir` (RPM), `minTime`, and `maxConcurrent` by checking overrides first, then falling back to global `RequestQueueSettings` (lines 53‑63).
- **`limiters`** – The active registry of Bottleneck instances. When overrides change, the manager calls `limiter.updateSettings()` to apply new constraints without restarting the service.

Overrides are persisted in the **`rateLimitOverrides`** JSON column of the `provider_connections` table, defined in [`src/lib/db/providers/columns.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/db/providers/columns.ts) (lines 44‑53).

## Configuring Rate Limit Overrides Per API Key

OmniRoute accepts five override parameters that map directly to Bottleneck configuration options:

| Override | Type | Description |
|----------|------|-------------|
| **`rpm`** | `number` | Requests per minute (reservoir). Set to `0` for unlimited. |
| **`tpm`** | `number` | Tokens per minute (tracked for quota accounting). |
| **`tpd`** | `number` | Tokens per day (quota tracking). |
| **`minTime`** | `number` | Minimum milliseconds between requests (e.g., `250`). |
| **`maxConcurrent`** | `number` | Maximum concurrent jobs (e.g., `5`). |

### Method 1: Using the Provider Connection REST API

The simplest way to update limits is through the built-in endpoint at `src/app/api/providers/[id]/route.ts`. Submit a `PUT` request with a `rateLimitOverrides` payload:

```bash
curl -X PUT "https://your-omniroute.com/api/providers/conn-12345" \
  -H "Content-Type: application/json" \
  -d '{
        "rateLimitOverrides": {
          "rpm": 150,
          "minTime": 400,
          "maxConcurrent": 2
        }
      }'

```

The route handler writes the JSON to the database and automatically calls `refreshConnectionRateLimits(connectionId, overrides)` at line 296, ensuring the running limiter picks up the new values immediately.

### Method 2: Direct Database Update

For programmatic control, write directly to the `rateLimitOverridesJson` column (aliased as `rateLimitOverrides` in the schema) and manually refresh the manager:

```typescript
import { getDbInstance } from "@/src/lib/db/core";
import { sanitizeRateLimitOverrides } from "@/src/lib/db/providers";
import { refreshConnectionRateLimits } from "open-sse/services/rateLimitManager";

async function setApiKeyLimits(connectionId: string, overrides: Record<string, number>) {
  const db = getDbInstance();
  const safe = sanitizeRateLimitOverrides(overrides);
  
  // Persist to provider_connections table
  db.prepare(`
    UPDATE provider_connections
    SET rateLimitOverridesJson = ?
    WHERE id = ?
  `).run(JSON.stringify(safe), connectionId);

  // Push changes to the in-memory RateLimitManager
  await refreshConnectionRateLimits(connectionId, safe);
}

```

## Refreshing and Activating Custom Limits

The **`refreshConnectionRateLimits(connectionId, overrides?)`** function (exposed from [`rateLimitManager.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/rateLimitManager.ts)) performs three critical actions:

1. Updates the `connectionRateLimitOverrides` map with the new configuration.
2. Retrieves the existing Bottleneck limiter for that connection (if any).
3. Invokes `limiter.updateSettings()` with the resolved `maxConcurrent`, `minTime`, and `reservoir` values.

If you used the REST API method, this happens automatically. For database-direct updates, you must invoke it manually as shown in the previous example.

## Verifying Limiter Status and Protection

After applying overrides, confirm the active configuration using **`getRateLimitStatus()`**:

```typescript
import { getRateLimitStatus } from "open-sse/services/rateLimitManager";

const status = await getRateLimitStatus("openai", "conn-12345");
console.log(status);
// { reservoir: 150, minTime: 400, maxConcurrent: 2, ... }

```

Rate-limit protection can be toggled per connection via **`enableRateLimitProtection(connectionId)`** and **`disableRateLimitProtection(connectionId)`** (lines 391‑398). API-key providers default to enabled protection, while OAuth providers allow disabling.

## Complete Implementation Example

The following TypeScript workflow demonstrates end-to-end configuration for a high-priority API key requiring stricter limits than the global defaults:

```typescript
import {
  refreshConnectionRateLimits,
  enableRateLimitProtection,
  getRateLimitStatus,
} from "open-sse/services/rateLimitManager";
import { getDbInstance } from "@/src/lib/db/core";

async function configureCustomRateLimit(connectionId: string) {
  // 1. Define per-API-key constraints
  const overrides = {
    rpm: 200,          // 200 requests per minute
    minTime: 300,      // 300ms gap between requests
    maxConcurrent: 3,  // Max 3 parallel requests
  };

  // 2. Persist to database
  const db = getDbInstance();
  db.prepare(`
    UPDATE provider_connections
    SET rateLimitOverridesJson = ?
    WHERE id = ?
  `).run(JSON.stringify(overrides), connectionId);

  // 3. Apply to running RateLimitManager
  await refreshConnectionRateLimits(connectionId, overrides);

  // 4. Ensure protection is active
  await enableRateLimitProtection(connectionId);

  // 5. Verify
  const status = await getRateLimitStatus("anthropic", connectionId);
  console.assert(status.reservoir === 200, "RPM override applied");
}

```

## Summary

- **Store overrides** in the `rateLimitOverrides` JSON column of the `provider_connections` table to persist per-API-key configuration.
- **Call `refreshConnectionRateLimits()`** after any database update to synchronize the in-memory Bottleneck limiter with new values.
- **Use `buildLimiterDefaults()`** logic (lines 53‑63) to understand how `rpm`, `minTime`, and `maxConcurrent` resolve into Bottleneck settings.
- **Verify active limits** via `getRateLimitStatus()` and toggle protection with `enableRateLimitProtection()` as needed.
- **Prefer the REST API** at `src/app/api/providers/[id]/route.ts` for dashboard-driven changes, or use direct DB access for automated provisioning.

## Frequently Asked Questions

### What override values can I specify for custom rate limiting per API key?

You can provide any combination of **`rpm`** (requests per minute), **`tpm`** (tokens per minute), **`tpd`** (tokens per day), **`minTime`** (minimum interval in milliseconds), and **`maxConcurrent`** (simultaneous requests). These map directly to Bottleneck configuration parameters in [`open-sse/services/rateLimitManager.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/rateLimitManager.ts).

### How does OmniRoute handle conflicts between global settings and per-API-key overrides?

The **`buildLimiterDefaults()`** function (lines 53‑63) prioritizes per-connection overrides stored in `connectionRateLimitOverrides`. If an override is `undefined`, the manager falls back to global `RequestQueueSettings` defined in [`src/lib/resilience/settings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/settings.ts).

### Can I disable rate limiting entirely for a specific API key?

Yes, but only for OAuth-based providers. Invoke **`disableRateLimitProtection(connectionId)`** to remove the connection from the `enabledConnections` set. API-key providers enforce rate-limit protection by default and should use `rpm: 0` in overrides to simulate unlimited throughput rather than disabling protection.

### Where are the custom rate limit overrides physically stored?

Overrides are serialized as JSON in the **`rateLimitOverridesJson`** column (exposed as `rateLimitOverrides`) of the `provider_connections` table, defined in [`src/lib/db/providers/columns.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/db/providers/columns.ts). The RateLimitManager hydrates these values into its `connectionRateLimitOverrides` Map at startup and refreshes them on-demand via `refreshConnectionRateLimits()`.