How to Configure Rate Limiting with Quota Trackers and Sliding Window Limiters in OmniRoute

OmniRoute implements a four-layer rate limiting architecture combining Redis-backed sliding windows, persistent database rate limits, real-time quota monitoring, and standardized HTTP 429 responses.

This guide walks through configuring each component of OmniRoute's rate limiting system. The repository diegosouzapw/OmniRoute provides both sliding window limiters for request throttling and quota trackers for provider-specific usage monitoring, making it suitable for multi-tenant API gateways.

Sliding-Window Rate Limiting: Redis or In-Memory

OmniRoute's core rate limiter lives in src/shared/utils/rateLimiter.ts. It supports dual backends: Redis for distributed deployments and an in-memory Map for single-node or test environments.

Defining Rate Limit Rules

A rule is a { limit, window } pair where window is expressed in seconds:

const rules = [
  { limit: 100, window: 60 },      // 100 requests per minute
  { limit: 1_000, window: 3_600 }  // 1,000 requests per hour
];

Using checkRateLimit()

The checkRateLimit(identifier, rules) function (lines 221-266) performs the actual enforcement:

import { checkRateLimit } from "@/shared/utils/rateLimiter";

export async function handler(req: Request) {
  const apiKey = req.headers.get("x-api-key");
  const result = await checkRateLimit(apiKey ?? "anonymous", rules);
  
  if (!result.allowed) {
    return new Response(
      `Rate limit exceeded (window: ${result.failedWindow}s)`,
      { status: 429 }
    );
  }
  // Continue processing...
}

Backend selection logic:

• If REDIS_URL is configured, executes the RATE_LIMIT_SCRIPT Lua script atomically
• Otherwise, uses a per-process Map with evictStaleRateLimitWindows() for memory management

Implementing Per-Route Rate Limits

Add custom rules to any API route. Here's a chat completions endpoint with tiered limits:

// src/app/api/v1/chat/completions/route.ts
import { checkRateLimit } from "@/shared/utils/rateLimiter";

const CHAT_RULES = [
  { limit: 50, window: 60 },      // 50 req/min per API key
  { limit: 1_000, window: 86_400 } // 1,000 req/day
];

export async function POST(req: Request) {
  const apiKey = req.headers.get("x-api-key") ?? "guest";
  const rl = await checkRateLimit(apiKey, CHAT_RULES);
  
  if (!rl.allowed) {
    return new Response("Too many requests", { status: 429 });
  }
  // Handle chat request...
}

Persistent Connection Rate Limiting

Unlike memory-only solutions, OmniRoute persists rate-limit state per provider connection in SQLite. This prevents OAuth token refreshes from clearing back-off timers.

Core Functions in src/lib/db/providers/rateLimit.ts

Function	Purpose
markConnectionRateLimitedUntil(connectionId, retryAfterMs)	Records when a connection can be reused
isConnectionRateLimited(connectionId)	Checks cooldown status during routing
setConnectionRateLimitUntil(connectionId, timestamp)	Direct timestamp assignment
clearStaleCrashCooldowns()	Cleanup utility for server startup

Handling Provider 429 Responses

When an upstream provider returns rate-limit errors, persist the back-off:

import { markConnectionRateLimitedUntil } from "@/lib/db/providers/rateLimit";

function handleProvider429(connId: string, retryAfterMs: number) {
  // Stores in provider_connections.rate_limited_until
  markConnectionRateLimitedUntil(connId, retryAfterMs);
}

During request routing (in open-sse/services/combo/), isConnectionRateLimited(connectionId) filters out cooling-down accounts before selection.

Startup Maintenance

Clear stale cooldowns on server bootstrap:

// src/server/startup.ts
import { clearStaleCrashCooldowns } from "@/lib/db/providers/rateLimit";

export async function bootstrap() {
  const { cleared } = clearStaleCrashCooldowns();
  console.info(`[RateLimit] Cleared ${cleared} stale cooldowns`);
  // Continue startup...
}

Quota Tracking: Pre-Flight and Monitor Services

OmniRoute's quota system tracks provider-specific usage limits (tokens, requests, cost) through three coordinated services.

Quota Fetcher Registration

Register provider-specific fetchers in open-sse/services/quotaTrackersBatch.ts (lines 10-31):

// Example pattern from quotaTrackersBatch.ts
registerAgentrouterQuotaFetcher();  // Registers fetcher for AgentRouter provider

Each fetcher returns:

{
  used: number,
  total: number,
  percentUsed: number,
  resetAt: Date
}

Enabling Session-Bound Quota Monitoring

Set quotaMonitorEnabled: true in providerSpecificData:

// src/lib/db/providers/providerConfig.ts
export function enableQuotaMonitor(connectionId: string) {
  const db = getDbInstance();
  db.prepare(`
    UPDATE provider_connections
    SET providerSpecificData = json_set(
      providerSpecificData, 
      '$.quotaMonitorEnabled', 
      true
    )
    WHERE id = ?
  `).run(connectionId);
}

When active sessions use this connection, startQuotaMonitor() (invoked from open-sse/handlers/chatCore.ts) begins automatic polling.

Monitor Behavior (quotaMonitor.ts)

The monitor service (lines 101-124) provides:

• Adaptive polling: Normal → Critical interval based on usage percentage
• Alert suppression: Deduplicated exhaustion warnings via alertSuppression
• State management: MonitorState tracks latest readings

Retrieve snapshots for UI display:

import { getQuotaMonitorSnapshots, getQuotaMonitorSummary } from "@/open-sse/services/quotaMonitor";

const health = getQuotaMonitorSummary(sessionId);

API-Level Rate-Limited Responses

Standardize 429 responses using src/app/api/v1/_shared/rateLimit.ts:

import { rateLimitedProviderResponse } from "@/app/api/v1/_shared/rateLimit";

// When all provider accounts are exhausted
if (allRateLimited) {
  const credentials: RateLimitedCredentials = { 
    allRateLimited: true, 
    retryAfter: retryAfterMs 
  };
  return rateLimitedProviderResponse("openai", credentials);
}

This helper returns properly formatted HTTP 429 or 509 responses with structured JSON bodies for client handling.

Rate Limit Manager and Semaphore

Additional coordination utilities in the open-sse/services/ layer:

• rateLimitManager.ts – Core coordination logic for multi-provider scenarios
• rateLimitSemaphore.ts – MCP-side rate-limit semaphore for concurrent request control

These integrate with the sliding window limiter for cross-cutting rate control across provider pools.

Configuration Checklist

Layer	Configuration Required	Key File
Sliding window	REDIS_URL env var (optional)	src/shared/utils/rateLimiter.ts
Connection persistence	SQLite schema auto-migrated	src/lib/db/providers/rateLimit.ts
Quota monitoring	quotaMonitorEnabled: true in connection config	open-sse/services/quotaMonitor.ts
Provider fetchers	Register in batch file	open-sse/services/quotaTrackersBatch.ts

Summary

• checkRateLimit() in src/shared/utils/rateLimiter.ts enforces sliding-window limits with Redis or in-memory fallback
• Connection-level rate limits persist in SQLite via markConnectionRateLimitedUntil() to survive OAuth refreshes
• Quota trackers combine pre-flight fetchers with optional session polling for real-time exhaustion alerts
• rateLimitedProviderResponse() standardizes HTTP 429 responses across all API routes
• Enable monitoring by setting quotaMonitorEnabled: true in provider connection configuration

Frequently Asked Questions

What's the difference between rate limiting and quota tracking in OmniRoute?

Rate limiting counts requests and rejects excess traffic with HTTP 429, implemented via checkRateLimit() using sliding windows. Quota tracking monitors provider-specific resources (tokens, cost, API calls) and predicts exhaustion before limits are hit. Rate limiting is enforced; quota tracking is monitored and alerted.

Can I use OmniRoute's rate limiter without Redis?

Yes. When REDIS_URL is unset, src/shared/utils/rateLimiter.ts falls back to a per-process Map with automatic eviction of stale windows via evictStaleRateLimitWindows(). This works for single-node deployments but won't share state across server instances.

How does connection-level rate limiting prevent OAuth token issues?

The markConnectionRateLimitedUntil() function in src/lib/db/providers/rateLimit.ts stores the back-off timestamp in the provider_connections table. Unlike memory-only stores, this survives token refreshes that recreate connection objects, ensuring consistent cool-down enforcement.

When should I enable quota monitoring for a provider?

Enable quotaMonitorEnabled when the provider exposes usage APIs and you need predictive exhaustion handling. The monitor in open-sse/services/quotaMonitor.ts polls periodically, emits alerts before hard limits, and adjusts polling frequency based on urgency—ideal for cost-controlled or token-limited APIs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}