How OmniRoute Handles Rate Limiting Across LLM Providers: A Multi-Layer Architecture

OmniRoute handles rate limiting across providers through a three-layer resilience strategy: per-provider circuit breakers, connection-level cooldowns with exponential backoff, and granular model lockouts, all coordinated by a central RateLimitManager that parses upstream 429 responses and Retry-After headers.

The OmniRoute open-source router (GitHub: `diegosouzapw/OmniRoute) implements a sophisticated rate-limiting system that protects both the proxy layer and downstream LLM providers from cascading failures. Rather than treating rate limits as simple errors to retry, the architecture isolates throttled resources at multiple granularities—provider, connection, and model—while maintaining automatic recovery and fallback routing.

---

The Three-Layer Rate Limiting Architecture

OmniRoute's resilience model operates at three distinct layers, each with specific responsibilities and implementation files:

Layer	Scope	Trigger	Implementation Location
Provider Circuit Breaker	Entire provider (all accounts, all models)	Sustained failures (429s, 500s, timeouts)	src/shared/utils/circuitBreaker.ts
Connection Cooldown	Single account/API key	Per-connection 429 or explicit rate-limit error	src/sse/services/auth.ts::markAccountUnavailable()
Model Lockout	Specific model on a specific connection	Model-scoped quota exhaustion	open-sse/services/accountFallback.ts::checkFallbackError()

This layered approach ensures that a rate limit on one model does not unnecessarily block traffic to other models, and a provider-wide issue does not stall the entire routing pipeline.

---

RateLimitManager: Centralized Quota Tracking

The core coordination mechanism is the RateLimitManager, which maintains a semaphore-like abstraction over per-resource quota windows. According to the test suite in tests/unit/rate-limit-manager.test.ts, the manager handles several critical responsibilities:

• Learning from upstream responses: When a provider returns a 429, the manager extracts the Retry-After header or parses response body hints to determine the exact reset timestamp
• Blocking admission: Subsequent requests to the same provider/connection/model combination are rejected until the window expires
• Soft over-limit warnings: The manager can signal approaching limits before hard blocking occurs
• Automatic reset: Expired windows are lazily cleared on read, enabling self-healing without background processes

// From tests/unit/rate-limit-manager.test.ts - demonstrating window learning
it('should block requests after a 429 with Retry-After header', async () => {
  const manager = new RateLimitManager({ store: 'memory' });
  
  // Simulate upstream 429 response
  await manager.recordLimit({
    provider: 'openai',
    connId: 'conn_123',
    model: 'gpt-4',
    retryAfter: 60 // seconds
  });
  
  // Immediately subsequent request should be blocked
  await expect(
    manager.acquire({ provider: 'openai', connId: 'conn_123', model: 'gpt-4' })
  ).rejects.toThrow('Rate limited for 59.9s');
});

The manager supports both in-memory storage (default for single-node deployments) and Redis-backed storage (when REDIS_URL is configured), enabling consistent rate-limit state across horizontally scaled instances.

---

Detecting and Classifying Rate Limit Errors

Before the RateLimitManager can act, OmniRoute must accurately identify rate-limit conditions from heterogeneous provider responses. The classifyFailure utility—exercised in tests/unit/failure-class.test.ts—recognizes rate limits through multiple signals:

• HTTP 429 status code (the standard "Too Many Requests" indicator)
• Error message patterns: "rate limit exceeded", "quota reached", "too many requests", "throttled", "capacity exceeded"
• Provider-specific headers: x-ratelimit-remaining, x-ratelimit-reset, x-request-throttled

// Pattern matching for rate limit detection (from failure classification tests)
const RATE_LIMIT_PATTERNS = [
  /rate\s*limit/i,
  /quota\s*(reached|exceeded)/i,
  /too\s*many\s*requests/i,
  /throttled/i,
  /capacity\s*exceeded/i
];

export function classifyFailure(response: Response, body: any): FailureType {
  if (response.status === 429) return 'RATE_LIMIT';
  
  const message = extractErrorMessage(body);
  if (RATE_LIMIT_PATTERNS.some(p => p.test(message))) {
    return 'RATE_LIMIT';
  }
  // ... other failure classifications
}

This classification determines which resilience layer activates: a 429 with explicit retry guidance triggers connection cooldown; sustained 429s across multiple connections trip the provider circuit breaker.

---

Connection Cooldown and Exponential Backoff

When a specific account encounters a rate limit, OmniRoute applies connection-level cooldown rather than punishing the entire provider. The markAccountUnavailable function in src/sse/services/auth.ts implements this with configurable backoff strategies:

• OAuth-based providers: Base cooldown of 5 seconds, doubling with each consecutive failure (5s, 10s, 20s, 40s...)
• API-key providers: Base cooldown of 3 seconds with identical exponential growth
• Maximum backoff ceiling: 300 seconds (5 minutes) to prevent indefinite exclusion

The cooldown state is stored in the rateLimitedUntil field of the connection record, checked on every routing decision:

// From src/sse/services/auth.ts - cooldown application
function markAccountUnavailable(
  connId: string,
  statusCode: number,
  reason: string,
  suggestedBackoff?: number
): void {
  const conn = getConnection(connId);
  const failureIndex = conn.consecutiveFailures;
  
  const baseMs = conn.authType === 'oauth' ? 5000 : 3000;
  const backoffMs = suggestedBackoff ?? Math.min(
    baseMs * Math.pow(2, failureIndex),
    300_000 // 5 minute ceiling
  );
  
  conn.rateLimitedUntil = Date.now() + backoffMs;
  conn.consecutiveFailures += 1;
  
  logger.warn({
    connId,
    backoffMs,
    until: new Date(conn.rateLimitedUntil).toISOString()
  }, 'Connection rate limited');
}

The open-sse/services/accountFallback.ts::checkFallbackError() routine then uses this state to exclude throttled connections from the routing pool while preserving healthy alternatives.

---

Model Lockout: Granular Quota Isolation

Some providers impose model-specific quotas distinct from account-level limits. OmniRoute's model lockout mechanism scopes rate-limit state to the triple of (provider, connection, model), ensuring that quota exhaustion on gpt-4 does not affect gpt-3.5-turbo requests through the same account.

This granularity is particularly important for providers like OpenRouter or grouped API keys that multiplex multiple underlying models. The implementation reuses the same cooldown infrastructure but maintains separate rateLimitedUntil entries per model in the connection's state map.

---

Integration with Combo Routing and Fallback

OmniRoute's advanced Combo routing engine integrates rate-limit awareness directly into target selection. As documented in tests/unit/ui/comboFlowModel.test.ts, when building a combo execution plan:

1. Each candidate target queries the RateLimitManager for admission
2. Permits are acquired synchronously before request dispatch
3. If acquisition fails, the target is skipped and the combo strategy evaluates alternatives
4. Persistent failures across all targets trigger the "All accounts rate limited" error response

// Combo target filtering with rate limit awareness (conceptual from combo tests)
function buildComboTargets(candidates: Target[], strategy: ComboStrategy): Target[] {
  const available = candidates.filter(t => {
    if (rateLimitManager.isLimited(t.provider, t.connId, t.model)) {
      strategy.addSkipReason(t, 'rate_limited');
      return false;
    }
    return true;
  });
  
  if (available.length === 0) {
    throw new AllAccountsRateLimitedError();
  }
  
  return strategy.select(available);
}

This integration enables graceful degradation: a rate-limited premium model automatically falls back to a standard alternative without client intervention.

---

Circuit Breaker: Provider-Level Protection

When rate limits and other failures cluster across multiple connections to the same provider, the Provider Circuit Breaker—implemented in src/shared/utils/circuitBreaker.ts—opens to halt all traffic. This prevents:

• Retry storms that exacerbate provider throttling
• Wasted latency on doomed requests
• Cascading overload to healthy providers

The breaker monitors failure rates across a sliding window (default: 60 seconds) and opens when errors exceed 50% with a minimum of 5 requests. Recovery uses a half-open state that probes with single requests before resuming full traffic.

The circuit breaker state is exposed through the monitoring API (src/app/api/monitoring/health/route.ts) for operational visibility.

---

Configuration and Defaults

Default rate-limit parameters are centralized in src/lib/resilience/settings.ts:

Parameter	Default Value	Description
oauthBaseCooldownMs	5000	Initial backoff for OAuth connections
apiKeyBaseCooldownMs	3000	Initial backoff for API-key connections
maxCooldownMs	300000	Absolute ceiling on any cooldown
circuitBreakerWindowMs	60000	Sliding window for failure rate calculation
circuitBreakerThreshold	0.5	Error ratio that triggers breaker open
circuitBreakerMinRequests	5	Minimum samples before breaker can open

These values can be overridden via environment variables or the runtime configuration API.

---

Persistent State and Self-Healing

Rate-limit state survives process restarts through SQLite persistence in the domain_rate_limiters table. Each entry stores:

• Composite key: (provider, connection_id, model)
• limited_until: Timestamp when the resource becomes available
• limit_reason: Classification (429, quota, throttled, etc.)
• created_at: Record creation time for audit

Expired entries are pruned lazily—when the RateLimitManager encounters a stored limit, it checks against current time and removes stale records. This eliminates the need for background garbage collection while ensuring accurate state after extended outages.

---

Summary

• OmniRoute implements three-layer rate limiting: provider circuit breakers, connection cooldowns, and model lockouts
• RateLimitManager centralizes quota tracking with support for in-memory and Redis storage, parsing upstream 429 responses and Retry-After headers
• Exponential backoff applies per-connection with provider-specific base values and a 5-minute ceiling
• Combo routing integration enables automatic fallback when targets are rate-limited, surfacing "All accounts rate limited" only when all options are exhausted
• Self-healing design uses lazy expiration of limit windows and persistent SQLite storage for state durability

---

Frequently Asked Questions

What happens when all providers are rate limited simultaneously?

When every available provider has active rate limits, OmniRoute returns an AllAccountsRateLimitedError with HTTP 503 status. The response includes a Retry-After header set to the minimum remaining window across all providers, allowing clients to implement intelligent backoff. This scenario is explicitly tested in tests/unit/route-edge-coverage.test.ts.

Does OmniRoute support custom rate limit policies per provider?

Yes. Provider-specific overrides can be configured in the resilience section of each provider definition. You can set custom baseCooldownMs, maxCooldownMs, and circuitBreaker parameters that take precedence over global defaults. These are loaded from src/lib/resilience/settings.ts at initialization.

How does OmniRoute distinguish between temporary rate limits and permanent quota exhaustion?

The system uses heuristic classification on error messages. Phrases like "monthly quota exceeded" or "billing limit reached" trigger a permanent lockout requiring manual intervention, while "rate limit exceeded" or "too many requests" trigger the standard temporary cooldown with automatic recovery. Permanent lockouts bypass the normal expiration and must be cleared through the admin API or database update.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}