# How OmniRoute Handles Rate Limiting Across LLM Providers: A Multi-Layer Architecture > OmniRoute implements robust rate limiting for LLM providers using a multi-layer architecture. Discover its circuit breakers, cooldowns, and model lockouts for seamless API interactions. - Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute) - Tags: architecture - Published: 2026-08-07 --- **OmniRoute handles rate limiting across providers through a three-layer resilience strategy: per-provider circuit breakers, connection-level cooldowns with exponential backoff, and granular model lockouts, all coordinated by a central RateLimitManager that parses upstream 429 responses and Retry-After headers.** The OmniRoute open-source router (GitHub: `diegosouzapw/OmniRoute) implements a sophisticated rate-limiting system that protects both the proxy layer and downstream LLM providers from cascading failures. Rather than treating rate limits as simple errors to retry, the architecture isolates throttled resources at multiple granularities—provider, connection, and model—while maintaining automatic recovery and fallback routing. --- ## The Three-Layer Rate Limiting Architecture OmniRoute's resilience model operates at three distinct layers, each with specific responsibilities and implementation files: | Layer | Scope | Trigger | Implementation Location | |-------|-------|---------|------------------------| | **Provider Circuit Breaker** | Entire provider (all accounts, all models) | Sustained failures (429s, 500s, timeouts) | [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts) | | **Connection Cooldown** | Single account/API key | Per-connection 429 or explicit rate-limit error | `src/sse/services/auth.ts::markAccountUnavailable()` | | **Model Lockout** | Specific model on a specific connection | Model-scoped quota exhaustion | `open-sse/services/accountFallback.ts::checkFallbackError()` | This layered approach ensures that a rate limit on one model does not unnecessarily block traffic to other models, and a provider-wide issue does not stall the entire routing pipeline. --- ## RateLimitManager: Centralized Quota Tracking The core coordination mechanism is the **RateLimitManager**, which maintains a semaphore-like abstraction over per-resource quota windows. According to the test suite in [`tests/unit/rate-limit-manager.test.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/tests/unit/rate-limit-manager.test.ts), the manager handles several critical responsibilities: - **Learning from upstream responses**: When a provider returns a 429, the manager extracts the `Retry-After` header or parses response body hints to determine the exact reset timestamp - **Blocking admission**: Subsequent requests to the same provider/connection/model combination are rejected until the window expires - **Soft over-limit warnings**: The manager can signal approaching limits before hard blocking occurs - **Automatic reset**: Expired windows are lazily cleared on read, enabling self-healing without background processes ```typescript // From tests/unit/rate-limit-manager.test.ts - demonstrating window learning it('should block requests after a 429 with Retry-After header', async () => { const manager = new RateLimitManager({ store: 'memory' }); // Simulate upstream 429 response await manager.recordLimit({ provider: 'openai', connId: 'conn_123', model: 'gpt-4', retryAfter: 60 // seconds }); // Immediately subsequent request should be blocked await expect( manager.acquire({ provider: 'openai', connId: 'conn_123', model: 'gpt-4' }) ).rejects.toThrow('Rate limited for 59.9s'); }); ``` The manager supports both **in-memory storage** (default for single-node deployments) and **Redis-backed storage** (when `REDIS_URL` is configured), enabling consistent rate-limit state across horizontally scaled instances. --- ## Detecting and Classifying Rate Limit Errors Before the RateLimitManager can act, OmniRoute must accurately identify rate-limit conditions from heterogeneous provider responses. The `classifyFailure` utility—exercised in [`tests/unit/failure-class.test.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/tests/unit/failure-class.test.ts)—recognizes rate limits through multiple signals: - **HTTP 429 status code** (the standard "Too Many Requests" indicator) - **Error message patterns**: "rate limit exceeded", "quota reached", "too many requests", "throttled", "capacity exceeded" - **Provider-specific headers**: `x-ratelimit-remaining`, `x-ratelimit-reset`, `x-request-throttled` ```typescript // Pattern matching for rate limit detection (from failure classification tests) const RATE_LIMIT_PATTERNS = [ /rate\s*limit/i, /quota\s*(reached|exceeded)/i, /too\s*many\s*requests/i, /throttled/i, /capacity\s*exceeded/i ]; export function classifyFailure(response: Response, body: any): FailureType { if (response.status === 429) return 'RATE_LIMIT'; const message = extractErrorMessage(body); if (RATE_LIMIT_PATTERNS.some(p => p.test(message))) { return 'RATE_LIMIT'; } // ... other failure classifications } ``` This classification determines which resilience layer activates: a 429 with explicit retry guidance triggers connection cooldown; sustained 429s across multiple connections trip the provider circuit breaker. --- ## Connection Cooldown and Exponential Backoff When a specific account encounters a rate limit, OmniRoute applies **connection-level cooldown** rather than punishing the entire provider. The `markAccountUnavailable` function in [`src/sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/auth.ts) implements this with configurable backoff strategies: - **OAuth-based providers**: Base cooldown of 5 seconds, doubling with each consecutive failure (`5s, 10s, 20s, 40s...`) - **API-key providers**: Base cooldown of 3 seconds with identical exponential growth - **Maximum backoff ceiling**: 300 seconds (5 minutes) to prevent indefinite exclusion The cooldown state is stored in the `rateLimitedUntil` field of the connection record, checked on every routing decision: ```typescript // From src/sse/services/auth.ts - cooldown application function markAccountUnavailable( connId: string, statusCode: number, reason: string, suggestedBackoff?: number ): void { const conn = getConnection(connId); const failureIndex = conn.consecutiveFailures; const baseMs = conn.authType === 'oauth' ? 5000 : 3000; const backoffMs = suggestedBackoff ?? Math.min( baseMs * Math.pow(2, failureIndex), 300_000 // 5 minute ceiling ); conn.rateLimitedUntil = Date.now() + backoffMs; conn.consecutiveFailures += 1; logger.warn({ connId, backoffMs, until: new Date(conn.rateLimitedUntil).toISOString() }, 'Connection rate limited'); } ``` The `open-sse/services/accountFallback.ts::checkFallbackError()` routine then uses this state to exclude throttled connections from the routing pool while preserving healthy alternatives. --- ## Model Lockout: Granular Quota Isolation Some providers impose model-specific quotas distinct from account-level limits. OmniRoute's **model lockout** mechanism scopes rate-limit state to the triple of `(provider, connection, model)`, ensuring that quota exhaustion on `gpt-4` does not affect `gpt-3.5-turbo` requests through the same account. This granularity is particularly important for providers like OpenRouter or grouped API keys that multiplex multiple underlying models. The implementation reuses the same cooldown infrastructure but maintains separate `rateLimitedUntil` entries per model in the connection's state map. --- ## Integration with Combo Routing and Fallback OmniRoute's advanced **Combo routing engine** integrates rate-limit awareness directly into target selection. As documented in [`tests/unit/ui/comboFlowModel.test.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/tests/unit/ui/comboFlowModel.test.ts), when building a combo execution plan: 1. Each candidate target queries the RateLimitManager for admission 2. Permits are acquired synchronously before request dispatch 3. If acquisition fails, the target is skipped and the combo strategy evaluates alternatives 4. Persistent failures across all targets trigger the "All accounts rate limited" error response ```typescript // Combo target filtering with rate limit awareness (conceptual from combo tests) function buildComboTargets(candidates: Target[], strategy: ComboStrategy): Target[] { const available = candidates.filter(t => { if (rateLimitManager.isLimited(t.provider, t.connId, t.model)) { strategy.addSkipReason(t, 'rate_limited'); return false; } return true; }); if (available.length === 0) { throw new AllAccountsRateLimitedError(); } return strategy.select(available); } ``` This integration enables graceful degradation: a rate-limited premium model automatically falls back to a standard alternative without client intervention. --- ## Circuit Breaker: Provider-Level Protection When rate limits and other failures cluster across multiple connections to the same provider, the **Provider Circuit Breaker**—implemented in [`src/shared/utils/circuitBreaker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/utils/circuitBreaker.ts)—opens to halt all traffic. This prevents: - **Retry storms** that exacerbate provider throttling - **Wasted latency** on doomed requests - **Cascading overload** to healthy providers The breaker monitors failure rates across a sliding window (default: 60 seconds) and opens when errors exceed 50% with a minimum of 5 requests. Recovery uses a half-open state that probes with single requests before resuming full traffic. The circuit breaker state is exposed through the monitoring API ([`src/app/api/monitoring/health/route.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/app/api/monitoring/health/route.ts)) for operational visibility. --- ## Configuration and Defaults Default rate-limit parameters are centralized in [`src/lib/resilience/settings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/settings.ts): | Parameter | Default Value | Description | |-----------|-------------|-------------| | `oauthBaseCooldownMs` | 5000 | Initial backoff for OAuth connections | | `apiKeyBaseCooldownMs` | 3000 | Initial backoff for API-key connections | | `maxCooldownMs` | 300000 | Absolute ceiling on any cooldown | | `circuitBreakerWindowMs` | 60000 | Sliding window for failure rate calculation | | `circuitBreakerThreshold` | 0.5 | Error ratio that triggers breaker open | | `circuitBreakerMinRequests` | 5 | Minimum samples before breaker can open | These values can be overridden via environment variables or the runtime configuration API. --- ## Persistent State and Self-Healing Rate-limit state survives process restarts through SQLite persistence in the `domain_rate_limiters` table. Each entry stores: - Composite key: `(provider, connection_id, model)` - `limited_until`: Timestamp when the resource becomes available - `limit_reason`: Classification (429, quota, throttled, etc.) - `created_at`: Record creation time for audit Expired entries are pruned lazily—when the RateLimitManager encounters a stored limit, it checks against current time and removes stale records. This eliminates the need for background garbage collection while ensuring accurate state after extended outages. --- ## Summary - **OmniRoute implements three-layer rate limiting**: provider circuit breakers, connection cooldowns, and model lockouts - **RateLimitManager** centralizes quota tracking with support for in-memory and Redis storage, parsing upstream 429 responses and Retry-After headers - **Exponential backoff** applies per-connection with provider-specific base values and a 5-minute ceiling - **Combo routing integration** enables automatic fallback when targets are rate-limited, surfacing "All accounts rate limited" only when all options are exhausted - **Self-healing design** uses lazy expiration of limit windows and persistent SQLite storage for state durability --- ## Frequently Asked Questions ### What happens when all providers are rate limited simultaneously? When every available provider has active rate limits, OmniRoute returns an `AllAccountsRateLimitedError` with HTTP 503 status. The response includes a `Retry-After` header set to the minimum remaining window across all providers, allowing clients to implement intelligent backoff. This scenario is explicitly tested in [`tests/unit/route-edge-coverage.test.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/tests/unit/route-edge-coverage.test.ts). ### Does OmniRoute support custom rate limit policies per provider? Yes. Provider-specific overrides can be configured in the `resilience` section of each provider definition. You can set custom `baseCooldownMs`, `maxCooldownMs`, and `circuitBreaker` parameters that take precedence over global defaults. These are loaded from [`src/lib/resilience/settings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/settings.ts) at initialization. ### How does OmniRoute distinguish between temporary rate limits and permanent quota exhaustion? The system uses heuristic classification on error messages. Phrases like "monthly quota exceeded" or "billing limit reached" trigger a **permanent lockout** requiring manual intervention, while "rate limit exceeded" or "too many requests" trigger the standard **temporary cooldown** with automatic recovery. Permanent lockouts bypass the normal expiration and must be cleared through the admin API or database update.