How to Configure Provider-Specific Rate Limits and Circuit Breaker Thresholds in OmniRoute

OmniRoute lets you configure provider-specific rate limits and circuit breaker thresholds through the /api/resilience PATCH endpoint, with defaults defined in src/lib/resilience/settings.ts and runtime enforcement handled by src/shared/utils/circuitBreaker.ts and src/shared/utils/rateLimitSemaphore.ts.

OmniRoute is an open-source LLM routing layer that isolates upstream failures through per-provider resilience controls. You can configure provider-specific rate limits and circuit breaker thresholds to prevent cascading outages and honor provider throttling without redeploying the service. The system stores type-based defaults and accepts live overrides through a dedicated REST API.

Default Provider Resilience Profiles

OmniRoute defines default thresholds per authentication type in src/lib/resilience/settings.ts. The PROVIDER_PROFILES object supplies fallback values when a connection record lacks its own configuration:

// src/lib/resilience/settings.ts
export const PROVIDER_PROFILES = {
  oauth: {
    circuitBreakerThreshold: 5,   // fail after 5 consecutive errors
    circuitBreakerReset: 30_000, // stay open 30 s before half-open trial
  },
  apikey: {
    circuitBreakerThreshold: 10,
    circuitBreakerReset: 60_000,
  },
};

These defaults differentiate OAuth connections, which use a lower failure tolerance, from API-key connections, which tolerate more errors before opening the circuit. If a provider record does not specify circuitBreakerThreshold or circuitBreakerReset, the routing layer falls back to the matching profile above.

Overriding Per-Provider Thresholds via the Resilience API

You can inspect and mutate thresholds at runtime through the /api/resilience endpoint. A GET request returns the current global and per-provider settings, while a PATCH request merges a partial payload into the live configuration.

The endpoint lives in src/app/api/resilience/route.ts. When it receives a PATCH, it merges the incoming JSON and immediately refreshes the in-memory utilities:

// src/app/api/resilience/route.ts
import { getCircuitBreaker } from "@/shared/utils/circuitBreaker";

export async function PATCH(req) {
  const body = await req.json();
  // merge logic → SETTINGS.providerBreaker[provider] = …
  // refresh in-memory circuit-breaker caches
  const breaker = getCircuitBreaker();
  breaker.updateProviderSettings(providerId, body.providerBreaker[providerId]);
}

A minimal payload adjusts both circuit-breaker and rate-limit behavior for a single provider:

{
  "providerBreaker": {
    "my-provider-id": {
      "failureThreshold": 3,
      "resetTimeoutMs": 15_000
    }
  },
  "rateLimit429": {
    "threshold": 1,
    "windowMs": 120_000,
    "cooldownMs": 10_000
  }
}

The same endpoint accepts rateLimit429 configuration, which propagates to src/shared/utils/rateLimitSemaphore.ts. Changes take effect immediately without restarting the service.

To update a provider's circuit breaker from the command line:

curl -X PATCH https://my.omniroute.local/api/resilience \
  -H "Content-Type: application/json" \
  -d '{
    "providerBreaker": {
      "openai": { "failureThreshold": 4, "resetTimeoutMs": 20000 }
    }
  }'

Runtime Interaction Between Circuit Breakers and Rate Limits

Before every upstream request, OmniRoute evaluates both controls. The check occurs in src/sse/handlers/chatHelpers.ts (called from src/sse/handlers/chat.ts):

// src/sse/handlers/chatHelpers.ts
if (circuitBreaker.isOpen(providerId)) {
  // short-circuit: return a "circuit-open" error without upstream call
}
if (rateLimitSemaphore.isRateLimited(modelId)) {
  // short-circuit: return a "rate-limited" error with Retry-After header
}

Rate-limit cooldown blocks a provider after receiving an HTTP 429 or any rate-limited response. The semaphore stores a rateLimitedUntil timestamp per model and queues excess calls. Circuit-breaker logic counts consecutive failures—such as network errors, 5xx responses, or explicit "circuit open" messages—and opens the circuit after the configured threshold. While open, the provider is bypassed entirely until the reset timeout expires.

The circuit breaker takes precedence over the rate-limit check. In classifyFailKind(), a message containing both "429" and "circuit open" is classified as a circuit-open event, a behavior validated by tests/unit/ui/comboFlowModel.test.ts.

Persistence and Observability

Rate-limit timestamps are stored as rateLimitedUntil values in the connections table managed by src/lib/db/connections.ts. Circuit-breaker state is maintained in an in-memory map populated at startup and refreshed on every successful request. Stale cooldowns and failure counters are cleared automatically when traffic resumes.

For real-time visibility, the monitoring API exposes the current circuit state via src/lib/monitoring/providerHealthMatrix.ts. Query the health endpoint to inspect provider status:

curl https://my.omniroute.local/api/monitoring/health

The response includes a per-provider circuitBreaker field:

{
  "provider": "openai",
  "circuitBreaker": { "state": "CLOSED", "failureCount": 0 }
}

You can also force a rate-limit cooldown programmatically for testing:

import { rateLimitSemaphore } from "@/shared/utils/rateLimitSemaphore";

await rateLimitSemaphore.markRateLimited("gpt-4", 30_000); // 30 s cooldown

Summary

• Default profiles in src/lib/resilience/settings.ts supply OAuth and API-key circuit-breaker baselines.
• Live overrides are applied through PATCH /api/resilience, which updates circuitBreaker and rateLimitSemaphore in memory without a restart.
• Request guards in src/sse/handlers/chatHelpers.ts check the circuit breaker first, then the rate-limit semaphore, with circuit state taking precedence.
• Observability is available through /api/monitoring/health, backed by src/lib/monitoring/providerHealthMatrix.ts.

Frequently Asked Questions

How do I change the default circuit-breaker threshold for all OAuth providers?

Edit the oauth block inside PROVIDER_PROFILES in src/lib/resilience/settings.ts and redeploy. Any connection record without its own threshold will inherit the new default value.

What happens when both rate-limit and circuit-breaker conditions are met?

The circuit breaker wins. According to the classifyFailKind() logic tested in tests/unit/ui/comboFlowModel.test.ts, a response payload that contains both "429" and "circuit open" signals is treated as a circuit-open event, and the request is short-circuited before the rate-limit check.

Does updating /api/resilience require a server restart?

No. The PATCH handler in src/app/api/resilience/route.ts merges the incoming payload into the live configuration and calls breaker.updateProviderSettings() directly, so in-memory state reflects the new thresholds immediately.

Where is rate-limit cooldown data stored?

The rateLimitedUntil timestamp for each model is persisted in the connections table via src/lib/db/connections.ts, while the active semaphore state is held in memory inside src/shared/utils/rateLimitSemaphore.ts. Successful requests automatically clear stale cooldowns.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}