How to Tune OmniRoute's Three-Layer Resilience System: Circuit Breaker, Connection Cooldown, and Model Lockout

To tune OmniRoute's three-layer resilience system, modify the providerBreaker, connectionCooldown, and modelLockout configurations in src/lib/resilience/settings.ts and src/lib/resilience/modelLockoutSettings.ts via the REST API, CLI, or environment variables to control failure thresholds, backoff delays, and model-specific error isolation.

OmniRoute, the open-source AI provider routing layer maintained at diegosouzapw/OmniRoute, protects downstream services through a sophisticated three-layer resilience stack. Tuning these layers—provider circuit breakers, connection cooldowns, and model lockouts—ensures traffic continues flowing during provider degradation, rate limits, or transient network failures.

The Three-Layer Architecture

OmniRoute's resilience system operates through three distinct mechanisms that work sequentially to handle different failure modes.

Layer 1: Provider Circuit Breaker

The provider circuit breaker stops sending requests to providers that exceed failure thresholds, preventing cascade failures. When the failure count exceeds failureThreshold, the breaker opens; after resetTimeoutMs, it attempts a retry.

Key configuration parameters in src/lib/resilience/settings.ts (lines 70-79):

• providerBreaker.oauth.failureThreshold – Number of failures before opening the breaker
• providerBreaker.oauth.resetTimeoutMs – Milliseconds to wait before attempting recovery
• providerBreaker.oauth.degradationThreshold – Optional threshold for entering degraded mode before full lockout

Layer 2: Connection Cooldown

The connection cooldown layer applies backoff delays after rate-limit (429) or transient errors. This can operate as a fixed delay or exponential backoff based on maxBackoffSteps.

Configuration in src/lib/resilience/settings.ts (lines 58-68):

• connectionCooldown.oauth.baseCooldownMs – Initial delay after hitting a rate limit
• connectionCooldown.oauth.maxBackoffSteps – Maximum exponential backoff iterations (set to 0 for fixed delays)
• connectionCooldown.apikey.useExponentialBackoff – Toggle between linear and exponential backoff

Layer 3: Model Lockout

Model lockout isolates specific models after encountering configurable error codes (e.g., 429, 502) without affecting the entire provider. This prevents a single misbehaving model from triggering a full provider circuit breaker.

Configuration in src/lib/resilience/modelLockoutSettings.ts (lines 12-19):

• modelLockout.enabled – Master toggle for the layer
• modelLockout.errorCodes – Array of HTTP status codes triggering lockout (e.g., [429, 502, 504])
• modelLockout.baseCooldownMs – Initial lockout duration
• modelLockout.maxCooldownMs – Maximum lockout duration
• modelLockout.maxBackoffSteps – Backoff iterations for repeated offenses
• modelLockout.useExponentialBackoff – Enable exponential escalation of lockout periods

Runtime Resolution and Validation

All three layers resolve at request time through resolveResilienceSettings in src/lib/resilience/settings.ts (lines 62-66) and resolveModelLockoutSettings in src/lib/resilience/modelLockoutSettings.ts (lines 56-94). The system reads current configurations via getCachedSettings(), which queries the SQLite settings table on every request.

User-provided values undergo validation through the normalize* helpers in src/lib/resilience/settings/normalize.ts, ensuring parameters like failureThreshold and baseCooldownMs remain within safe operational ranges.

Tuning Strategies by Layer

Adjusting Circuit Breaker Sensitivity

For providers experiencing intermittent instability, reduce the failureThreshold in DEFAULT_RESILIENCE_SETTINGS.providerBreaker to open the breaker sooner. Increase resetTimeoutMs when providers require extended recovery periods after overload conditions.

Configuring Backoff Behavior

To implement aggressive exponential backoff for rate-limited providers, increase connectionCooldown.oauth.maxBackoffSteps and ensure useExponentialBackoff is enabled. For consistent, predictable delays, set maxBackoffSteps to 0 to use fixed baseCooldownMs intervals only.

Isolating Problematic Models

Add provider-specific error codes to modelLockout.errorCodes (such as 511 for network authentication required) when particular models return non-standard errors. Extend maxCooldownMs to 3600000 (one hour) for models that cause long-running errors, preventing repeated attempts during maintenance windows.

Applying Configuration Changes

Via REST API

Update resilience settings programmatically by sending a PATCH request to /api/resilience. The mergeResilienceSettings function (lines 18-22 in src/lib/resilience/settings.ts) merges your payload with existing configurations:

PATCH /api/resilience HTTP/1.1
Content-Type: application/json

{
  "providerBreaker": {
    "oauth": { "failureThreshold": 5, "resetTimeoutMs": 120000 }
  },
  "connectionCooldown": {
    "apikey": { "baseCooldownMs": 30000, "maxBackoffSteps": 5 }
  },
  "modelLockout": {
    "enabled": true,
    "errorCodes": [429, 502, 504],
    "baseCooldownMs": 180000,
    "maxCooldownMs": 3600000,
    "maxBackoffSteps": 8,
    "useExponentialBackoff": true
  }
}

Via CLI

Reset all circuit breakers and model lockouts immediately using the CLI command, which invokes the POST handler at src/app/api/resilience/reset/route.ts (lines 5-38):

omniroute resilience reset

Via Environment Variables

Enable the legacy provider cooldown layer globally by setting the environment variable, which activates DEFAULT_RESILIENCE_SETTINGS.providerCooldown.enabled (lines 101-103 in src/lib/resilience/settings.ts):

export PROVIDER_COOLDOWN_ENABLED=true

Programmatic Inspection

Monitor current resilience states using the inspectTargetResilience helper from src/lib/usage/resilienceExplain.ts:

import { inspectTargetResilience } from "@/lib/usage/resilienceExplain";

const info = await inspectTargetResilience({
  provider: "openai",
  model: "gpt-4o",
  connectionId: "conn-123",
});
console.log(info);

This returns the current provider breaker state, active connection cooldown timers, and any model-specific lockouts for diagnostic purposes.

Summary

• Circuit Breaker: Configure failureThreshold and resetTimeoutMs in src/lib/resilience/settings.ts to control when OmniRoute stops sending traffic to failing providers.
• Connection Cooldown: Adjust baseCooldownMs and maxBackoffSteps to manage rate-limit recovery behavior with either fixed or exponential delays.
• Model Lockout: Use src/lib/resilience/modelLockoutSettings.ts to isolate specific models by error code without affecting provider-wide traffic.
• Runtime Application: Changes apply via resolveResilienceSettings and resolveModelLockoutSettings, with persistence through the SQLite-backed getCachedSettings() system.
• Multiple Interfaces: Tune via REST API (/api/resilience), CLI (omniroute resilience reset), or environment variables depending on operational requirements.

Frequently Asked Questions

How do I disable the model lockout layer entirely?

Set modelLockout.enabled to false in your configuration payload when calling the PATCH /api/resilience endpoint. According to the source in src/lib/resilience/modelLockoutSettings.ts (lines 12-19), this boolean acts as a master toggle that bypasses all model-specific isolation logic, causing the system to rely solely on the connection cooldown and circuit breaker layers for error handling.

What is the difference between connection cooldown and circuit breaker?

The connection cooldown (src/lib/resilience/settings.ts lines 58-68) applies temporary delays after rate-limit or transient errors, allowing the provider to recover while queuing requests. The circuit breaker (lines 70-79) completely stops traffic to the provider after failureThreshold consecutive failures, requiring a full resetTimeoutMs period before attempting recovery. Cooldown handles temporary congestion; the breaker handles sustained failure states.

How can I implement exponential backoff for a specific provider?

Set useExponentialBackoff to true and increase maxBackoffSteps in either the connectionCooldown or modelLockout configuration sections. In src/lib/resilience/settings.ts, the system calculates delay as baseCooldownMs * 2^step up to the maximum step count, creating progressively longer intervals between retry attempts until the provider recovers or the maximum cooldown duration elapses.

Where does OmniRoute store resilience configuration changes?

All settings persist in the SQLite settings table and are re-read on every request via getCachedSettings(). When you update via the REST API at /api/resilience (handled in src/app/api/resilience/route.ts), the changes are immediately cached and applied to subsequent routing decisions without requiring a server restart.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}