# How to Tune OmniRoute's Three-Layer Resilience System: Circuit Breaker, Connection Cooldown, and Model Lockout

> Master OmniRoute's three-layer resilience system. Learn to tune circuit breaker, connection cooldown, and model lockout settings via API, CLI, or env vars for robust failure handling.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: how-to-guide
- Published: 2026-07-19

---

**To tune OmniRoute's three-layer resilience system, modify the `providerBreaker`, `connectionCooldown`, and `modelLockout` configurations in [`src/lib/resilience/settings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/settings.ts) and [`src/lib/resilience/modelLockoutSettings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/modelLockoutSettings.ts) via the REST API, CLI, or environment variables to control failure thresholds, backoff delays, and model-specific error isolation.**

OmniRoute, the open-source AI provider routing layer maintained at diegosouzapw/OmniRoute, protects downstream services through a sophisticated three-layer resilience stack. Tuning these layers—**provider circuit breakers**, **connection cooldowns**, and **model lockouts**—ensures traffic continues flowing during provider degradation, rate limits, or transient network failures.

## The Three-Layer Architecture

OmniRoute's resilience system operates through three distinct mechanisms that work sequentially to handle different failure modes.

### Layer 1: Provider Circuit Breaker

The **provider circuit breaker** stops sending requests to providers that exceed failure thresholds, preventing cascade failures. When the failure count exceeds `failureThreshold`, the breaker opens; after `resetTimeoutMs`, it attempts a retry.

Key configuration parameters in [`src/lib/resilience/settings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/settings.ts) (lines 70-79):

- `providerBreaker.oauth.failureThreshold` – Number of failures before opening the breaker
- `providerBreaker.oauth.resetTimeoutMs` – Milliseconds to wait before attempting recovery  
- `providerBreaker.oauth.degradationThreshold` – Optional threshold for entering degraded mode before full lockout

### Layer 2: Connection Cooldown

The **connection cooldown** layer applies backoff delays after rate-limit (429) or transient errors. This can operate as a fixed delay or exponential backoff based on `maxBackoffSteps`.

Configuration in [`src/lib/resilience/settings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/settings.ts) (lines 58-68):

- `connectionCooldown.oauth.baseCooldownMs` – Initial delay after hitting a rate limit
- `connectionCooldown.oauth.maxBackoffSteps` – Maximum exponential backoff iterations (set to `0` for fixed delays)
- `connectionCooldown.apikey.useExponentialBackoff` – Toggle between linear and exponential backoff

### Layer 3: Model Lockout

**Model lockout** isolates specific models after encountering configurable error codes (e.g., 429, 502) without affecting the entire provider. This prevents a single misbehaving model from triggering a full provider circuit breaker.

Configuration in [`src/lib/resilience/modelLockoutSettings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/modelLockoutSettings.ts) (lines 12-19):

- `modelLockout.enabled` – Master toggle for the layer
- `modelLockout.errorCodes` – Array of HTTP status codes triggering lockout (e.g., `[429, 502, 504]`)
- `modelLockout.baseCooldownMs` – Initial lockout duration
- `modelLockout.maxCooldownMs` – Maximum lockout duration
- `modelLockout.maxBackoffSteps` – Backoff iterations for repeated offenses
- `modelLockout.useExponentialBackoff` – Enable exponential escalation of lockout periods

## Runtime Resolution and Validation

All three layers resolve at request time through `resolveResilienceSettings` in [`src/lib/resilience/settings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/settings.ts) (lines 62-66) and `resolveModelLockoutSettings` in [`src/lib/resilience/modelLockoutSettings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/modelLockoutSettings.ts) (lines 56-94). The system reads current configurations via `getCachedSettings()`, which queries the SQLite `settings` table on every request.

User-provided values undergo validation through the `normalize*` helpers in [`src/lib/resilience/settings/normalize.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/settings/normalize.ts), ensuring parameters like `failureThreshold` and `baseCooldownMs` remain within safe operational ranges.

## Tuning Strategies by Layer

### Adjusting Circuit Breaker Sensitivity

For providers experiencing intermittent instability, reduce the `failureThreshold` in `DEFAULT_RESILIENCE_SETTINGS.providerBreaker` to open the breaker sooner. Increase `resetTimeoutMs` when providers require extended recovery periods after overload conditions.

### Configuring Backoff Behavior

To implement aggressive exponential backoff for rate-limited providers, increase `connectionCooldown.oauth.maxBackoffSteps` and ensure `useExponentialBackoff` is enabled. For consistent, predictable delays, set `maxBackoffSteps` to `0` to use fixed `baseCooldownMs` intervals only.

### Isolating Problematic Models

Add provider-specific error codes to `modelLockout.errorCodes` (such as `511` for network authentication required) when particular models return non-standard errors. Extend `maxCooldownMs` to `3600000` (one hour) for models that cause long-running errors, preventing repeated attempts during maintenance windows.

## Applying Configuration Changes

### Via REST API

Update resilience settings programmatically by sending a PATCH request to `/api/resilience`. The `mergeResilienceSettings` function (lines 18-22 in [`src/lib/resilience/settings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/settings.ts)) merges your payload with existing configurations:

```http
PATCH /api/resilience HTTP/1.1
Content-Type: application/json

{
  "providerBreaker": {
    "oauth": { "failureThreshold": 5, "resetTimeoutMs": 120000 }
  },
  "connectionCooldown": {
    "apikey": { "baseCooldownMs": 30000, "maxBackoffSteps": 5 }
  },
  "modelLockout": {
    "enabled": true,
    "errorCodes": [429, 502, 504],
    "baseCooldownMs": 180000,
    "maxCooldownMs": 3600000,
    "maxBackoffSteps": 8,
    "useExponentialBackoff": true
  }
}

```

### Via CLI

Reset all circuit breakers and model lockouts immediately using the CLI command, which invokes the POST handler at [`src/app/api/resilience/reset/route.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/app/api/resilience/reset/route.ts) (lines 5-38):

```bash
omniroute resilience reset

```

### Via Environment Variables

Enable the legacy provider cooldown layer globally by setting the environment variable, which activates `DEFAULT_RESILIENCE_SETTINGS.providerCooldown.enabled` (lines 101-103 in [`src/lib/resilience/settings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/settings.ts)):

```bash
export PROVIDER_COOLDOWN_ENABLED=true

```

### Programmatic Inspection

Monitor current resilience states using the `inspectTargetResilience` helper from [`src/lib/usage/resilienceExplain.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/usage/resilienceExplain.ts):

```typescript
import { inspectTargetResilience } from "@/lib/usage/resilienceExplain";

const info = await inspectTargetResilience({
  provider: "openai",
  model: "gpt-4o",
  connectionId: "conn-123",
});
console.log(info);

```

This returns the current provider breaker state, active connection cooldown timers, and any model-specific lockouts for diagnostic purposes.

## Summary

- **Circuit Breaker**: Configure `failureThreshold` and `resetTimeoutMs` in [`src/lib/resilience/settings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/settings.ts) to control when OmniRoute stops sending traffic to failing providers.
- **Connection Cooldown**: Adjust `baseCooldownMs` and `maxBackoffSteps` to manage rate-limit recovery behavior with either fixed or exponential delays.
- **Model Lockout**: Use [`src/lib/resilience/modelLockoutSettings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/modelLockoutSettings.ts) to isolate specific models by error code without affecting provider-wide traffic.
- **Runtime Application**: Changes apply via `resolveResilienceSettings` and `resolveModelLockoutSettings`, with persistence through the SQLite-backed `getCachedSettings()` system.
- **Multiple Interfaces**: Tune via REST API (`/api/resilience`), CLI (`omniroute resilience reset`), or environment variables depending on operational requirements.

## Frequently Asked Questions

### How do I disable the model lockout layer entirely?

Set `modelLockout.enabled` to `false` in your configuration payload when calling the PATCH `/api/resilience` endpoint. According to the source in [`src/lib/resilience/modelLockoutSettings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/modelLockoutSettings.ts) (lines 12-19), this boolean acts as a master toggle that bypasses all model-specific isolation logic, causing the system to rely solely on the connection cooldown and circuit breaker layers for error handling.

### What is the difference between connection cooldown and circuit breaker?

The **connection cooldown** ([`src/lib/resilience/settings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/settings.ts) lines 58-68) applies temporary delays after rate-limit or transient errors, allowing the provider to recover while queuing requests. The **circuit breaker** (lines 70-79) completely stops traffic to the provider after `failureThreshold` consecutive failures, requiring a full `resetTimeoutMs` period before attempting recovery. Cooldown handles temporary congestion; the breaker handles sustained failure states.

### How can I implement exponential backoff for a specific provider?

Set `useExponentialBackoff` to `true` and increase `maxBackoffSteps` in either the `connectionCooldown` or `modelLockout` configuration sections. In [`src/lib/resilience/settings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/settings.ts), the system calculates delay as `baseCooldownMs * 2^step` up to the maximum step count, creating progressively longer intervals between retry attempts until the provider recovers or the maximum cooldown duration elapses.

### Where does OmniRoute store resilience configuration changes?

All settings persist in the SQLite `settings` table and are re-read on every request via `getCachedSettings()`. When you update via the REST API at `/api/resilience` (handled in [`src/app/api/resilience/route.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/app/api/resilience/route.ts)), the changes are immediately cached and applied to subsequent routing decisions without requiring a server restart.