# How Does Model Lockout in OmniRoute Isolate Per-Model Failures?

> Learn how OmniRoute's model lockout isolates failures by tracking cooldowns per model, ensuring other models stay operational during routing errors.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: internals
- Published: 2026-08-20

---

**OmniRoute's model lockout mechanism isolates per‑model failures by tracking error‑code‑triggered cooldowns at the provider‑connection‑model scope, automatically skipping locked models during routing while keeping other models on the same credential fully operational.**

Model lockout is a core **resilience** feature in the OmniRoute routing engine (diegosouzapw/OmniRoute). It prevents a single misbehaving AI model—whether hitting rate limits, quota exhaustion, or temporary outages—from degrading the entire provider connection or forcing a complete combo routing failure. This article explains the architectural flow, configuration options, and source code implementation that make granular failure isolation possible.

## Core Architecture of Model Lockout

### Scope and Granularity

Unlike coarse‑grained circuit breakers that disable entire providers, model lockout operates at three nested levels:

- **Provider** – The upstream service (e.g., OpenAI, Anthropic)
- **Connection** – A specific credential (OAuth token or API key)
- **Model** – The individual model identifier (e.g., `gpt-4-turbo`, `claude-3-opus-20240229`)

This means a 429 error on `gpt-4-turbo` for one API key does **not** affect `gpt-3.5-turbo` on that same key, nor does it impact other connections to the same provider.

### Lockout State Structure

When triggered, a lockout entry is stored on the connection object with these fields:

| Field | Purpose |
|-------|---------|
| `remainingMs` | Milliseconds until the model becomes eligible again |
| `reason` | Human‑readable cause (e.g., *quota‑exhausted*, *rate‑limited*) |
| `consecutiveFailures` | Counter for exponential back‑off calculation |

The entry is keyed by `<providerId>::<modelName>` in the connection's `modelLockouts` Map.

## How Model Lockout Is Triggered and Enforced

### Step 1: Configuration Resolution

Every request begins by resolving active lockout settings. In [`src/lib/resilience/modelLockoutSettings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/modelLockoutSettings.ts), the `resolveModelLockoutSettings` function parses global configuration:

```typescript
// Excerpt pattern from src/sse/services/auth.ts
const mlSettings = resolveModelLockoutSettings(await getCachedSettings());

```

This returns a normalized structure including:
- `enabled: boolean`
- `errorCodes: number[]` – Which HTTP status codes trigger lockout (default: `[429, 503]`)
- `baseCooldownMs: number` – Initial back‑off duration
- `maxCooldownMs: number` – Ceiling for back‑off
- `exponentialBackoff: boolean` – Whether to double duration per consecutive failure
- `maxBackoffSteps: number` – Maximum doublings before hitting ceiling

### Step 2: Error Detection and State Creation

After an upstream error response, the account‑fallback logic in the SSE layer evaluates whether the status code matches the configured lockout list. If matched, it calculates cooldown duration:

```

cooldown = exponentialBackoff
  ? min(baseCooldownMs × 2^consecutiveFailures, maxCooldownMs)
  : baseCooldownMs

```

The lockout entry is then attached to the active connection's state.

### Step 3: Pre‑Request Lockout Check

Before dispatching to any model, the routing handler queries lockout status. From [`src/sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/auth.ts) (lines ≈ 1577‑1584):

```typescript
const modelLockout = requestedModel
    && modelLockout?.remainingMs > 0
    && isRetryableModelLockoutReason(modelLockout.reason)
        ? Date.now() + modelLockout.remainingMs
        : undefined;

```

If `remainingMs > 0`, the model is **skipped entirely** and the routing engine proceeds to the next viable target in the combo or fallback chain.

### Step 4: Automatic Recovery and Cleanup

Expired lockouts are removed through two mechanisms:

1. **Lazy eviction** – Checked on each access before routing decisions
2. **Periodic cleanup** – [`src/open-sse/services/accountFallback/lockoutEviction.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/open-sse/services/accountFallback/lockoutEviction.ts) sweeps stale entries

When cooldown expires, the model is immediately eligible again with no manual intervention.

## Configuring Model Lockout via Settings API

Enable and tune model lockout through the REST settings endpoint:

```typescript
// POST /api/settings
await fetch('/api/settings', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    modelLockout: {
      enabled: true,
      errorCodes: [429, 503, 529],
      baseCooldownMs: 10_000,      // 10 seconds initial
      maxCooldownMs: 300_000,      // 5 minute maximum
      exponentialBackoff: true,
      maxBackoffSteps: 5,
    },
  }),
});

```

The request is handled by [`src/app/api/settings/route.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/app/api/settings/route.ts), which persists configuration and invalidates cached settings.

## Inspecting Lockout State in Application Code

### Skip Logic in Chat Handlers

[`src/sse/handlers/chat.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/handlers/chat.ts) (line ≈ 2173) implements the same check pattern for chat completion flows:

```typescript
import { resolveModelLockoutSettings } from '@/lib/resilience/modelLockoutSettings';

export async function handleChatCore(request) {
  const settings = await getCachedSettings();
  const mlSettings = resolveModelLockoutSettings(settings);

  const lockout = mlSettings[request.model];
  if (lockout && lockout.remainingMs > 0) {
    // Skip this model – routing engine selects alternative
    return selectFallbackModel(request);
  }

  // Proceed with normal request flow
  const response = await dispatchToModel(request);
  // ...
}

```

### Monitoring and Observability

Operational visibility is provided through [`src/lib/monitoring/providerHealthMatrix.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/monitoring/providerHealthMatrix.ts):

```typescript
import { providerHealthMatrix } from '@/lib/monitoring/providerHealthMatrix';

const status = providerHealthMatrix.getProviderStatus('openai');
console.log(`Locked models: ${status.modelLockoutCount}`);
console.log(`Lockout details:`, status.modelLockoutDetails);

```

These metrics feed into the autopilot system ([`src/lib/monitoring/providerHealthAutopilot.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/monitoring/providerHealthAutopilot.ts)) for automated capacity decisions.

## Isolation Benefits

- **Granular Failure Containment** – Only the specific model‑connection pair is blocked; peer models continue serving traffic
- **Retry Storm Prevention** – Cooldown enforcement eliminates rapid retry loops that exacerbate rate limiting
- **Zero‑Touch Recovery** – Automatic expiration removes operational burden
- **Full Observability** – Per‑provider lockout counters enable data‑driven routing decisions

## Summary

- Model lockout in OmniRoute isolates failures at **provider‑connection‑model** granularity, not provider‑level
- Lockouts trigger on **configurable error codes** (429, 503 by default) with **exponential back‑off** support
- The `resolveModelLockoutSettings` function in [`src/lib/resilience/modelLockoutSettings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/modelLockoutSettings.ts) centralizes configuration parsing
- Pre‑request checks in [`src/sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/services/auth.ts) and [`src/sse/handlers/chat.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/sse/handlers/chat.ts) skip locked models before any upstream dispatch
- Automatic eviction via [`lockoutEviction.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/lockoutEviction.ts) ensures timely recovery without manual state management
- Health matrix integration provides operational metrics for monitoring and autopilot systems

## Frequently Asked Questions

### What error codes trigger model lockout by default?

OmniRoute defaults to **429** (Too Many Requests) and **503** (Service Unavailable). These are configurable through the `errorCodes` array in settings. The system only creates lockout entries for codes explicitly listed in your configuration.

### Can multiple models be locked out on the same API key simultaneously?

Yes. Each model maintains **independent lockout state** on the connection object. A quota exhaustion on `gpt-4-turbo` locks only that model; `gpt-4o` and other models on identical credentials remain available for routing.

### How quickly does a locked model become available again?

Recovery depends on your **back‑off configuration**. With `baseCooldownMs: 10000` and `exponentialBackoff: false`, recovery takes 10 seconds. With exponential back‑off enabled, consecutive failures double the wait time up to `maxCooldownMs`. The first successful request after recovery clears the failure counter.

### Does model lockout work across different provider connections?

No. Lockout state is **scoped to a single connection** (one API key or OAuth token). The same model name on a different credential has independent health status. This design prevents one customer's quota issue from affecting another's routing.