# Understanding OmniRoute's Model Lockout Feature: Isolating Failing Models to Preserve Provider Uptime

> Learn how OmniRoute's model lockout feature isolates failing models to prevent connection-wide cooldowns and maintain provider uptime. Keep other models operational even during errors.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: deep-dive
- Published: 2026-08-21

---

**OmniRoute's model lockout is a targeted resilience mechanism that isolates individual failing models on a provider connection—preventing errors like quota exhaustion or model-not-found from triggering connection-wide cooldowns while keeping other models operational.**

OmniRoute implements **model lockout** as a critical fault-isolation layer within its AI routing infrastructure. When a specific model encounters persistent errors such as 404 model-not-found, 429 quota-exhausted, or 403 forbidden responses, this feature locks only that model rather than placing the entire provider connection into cooldown. This granular approach ensures that multimodal providers backing many models behind a single API key remain partially available even when individual models fail.

## How Model Lockout Works in OmniRoute

The model lockout system operates through a five-stage lifecycle designed to minimize blast radius while maximizing diagnostic visibility.

**Preventing Connection Poisoning**

Providers that multiplex many models behind one API key—such as OpenAI-compatible endpoints, Gemini, Codex, or Antigravity—risk having a single model error trigger a connection-wide circuit breaker. The model lockout feature prevents this by isolating failures to the specific `provider:connectionId:model` combination. In [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts), the `hasPerModelQuota()` function determines whether a provider should use per-model lockouts instead of connection-wide cooldowns.

**Persisting Failures with Configurable Back-Off**

Transient errors should not trigger immediate retries. The system persists lockout states for configurable durations using exponential back-off algorithms. The `lockModel()` and `recordModelLockoutFailure()` functions manage these durations, while `ensureCleanupTimer` handles automatic expiration of stale entries. When upstream providers return precise reset timestamps—such as Antigravity's 429 rate_limit_exceeded responses—the system honors these via the `exactCooldownMs` parameter rather than applying generic limits.

**Maintaining Service for Healthy Models**

By restricting the failure domain to the offending model, the router continues servicing requests for remaining healthy models on the same connection. The routing logic consults `isModelLocked()` before adding candidates to request pools, automatically excluding locked models from auto-combo and combo-quota selections.

**Exposing Operational Controls**

Operators gain visibility and manual intervention capabilities through the Resilience API endpoint `/api/resilience/model-cooldowns` and the management dashboard. Functions like `getAllModelLockouts()` and `clearModelLock()` enable rapid recovery when upstream models are restored.

## Architecture Context: The Third Resilience Layer

Model lockout represents the **third layer** in OmniRoute's defense-in-depth resilience strategy, operating after provider circuit-breakers and connection-wide cooldowns. According to the architecture documentation in [`docs/architecture/RESILIENCE_GUIDE.md`](https://github.com/diegosouzapw/OmniRoute/blob/main/docs/architecture/RESILIENCE_GUIDE.md), this layering ensures that only the smallest possible failure unit—the individual model—is isolated during fault conditions.

The implementation stores lockout entries as `ModelLockoutEntry` objects mapped to composite keys (`provider:connectionId:model`). These entries track `reason`, `until` timestamps, `failureCount`, and upstream reset indicators to inform routing decisions and decay algorithms.

## Core Implementation Files

**[`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts)**

This file contains the primary lockout logic, including:
- `hasPerModelQuota()` (provider capability detection)
- `lockModel()` (lines 1000–1035) and `recordModelLockoutFailure()` (creation and back-off management)
- `isModelLocked()` (lines 883–894) and `getModelLockoutInfo()` (lines 999–1016) for state queries
- `decayModelFailureCount()` for reducing failure counts after successful cooldown periods

**[`src/lib/resilience/modelLockoutSettings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/resilience/modelLockoutSettings.ts)**

Defines configuration parameters controlling maximum cooldown durations, exponential back-off multipliers, and default behaviors for per-model isolation.

**[`src/app/api/v1/resilience/model-cooldowns/route.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/app/api/v1/resilience/model-cooldowns/route.ts)**

Implements the HTTP handler exposing CRUD operations for lockout management, including bulk deletion and filtered querying capabilities.

## Practical Implementation Examples

### Manually Locking a Model After Quota Exhaustion

When detecting provider-specific rate limits, applications can programmatically lock models to prevent further requests:

```typescript
import { lockModel } from "@/open-sse/services/accountFallback.ts";

const provider = "openai";
const connectionId = "openai-conn-1";
const model = "gpt-4";
const reason = "quota_exhausted";
const cooldownMs = 3_600_000; // 1 hour or upstream-provided value

lockModel(provider, connectionId, model, reason, cooldownMs);

```

*Source*: `lockModel()` implementation in [`accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/accountFallback.ts) (lines 1000–1035).

### Checking Lockout Status Before Routing

Routing logic should verify model availability before inclusion in candidate pools:

```typescript
import { isModelLocked } from "@/open-sse/services/accountFallback.ts";

if (await isModelLocked("openai", "openai-conn-1", "gpt-4")) {
  // Skip this model, try another one
}

```

*Source*: `isModelLocked()` in [`accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/accountFallback.ts) (lines 883–894).

### Retrieving Lockout Diagnostics

For monitoring dashboards or debugging interfaces:

```typescript
import { getModelLockoutInfo } from "@/open-sse/services/accountFallback.ts";

const info = await getModelLockoutInfo("openai", "openai-conn-1", "gpt-4");
if (info) {
  console.log(`Model locked for ${info.reason}, ${info.remainingMs / 1000}s remaining`);
}

```

*Source*: `getModelLockoutInfo()` in [`accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/accountFallback.ts) (lines 999–1016).

### Clearing Lockouts via Management API

Operators can manually restore model availability:

```bash
curl -X DELETE \
  -H "Authorization: Bearer <management-token>" \
  -d '{"provider":"openai","model":"gpt-4"}' \
  https://localhost:20128/api/resilience/model-cooldowns

```

*Reference*: API documentation in [`docs/reference/API_REFERENCE.md`](https://github.com/diegosouzapw/OmniRoute/blob/main/docs/reference/API_REFERENCE.md).

## Supported Error Classifications

The `checkFallbackError` function classifies specific HTTP status codes and error patterns to trigger model lockout:

- **404 Model Not Found**: Permanent errors indicating the model identifier is invalid or deprecated
- **429 Quota Exhausted**: Rate limiting errors where `reason` maps to `quota_exhausted` or `rate_limit_exceeded`
- **403 Forbidden**: Authorization failures specific to the model tier or region

When `hasPerModelQuota()` returns true for a provider, these errors trigger `recordModelLockoutFailure()` rather than connection-wide cooldown mechanisms.

## Summary

- **OmniRoute's model lockout** isolates individual failing models without disrupting entire provider connections, preserving availability for healthy models on the same API key.
- The mechanism stores state using composite keys (`provider:connectionId:model`) with configurable exponential back-off and upstream reset time respect.
- Core functions reside in [`accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/accountFallback.ts), including `isModelLocked()` for routing guards and `lockModel()` for programmatic control.
- Operators manage lockouts through the `/api/resilience/model-cooldowns` endpoint and dashboard interfaces.
- The feature supports precise error classification (404, 429, 403) and integrates with auto-combo routing logic to exclude unavailable models from candidate pools.

## Frequently Asked Questions

### What triggers a model lockout in OmniRoute?

A model lockout triggers when `checkFallbackError` classifies a response as a permanent or quota-related failure—specifically HTTP 404 (model-not-found), 429 (quota-exhausted/rate-limit-exceeded), or 403 (forbidden) errors. If `hasPerModelQuota()` indicates the provider supports per-model isolation, the system calls `recordModelLockoutFailure()` to create a `ModelLockoutEntry` with exponential back-off rather than applying a connection-wide cooldown.

### How does model lockout differ from connection cooldown?

Connection cooldown blocks all traffic to a specific provider connection ID, affecting every model behind that API key. **Model lockout** restricts the failure domain to a single `provider:connectionId:model` combination, allowing requests to other models on the same connection to proceed normally. This distinction prevents "noisy neighbor" scenarios where one deprecated or exhausted model disables access to dozens of functional models.

### Can operators manually clear a model lockout?

Yes. The Resilience API exposes management endpoints under `/api/resilience/model-cooldowns` that accept DELETE requests to clear specific lockouts. The `clearModelLock()` function in [`accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/accountFallback.ts) removes entries from the lockout map immediately, while `getAllModelLockouts()` supports bulk operations and filtered queries for administrative dashboards.

### Which providers support per-model lockout behavior?

Providers multiplexing multiple models behind unified API endpoints—such as OpenAI-compatible services, Google Gemini, Codex, and Antigravity—utilize per-model lockout when configured via `hasPerModelQuota()`. The [`modelLockoutSettings.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/modelLockoutSettings.ts) configuration file controls default behaviors and maximum cooldown thresholds for these providers, while passthrough providers may utilize alternative resilience strategies documented in [`tests/unit/vertex-passthrough-model-lockout.test.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/tests/unit/vertex-passthrough-model-lockout.test.ts).