Understanding OmniRoute's Model Lockout Feature: Isolating Failing Models to Preserve Provider Uptime

OmniRoute's model lockout is a targeted resilience mechanism that isolates individual failing models on a provider connection—preventing errors like quota exhaustion or model-not-found from triggering connection-wide cooldowns while keeping other models operational.

OmniRoute implements model lockout as a critical fault-isolation layer within its AI routing infrastructure. When a specific model encounters persistent errors such as 404 model-not-found, 429 quota-exhausted, or 403 forbidden responses, this feature locks only that model rather than placing the entire provider connection into cooldown. This granular approach ensures that multimodal providers backing many models behind a single API key remain partially available even when individual models fail.

How Model Lockout Works in OmniRoute

The model lockout system operates through a five-stage lifecycle designed to minimize blast radius while maximizing diagnostic visibility.

Preventing Connection Poisoning

Providers that multiplex many models behind one API key—such as OpenAI-compatible endpoints, Gemini, Codex, or Antigravity—risk having a single model error trigger a connection-wide circuit breaker. The model lockout feature prevents this by isolating failures to the specific provider:connectionId:model combination. In open-sse/services/accountFallback.ts, the hasPerModelQuota() function determines whether a provider should use per-model lockouts instead of connection-wide cooldowns.

Persisting Failures with Configurable Back-Off

Transient errors should not trigger immediate retries. The system persists lockout states for configurable durations using exponential back-off algorithms. The lockModel() and recordModelLockoutFailure() functions manage these durations, while ensureCleanupTimer handles automatic expiration of stale entries. When upstream providers return precise reset timestamps—such as Antigravity's 429 rate_limit_exceeded responses—the system honors these via the exactCooldownMs parameter rather than applying generic limits.

Maintaining Service for Healthy Models

By restricting the failure domain to the offending model, the router continues servicing requests for remaining healthy models on the same connection. The routing logic consults isModelLocked() before adding candidates to request pools, automatically excluding locked models from auto-combo and combo-quota selections.

Exposing Operational Controls

Operators gain visibility and manual intervention capabilities through the Resilience API endpoint /api/resilience/model-cooldowns and the management dashboard. Functions like getAllModelLockouts() and clearModelLock() enable rapid recovery when upstream models are restored.

Architecture Context: The Third Resilience Layer

Model lockout represents the third layer in OmniRoute's defense-in-depth resilience strategy, operating after provider circuit-breakers and connection-wide cooldowns. According to the architecture documentation in docs/architecture/RESILIENCE_GUIDE.md, this layering ensures that only the smallest possible failure unit—the individual model—is isolated during fault conditions.

The implementation stores lockout entries as ModelLockoutEntry objects mapped to composite keys (provider:connectionId:model). These entries track reason, until timestamps, failureCount, and upstream reset indicators to inform routing decisions and decay algorithms.

Core Implementation Files

open-sse/services/accountFallback.ts

This file contains the primary lockout logic, including:

• hasPerModelQuota() (provider capability detection)
• lockModel() (lines 1000–1035) and recordModelLockoutFailure() (creation and back-off management)
• isModelLocked() (lines 883–894) and getModelLockoutInfo() (lines 999–1016) for state queries
• decayModelFailureCount() for reducing failure counts after successful cooldown periods

src/lib/resilience/modelLockoutSettings.ts

Defines configuration parameters controlling maximum cooldown durations, exponential back-off multipliers, and default behaviors for per-model isolation.

src/app/api/v1/resilience/model-cooldowns/route.ts

Implements the HTTP handler exposing CRUD operations for lockout management, including bulk deletion and filtered querying capabilities.

Practical Implementation Examples

Manually Locking a Model After Quota Exhaustion

When detecting provider-specific rate limits, applications can programmatically lock models to prevent further requests:

import { lockModel } from "@/open-sse/services/accountFallback.ts";

const provider = "openai";
const connectionId = "openai-conn-1";
const model = "gpt-4";
const reason = "quota_exhausted";
const cooldownMs = 3_600_000; // 1 hour or upstream-provided value

lockModel(provider, connectionId, model, reason, cooldownMs);

Source: lockModel() implementation in accountFallback.ts (lines 1000–1035).

Checking Lockout Status Before Routing

Routing logic should verify model availability before inclusion in candidate pools:

import { isModelLocked } from "@/open-sse/services/accountFallback.ts";

if (await isModelLocked("openai", "openai-conn-1", "gpt-4")) {
  // Skip this model, try another one
}

Source: isModelLocked() in accountFallback.ts (lines 883–894).

Retrieving Lockout Diagnostics

For monitoring dashboards or debugging interfaces:

import { getModelLockoutInfo } from "@/open-sse/services/accountFallback.ts";

const info = await getModelLockoutInfo("openai", "openai-conn-1", "gpt-4");
if (info) {
  console.log(`Model locked for ${info.reason}, ${info.remainingMs / 1000}s remaining`);
}

Source: getModelLockoutInfo() in accountFallback.ts (lines 999–1016).

Clearing Lockouts via Management API

Operators can manually restore model availability:

curl -X DELETE \
  -H "Authorization: Bearer <management-token>" \
  -d '{"provider":"openai","model":"gpt-4"}' \
  https://localhost:20128/api/resilience/model-cooldowns

Reference: API documentation in docs/reference/API_REFERENCE.md.

Supported Error Classifications

The checkFallbackError function classifies specific HTTP status codes and error patterns to trigger model lockout:

• 404 Model Not Found: Permanent errors indicating the model identifier is invalid or deprecated
• 429 Quota Exhausted: Rate limiting errors where reason maps to quota_exhausted or rate_limit_exceeded
• 403 Forbidden: Authorization failures specific to the model tier or region

When hasPerModelQuota() returns true for a provider, these errors trigger recordModelLockoutFailure() rather than connection-wide cooldown mechanisms.

Summary

• OmniRoute's model lockout isolates individual failing models without disrupting entire provider connections, preserving availability for healthy models on the same API key.
• The mechanism stores state using composite keys (provider:connectionId:model) with configurable exponential back-off and upstream reset time respect.
• Core functions reside in accountFallback.ts, including isModelLocked() for routing guards and lockModel() for programmatic control.
• Operators manage lockouts through the /api/resilience/model-cooldowns endpoint and dashboard interfaces.
• The feature supports precise error classification (404, 429, 403) and integrates with auto-combo routing logic to exclude unavailable models from candidate pools.

Frequently Asked Questions

What triggers a model lockout in OmniRoute?

A model lockout triggers when checkFallbackError classifies a response as a permanent or quota-related failure—specifically HTTP 404 (model-not-found), 429 (quota-exhausted/rate-limit-exceeded), or 403 (forbidden) errors. If hasPerModelQuota() indicates the provider supports per-model isolation, the system calls recordModelLockoutFailure() to create a ModelLockoutEntry with exponential back-off rather than applying a connection-wide cooldown.

How does model lockout differ from connection cooldown?

Connection cooldown blocks all traffic to a specific provider connection ID, affecting every model behind that API key. Model lockout restricts the failure domain to a single provider:connectionId:model combination, allowing requests to other models on the same connection to proceed normally. This distinction prevents "noisy neighbor" scenarios where one deprecated or exhausted model disables access to dozens of functional models.

Can operators manually clear a model lockout?

Yes. The Resilience API exposes management endpoints under /api/resilience/model-cooldowns that accept DELETE requests to clear specific lockouts. The clearModelLock() function in accountFallback.ts removes entries from the lockout map immediately, while getAllModelLockouts() supports bulk operations and filtered queries for administrative dashboards.

Which providers support per-model lockout behavior?

Providers multiplexing multiple models behind unified API endpoints—such as OpenAI-compatible services, Google Gemini, Codex, and Antigravity—utilize per-model lockout when configured via hasPerModelQuota(). The modelLockoutSettings.ts configuration file controls default behaviors and maximum cooldown thresholds for these providers, while passthrough providers may utilize alternative resilience strategies documented in tests/unit/vertex-passthrough-model-lockout.test.ts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}