How Does Model Lockout in OmniRoute Isolate Per-Model Failures?

OmniRoute's model lockout mechanism isolates per‑model failures by tracking error‑code‑triggered cooldowns at the provider‑connection‑model scope, automatically skipping locked models during routing while keeping other models on the same credential fully operational.

Model lockout is a core resilience feature in the OmniRoute routing engine (diegosouzapw/OmniRoute). It prevents a single misbehaving AI model—whether hitting rate limits, quota exhaustion, or temporary outages—from degrading the entire provider connection or forcing a complete combo routing failure. This article explains the architectural flow, configuration options, and source code implementation that make granular failure isolation possible.

Core Architecture of Model Lockout

Scope and Granularity

Unlike coarse‑grained circuit breakers that disable entire providers, model lockout operates at three nested levels:

• Provider – The upstream service (e.g., OpenAI, Anthropic)
• Connection – A specific credential (OAuth token or API key)
• Model – The individual model identifier (e.g., gpt-4-turbo, claude-3-opus-20240229)

This means a 429 error on gpt-4-turbo for one API key does not affect gpt-3.5-turbo on that same key, nor does it impact other connections to the same provider.

Lockout State Structure

When triggered, a lockout entry is stored on the connection object with these fields:

Field	Purpose
remainingMs	Milliseconds until the model becomes eligible again
reason	Human‑readable cause (e.g., quota‑exhausted, rate‑limited)
consecutiveFailures	Counter for exponential back‑off calculation

The entry is keyed by <providerId>::<modelName> in the connection's modelLockouts Map.

How Model Lockout Is Triggered and Enforced

Step 1: Configuration Resolution

Every request begins by resolving active lockout settings. In src/lib/resilience/modelLockoutSettings.ts, the resolveModelLockoutSettings function parses global configuration:

// Excerpt pattern from src/sse/services/auth.ts
const mlSettings = resolveModelLockoutSettings(await getCachedSettings());

This returns a normalized structure including:

• enabled: boolean
• errorCodes: number[] – Which HTTP status codes trigger lockout (default: [429, 503])
• baseCooldownMs: number – Initial back‑off duration
• maxCooldownMs: number – Ceiling for back‑off
• exponentialBackoff: boolean – Whether to double duration per consecutive failure
• maxBackoffSteps: number – Maximum doublings before hitting ceiling

Step 2: Error Detection and State Creation

After an upstream error response, the account‑fallback logic in the SSE layer evaluates whether the status code matches the configured lockout list. If matched, it calculates cooldown duration:

cooldown = exponentialBackoff
  ? min(baseCooldownMs × 2^consecutiveFailures, maxCooldownMs)
  : baseCooldownMs

The lockout entry is then attached to the active connection's state.

Step 3: Pre‑Request Lockout Check

Before dispatching to any model, the routing handler queries lockout status. From src/sse/services/auth.ts (lines ≈ 1577‑1584):

const modelLockout = requestedModel
    && modelLockout?.remainingMs > 0
    && isRetryableModelLockoutReason(modelLockout.reason)
        ? Date.now() + modelLockout.remainingMs
        : undefined;

If remainingMs > 0, the model is skipped entirely and the routing engine proceeds to the next viable target in the combo or fallback chain.

Step 4: Automatic Recovery and Cleanup

Expired lockouts are removed through two mechanisms:

1. Lazy eviction – Checked on each access before routing decisions
2. Periodic cleanup – src/open-sse/services/accountFallback/lockoutEviction.ts sweeps stale entries

When cooldown expires, the model is immediately eligible again with no manual intervention.

Configuring Model Lockout via Settings API

Enable and tune model lockout through the REST settings endpoint:

// POST /api/settings
await fetch('/api/settings', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    modelLockout: {
      enabled: true,
      errorCodes: [429, 503, 529],
      baseCooldownMs: 10_000,      // 10 seconds initial
      maxCooldownMs: 300_000,      // 5 minute maximum
      exponentialBackoff: true,
      maxBackoffSteps: 5,
    },
  }),
});

The request is handled by src/app/api/settings/route.ts, which persists configuration and invalidates cached settings.

Inspecting Lockout State in Application Code

Skip Logic in Chat Handlers

src/sse/handlers/chat.ts (line ≈ 2173) implements the same check pattern for chat completion flows:

import { resolveModelLockoutSettings } from '@/lib/resilience/modelLockoutSettings';

export async function handleChatCore(request) {
  const settings = await getCachedSettings();
  const mlSettings = resolveModelLockoutSettings(settings);

  const lockout = mlSettings[request.model];
  if (lockout && lockout.remainingMs > 0) {
    // Skip this model – routing engine selects alternative
    return selectFallbackModel(request);
  }

  // Proceed with normal request flow
  const response = await dispatchToModel(request);
  // ...
}

Monitoring and Observability

Operational visibility is provided through src/lib/monitoring/providerHealthMatrix.ts:

import { providerHealthMatrix } from '@/lib/monitoring/providerHealthMatrix';

const status = providerHealthMatrix.getProviderStatus('openai');
console.log(`Locked models: ${status.modelLockoutCount}`);
console.log(`Lockout details:`, status.modelLockoutDetails);

These metrics feed into the autopilot system (src/lib/monitoring/providerHealthAutopilot.ts) for automated capacity decisions.

Isolation Benefits

• Granular Failure Containment – Only the specific model‑connection pair is blocked; peer models continue serving traffic
• Retry Storm Prevention – Cooldown enforcement eliminates rapid retry loops that exacerbate rate limiting
• Zero‑Touch Recovery – Automatic expiration removes operational burden
• Full Observability – Per‑provider lockout counters enable data‑driven routing decisions

Summary

• Model lockout in OmniRoute isolates failures at provider‑connection‑model granularity, not provider‑level
• Lockouts trigger on configurable error codes (429, 503 by default) with exponential back‑off support
• The resolveModelLockoutSettings function in src/lib/resilience/modelLockoutSettings.ts centralizes configuration parsing
• Pre‑request checks in src/sse/services/auth.ts and src/sse/handlers/chat.ts skip locked models before any upstream dispatch
• Automatic eviction via lockoutEviction.ts ensures timely recovery without manual state management
• Health matrix integration provides operational metrics for monitoring and autopilot systems

Frequently Asked Questions

What error codes trigger model lockout by default?

OmniRoute defaults to 429 (Too Many Requests) and 503 (Service Unavailable). These are configurable through the errorCodes array in settings. The system only creates lockout entries for codes explicitly listed in your configuration.

Can multiple models be locked out on the same API key simultaneously?

Yes. Each model maintains independent lockout state on the connection object. A quota exhaustion on gpt-4-turbo locks only that model; gpt-4o and other models on identical credentials remain available for routing.

How quickly does a locked model become available again?

Recovery depends on your back‑off configuration. With baseCooldownMs: 10000 and exponentialBackoff: false, recovery takes 10 seconds. With exponential back‑off enabled, consecutive failures double the wait time up to maxCooldownMs. The first successful request after recovery clears the failure counter.

Does model lockout work across different provider connections?

No. Lockout state is scoped to a single connection (one API key or OAuth token). The same model name on a different credential has independent health status. This design prevents one customer's quota issue from affecting another's routing.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}