How OmniRoute's Model Lockout Feature Prevents Repeated Failures

OmniRoute prevents repeated failures by recording a temporary model lockout that excludes failing provider/model pairs from the candidate pool until their cooldown expires.

OmniRoute's model lockout system is a granular resilience mechanism that isolates problematic AI models without penalizing entire provider connections. Implemented in the accountFallback.ts service, this feature tracks failure patterns, applies intelligent cooldowns, and automatically decays lockouts to restore healthy traffic flow.

---

Detecting Failures That Trigger Lockouts

The lockout process begins when an upstream request returns a status code ≥ 400 with a classified error. In open-sse/services/accountFallback.ts, the recordModelLockoutFailure function handles this detection:

// Lines 604-610 in accountFallback.ts
// Called when errors like 'rate_limit' or 'quota_exhausted' occur
recordModelLockoutFailure(
  provider: string,
  connectionId: string,
  model: string,
  reason: string,
  status: number,
  fallbackCooldownMs: number,
  profile?: ProviderProfile,
  options?: { exactCooldownIsUpstreamReset?: boolean }
)

The function captures critical context: which provider, connection, and model failed, plus the specific failure reason that determines how the cooldown is calculated.

---

Building Unique Lock Keys

Each lockout receives a deterministic identifier to prevent collisions and enable precise targeting. The lock key construction appears at lines 632-635:

// Standard key format: provider:connectionId:model
const lockKey = `${provider}:${connectionId}:${model}`;

// For quota-family providers, an "exact" key may be used
const exactKey = `${provider}:${connectionId}:${model}:exact`;

This string-based keying ensures that lockouts are scoped to specific model instances rather than applied broadly across unrelated workloads.

---

Computing Intelligent Cooldown Periods

OmniRoute calculates cooldowns through two complementary strategies in accountFallback.ts:

Explicit Upstream Resets

When providers supply a Retry-After header or similar signal, selectLockoutCooldownMs (lines 94-102) respects that exact duration:

// Use the upstream's requested reset time
const cooldownMs = selectLockoutCooldownMs(
  reason,
  upstreamResetTimestamp,  // From Retry-After or API response
  profile
);

Exponential Back-Off

Without explicit guidance, getScaledCooldown (lines 68-74) applies failure-count-based scaling:

// Back-off increases with repeated failures
const cooldownMs = getScaledCooldown(
  baseCooldown,
  failureCount,
  profile?.resilience?.backoffMultiplier
);

The combination ensures fast recovery for transient errors and aggressive throttling for persistent problems.

---

Persisting Lockout State

Lockout entries are stored in an in-memory modelLockouts map with full audit context. The ModelLockoutEntry type (lines 86-93) captures:

interface ModelLockoutEntry {
  reason: string;        // e.g., 'quota_exhausted', 'rate_limit'
  until: number;         // Unix timestamp when lock expires
  lockedAt: number;      // When lock was created
  failureCount: number;  // Cumulative failures for this key
  cooldownMs: number;    // Applied cooldown duration
  isExact: boolean;      // Whether this is a quota-family exact lock
}

This structured storage enables both runtime decisions and operational visibility without external database dependencies.

---

Applying Granular Locks

OmniRoute distinguishes between per-model quotas and connection-wide limits through conditional logic at lines 223-238:

Per-Model Quota Providers

For providers flagging hasPerModelQuota, lockModelIfPerModelQuota isolates only the offending model:

// Lock applies to this specific model only
lockModelIfPerModelQuota(provider, connectionId, model, entry);
// Other models on same connection remain available

Standard Providers

Without per-model quota support, the lock functions as a connection-level cooldown, protecting upstream resources while minimizing blast radius.

---

Enforcing Locks During Request Routing

The critical enforcement point occurs during credential selection. The isModelLocked function (lines 220-227) filters the candidate pool:

// Before selecting credentials for a request
if (isModelLocked(provider, connectionId, model)) {
  // Skip this model—it's under active lockout
  continue; // Move to next candidate
}

This pre-selection filtering guarantees that locked models never receive traffic, eliminating wasted requests and repeated failures.

---

Automatic Decay and Recovery

Lockouts self-heal through two mechanisms (lines 86-96):

1. Explicit expiration — locks are automatically removed when until timestamp passes
2. Failure count decay — decayModelFailureCount reduces historical failure weight over time, preventing permanent blacklisting

// Gradual forgiveness for stabilized models
decayModelFailureCount(lockKey, decayFactor);

This adaptive recovery ensures models can re-enter rotation once conditions improve.

---

Operational Visibility

Administrators monitor lockout state through dedicated query functions (lines 336-354):

Function	Purpose
getModelLockoutInfo(provider, connectionId, model)	Check single model status
getAllModelLockouts()	List all active lockouts for dashboard display

The providerHealthAutopilot.ts module surfaces this data for real-time monitoring and manual lockout clearance when needed.

---

Summary

• Detection: recordModelLockoutFailure captures classified errors (≥400 status) with full context
• Identification: Unique provider:connectionId:model keys enable precise targeting
• Cooldown: Respects upstream Retry-After or applies exponential back-off via getScaledCooldown
• Storage: ModelLockoutEntry structs in modelLockouts map capture reason, expiration, and failure history
• Enforcement: isModelLocked filters failing models before request assignment
• Recovery: Automatic expiration and failure count decay restore healthy models
• Visibility: getAllModelLockouts exposes state for operational dashboards

---

Frequently Asked Questions

What types of errors trigger a model lockout?

Lockouts activate on HTTP status ≥400 when accompanied by classified reasons like rate_limit, quota_exhausted, or invalid_auth. The recordModelLockoutFailure function in accountFallback.ts only creates entries for explicitly categorized failures, ensuring transient network errors don't unnecessarily disable models.

How does OmniRoute handle providers with per-model quotas?

For providers advertising hasPerModelQuota, lockModelIfPerModelQuota applies the lock exclusively to the failing model. Other models on the same connection continue serving traffic. Without this flag, the cooldown applies connection-wide as a protective measure.

Can operators manually clear a model lockout?

Yes. The providerHealthAutopilot.ts monitoring module exposes administrative endpoints for lockout inspection and manual clearance. Operators can view active lockouts via getAllModelLockouts and remove entries when confident the underlying issue is resolved.

What prevents permanent blacklisting of recovered models?

The decayModelFailureCount mechanism gradually reduces historical failure counts, and all locks carry explicit until timestamps. Once expired, models automatically return to the candidate pool with refreshed back-off calculations, ensuring no model remains locked indefinitely.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}